Commit Graph

634 Commits (3f65ccc5739b6aff616485eb46544dafc194062d)

Author SHA1 Message Date
Raphaël Vinot f886b8676b fix: More exceptions catching for the the new caching method 2022-09-26 20:55:16 +02:00
Raphaël Vinot 159d444b20 fix: Avoid exception on broken captures 2022-09-26 20:50:42 +02:00
Raphaël Vinot 31261e84c2 fix: Better handling of half broken captures without HAR files 2022-09-26 14:58:30 +02:00
Raphaël Vinot 1c2861ebec chg: Bump jquery 2022-09-23 13:13:09 +02:00
Raphaël Vinot d38b612c37 chg: Bump lacuscore 2022-09-23 13:13:09 +02:00
Raphaël Vinot c6464936fc chg: Bump to poetry v1.2, remove dep on setuptools 2022-08-31 16:33:13 +02:00
Raphaël Vinot c201592c22 chg: Open hashlookup to everyone 2022-08-26 13:41:49 +02:00
Raphaël Vinot 0ae4610270 chg: Notify pandora link copied in clipboard 2022-08-25 15:16:28 +02:00
Raphaël Vinot 2be8a4e4ff chg: Copy pandora link into clipboard 2022-08-25 15:08:08 +02:00
Raphaël Vinot 3d2029956d chg: Replace dl icon 2022-08-25 13:44:30 +02:00
Raphaël Vinot ec07429e65 chg: Much better handling of downloaded files 2022-08-25 13:28:02 +02:00
Raphaël Vinot cbe73fa13f fix: better handling of downloaded content from URL 2022-08-24 14:06:48 +02:00
Raphaël Vinot f4dfa0343e new: Submit to Pandora button for file URLs 2022-08-23 17:48:36 +02:00
Raphaël Vinot f232eba662 chg: Improve UA rendering 2022-08-23 17:44:48 +02:00
Raphaël Vinot ebbe6e3ce9 new: Pick mobile devices on capture page 2022-08-22 17:34:00 +02:00
Raphaël Vinot d63ea473f5 new: Autoselect browser engine based on the UA 2022-08-19 14:26:22 +02:00
Raphaël Vinot 998ef12b06 new: Add support for playwright devices and browser name (API only) 2022-08-18 11:19:32 +02:00
Raphaël Vinot 1c8ae0db52 chg: Make user-agent selection more clear 2022-08-17 21:37:53 +02:00
Raphaël Vinot a71950866e chg: Improve hostnode view 2022-08-14 17:49:04 +02:00
Raphaël Vinot 3c30218cf5 Merge branch 'Arhamyss-HostnodeDl' 2022-08-14 17:23:23 +02:00
Raphaël Vinot d6dc30b260 chg: cleanup formating 2022-08-14 17:14:48 +02:00
Raphaël Vinot 13eaa72a57 chg: Display size of rendered page
Fix #482
2022-08-14 16:43:37 +02:00
Raphaël Vinot 87be90af83 fix: Avoid exceptions on invalid requests 2022-08-14 16:41:37 +02:00
Raphaël Vinot 1ddd38fdcd chg: Improve wording on capture page 2022-08-14 16:41:37 +02:00
Raphaël Vinot 72c4e43474 new: Upload a file instead of submitting a URL. 2022-08-14 16:41:34 +02:00
Raphaël Vinot 2ce8b5a96c new: dropdown to pass DoNotTrack HTTP header
Improvments on the capture page.
2022-08-14 16:31:51 +02:00
Raphaël Vinot fdfa2e25cb new: Initial listing of all resources loaded on the capture 2022-08-14 16:31:49 +02:00
Arhamyss c589933a40 fix: changed the way things are displayed and fixed errors to make the download working 2022-08-10 11:40:11 +02:00
Arhamyss 5a77044530 new: first try to add an url in the hostnode to dl the file if there is one to be downloaded 2022-08-09 18:23:11 +02:00
Raphaël Vinot 474b3275d3 chg: Display size of rendered page
Fix #482
2022-08-05 16:43:41 +02:00
Raphaël Vinot 49f335405e fix: Avoid exceptions on invalid requests 2022-08-05 11:28:44 +02:00
Raphaël Vinot 2f87a77c1b chg: Improve wording on capture page 2022-08-04 17:15:21 +02:00
Raphaël Vinot 4f72d64735 new: Upload a file instead of submitting a URL. 2022-08-04 16:58:07 +02:00
Raphaël Vinot 3170038db7 new: dropdown to pass DoNotTrack HTTP header
Improvments on the capture page.
2022-08-03 12:07:45 +02:00
Raphaël Vinot 83c6c308c5 new: Initial listing of all resources loaded on the capture 2022-08-01 17:51:43 +02:00
Raphaël Vinot 9b4d56a0b6 fix: Do not display all the recent captures with the same hash in popup 2022-07-28 16:14:47 +02:00
Raphaël Vinot b8cd1319a5 fix: Speedup display of hostnode popup 2022-07-27 14:36:56 +02:00
Raphaël Vinot e94b1e05b8 new: Support for in-HAR IPs, TLS/SSL info 2022-07-19 11:22:33 +02:00
Raphaël Vinot ce8eeda9eb chg: Improve RiskIQ module 2022-07-18 13:24:20 +02:00
Raphaël Vinot c0bde711d2 new: Very basic RiskIQ support for PDNS. 2022-07-15 18:53:49 +02:00
Raphaël Vinot 81f10c7cb8 new: config key to display a link to the github page 2022-07-12 10:24:10 +02:00
Raphaël Vinot 0b93a0b356 chg: Update SRIs for new version of d3js 2022-07-11 09:30:44 +02:00
Raphaël Vinot 89782155c4 chg: Bump js deps 2022-06-26 16:05:18 +01:00
Raphaël Vinot ce9ecfefdb chg: remove reference to max depth (not used since playwright), cleanup 2022-06-11 14:49:58 +02:00
Raphaël Vinot 6ba019ec83 chg: Improve somewhat the useragents available for capturing
Fix #416
2022-06-09 18:58:17 +02:00
Raphaël Vinot c60da37707 fix: properly display warning if a capture failed 2022-05-25 16:04:23 +02:00
Raphaël Vinot 358edc9157 chg: Bump datatables 2022-05-23 00:20:38 +02:00
Raphaël Vinot e4a2b0752c chg: dirty improve default UA 2022-05-05 16:30:52 +02:00
Raphaël Vinot cc17b414ff chg: Remove refs to splash 2022-04-25 14:43:02 +02:00
Raphaël Vinot 8d159ffba0 new: Switch away from splash to use playwright 2022-04-21 14:55:07 +03:00
Raphaël Vinot 150171b814 chg: Bump sri 2022-04-15 01:20:38 +02:00
Raphaël Vinot 59b1112aff chg: Bump deps, sri 2022-04-09 12:23:30 +02:00
Raphaël Vinot b2566865b4 chg: Make logo on capture page smaller 2022-03-31 16:21:00 +02:00
Raphaël Vinot 248f9a2340 fix: Wrap long URLs on subsequent capture page 2022-03-31 13:35:33 +02:00
Raphaël Vinot cf46dde1ed chg: Add basic pre-hook config 2022-03-31 11:30:53 +02:00
Raphaël Vinot ee69e66750 new: Multiple queries at once on web interface 2022-03-31 00:48:58 +02:00
Raphaël Vinot f17f6f1a37 new: toggle select in subsequent URLs capture 2022-03-30 13:48:10 +02:00
Raphaël Vinot 954121a363 chg: Improve rendering capture page 2022-03-30 12:32:20 +02:00
Raphaël Vinot 5cdc8c56ef chg: Bump JS deps 2022-03-30 12:20:43 +02:00
Raphaël Vinot 66dcbafaf6 fix: Use same color for all blue buttons
Fix #339
2022-03-18 10:48:12 +01:00
Raphaël Vinot acf5ff2fdf chg: Improve stats page, bump deps 2022-02-28 13:38:45 +01:00
Raphaël Vinot e496277811 fix: font size in top menu, top margin for tree. 2022-01-14 17:03:39 +01:00
Raphaël Vinot 787258dec1 chg: Bump bootstrap-flask, use bootstrap 5 2022-01-14 15:58:06 +01:00
Raphaël Vinot 45ec8efe64 chg: Bump d3js 2022-01-10 11:48:46 +01:00
Raphaël Vinot f8ed5567fa new: Link to changelog on index 2022-01-03 17:04:10 +01:00
Raphaël Vinot c551e4b725 chg: Improve legend with titles
Related: #279
2021-12-22 14:44:56 +01:00
Raphaël Vinot 83f6674dca chg: Add sri to stats page 2021-12-13 17:02:25 +01:00
Raphaël Vinot ec5563337b chg: Bump d3js 2021-12-13 16:39:12 +01:00
Raphaël Vinot 8cceb948f3 new: Get any supported hash algorithm over the API 2021-12-02 17:55:02 +01:00
Raphaël Vinot e291fc6167 chg: avoid duplicate URLs in hashlookup view 2021-12-01 17:03:36 +01:00
Raphaël Vinot fc4c342316 new: Add stats on hits on hashlookup 2021-11-30 15:43:42 +01:00
Raphaël Vinot 8f4040b806 new: Hashlookup integration 2021-11-30 14:59:48 +01:00
Raphaël Vinot 26948ed025 new: free text for UA, change order on capture page 2021-11-25 19:36:01 -05:00
Raphaël Vinot 58b50f2b24 new: Pass optional arbitrary HTTP headers to capture 2021-11-23 12:59:56 -08:00
Raphaël Vinot 4042ee9116 new: Add CORS config to submit 2021-11-08 16:47:39 -08:00
Raphaël Vinot c11e9d8110 chg: normalize tooltips on title style
Related: #279
2021-10-26 16:33:22 -04:00
Raphaël Vinot d9932610ae chg: Hide capture config, fix capture buttons, improve css
Related: #279
2021-10-26 13:31:48 -04:00
Raphaël Vinot b4771f63a7 new: config option to make capture private by default, better working on capture page
Related #279
2021-10-26 10:47:06 -04:00
Raphaël Vinot 9c9e26dde5 chg: Improve capture page
Related to #279
2021-10-25 17:19:58 -04:00
Raphaël Vinot 1f998b457f chg: use template 2021-10-18 13:06:43 +02:00
Raphaël Vinot 63638a154f chg: Bump d3js 2021-10-07 23:55:09 +02:00
Raphaël Vinot 1a0fe677ab fix: Copy bodies before sending them over.
send_file in flask close the BytesIO, causing issue if we need to
re-access it.
2021-09-29 15:00:10 +02:00
Raphaël Vinot 570e7e6e8a chg: Bump datatables (security fix) 2021-09-28 17:56:16 +02:00
Raphaël Vinot 88020028ac chg: Improve phishtank results rendering 2021-09-28 15:30:41 +02:00
Raphaël Vinot 708ff8194a chg: Bump d3js 2021-09-27 11:46:17 +02:00
Raphaël Vinot 6e9e3990c4 fix: Indexes not updated on tree rebuild, better handling of tree cache 2021-09-24 16:16:41 +02:00
Raphaël Vinot 48fc807e7d new: Add monitoring for pickle cache status 2021-09-24 12:02:28 +02:00
Raphaël Vinot 27836b5bdd fix: Skip phishtank if URLs and IPs are unknown (modules) 2021-09-23 14:22:21 +02:00
Raphaël Vinot 22315ac9b6 fix: Typo in module template 2021-09-23 14:14:14 +02:00
Raphaël Vinot 52f6d91fba fix: Skip block if urls not on phishtank. 2021-09-23 14:08:39 +02:00
Raphaël Vinot bf94dd6578 fix: missing endlist on template 2021-09-23 14:05:56 +02:00
Raphaël Vinot 87c6925c7b new: IPs lookup against phishtank 2021-09-23 13:58:40 +02:00
Raphaël Vinot 7f6a59a441 chg: Bump SRI for d3js 2021-09-20 22:42:51 +02:00
Raphaël Vinot 404f9a3112 fix: Avoid exception if none key 2021-09-17 10:06:59 +02:00
Raphaël Vinot e7d9c1cb1a fix: return error message if urlscan isn't able to run the capture. 2021-09-17 09:51:52 +02:00
Raphaël Vinot 963af434dd chg: Improve wording for Phishtank in modules 2021-09-16 17:35:45 +02:00
Raphaël Vinot cefb3fddbf new: Phishtank lookup. 2021-09-16 16:33:44 +02:00
Raphaël Vinot 77a94e017e chg: Bump datatables 2021-09-16 10:10:31 +02:00
Raphaël Vinot d1f673f3a7 chg: Cleanup passing listing key to and from bool in redis 2021-09-10 14:20:58 +02:00
Raphaël Vinot 39c7d14264 fix: make mypy happy 2021-09-08 12:50:56 +02:00
Raphaël Vinot cc4beeef35 fix: Pass listing (y/n) and proper referer to subsequent captures 2021-09-08 12:24:00 +02:00
Raphaël Vinot 902c8f81b6 chg: Improve error message if the capture fails
Fix #257
2021-09-07 18:16:01 +02:00
Raphaël Vinot dfbe40a52e chg: reorder imports 2021-09-07 16:00:07 +02:00
Raphaël Vinot 60f4a09d28 fix: Avoid "None" in URL field if no predefined URL is given. 2021-09-03 17:07:48 +02:00
Raphaël Vinot e213cb5ae8 fix: make mypy happy. 2021-09-03 17:05:26 +02:00
Raphaël Vinot dcfd9cc6a9 new: Re-capture a URL and change parameters. 2021-09-03 16:57:48 +02:00
Raphaël Vinot 7ea37ec617 fix: Avoid exception if capture finishes between two calls. 2021-09-01 16:57:20 +02:00
Raphaël Vinot c09adec333 chg: Improve logging. 2021-09-01 14:08:25 +02:00
Raphaël Vinot 7870e918de chg: Bump datatables 2021-08-30 15:26:50 +02:00
Raphaël Vinot d41b7735dd chg: Improve storage, support both modes. 2021-08-26 15:49:19 +02:00
Raphaël Vinot bf700e7a7b chg: Major refactoring, move capture code to external script. 2021-08-25 13:36:48 +02:00
Raphaël Vinot 81390d5ea0 chg: cleanup in the mail lookyloo class 2021-08-24 18:32:54 +02:00
Raphaël Vinot 725a43cc44 new: Trigger a capture with a GET request from a browser.
Fix #248
2021-08-18 10:53:32 +02:00
Raphaël Vinot 48d81815a0 chg: Bump d3js 2021-08-17 12:22:10 +02:00
Raphaël Vinot 021f53716b new: Submit a capture in a HTTP GET request
Fix #246
2021-08-17 12:12:23 +02:00
Raphaël Vinot 09a69622c0 fix: urlscan dislikes URLs without scheme. 2021-08-16 11:01:00 +02:00
Raphaël Vinot 9ff2bf8157 fix: avoid exception if submission is missing 2021-08-13 16:24:34 +02:00
Raphaël Vinot 50dbbd6eba new: [API] Trigger modules for a capture 2021-08-13 13:50:26 +02:00
Raphaël Vinot 92a466d78b fix: Improve legend
Fix #161
2021-08-12 16:31:53 +02:00
Raphaël Vinot 3436f5bd4e chg: Improve urlscan support, get results. 2021-08-11 15:26:12 +02:00
Raphaël Vinot 7933670941 new: Integration with urlscan.io 2021-08-10 17:38:47 +02:00
Raphaël Vinot 8668a15cd8 chg: use d3js v7 2021-07-21 11:56:38 +02:00
Raphaël Vinot 28e3162187 Merge branch 'restx' 2021-07-21 11:05:25 +02:00
Raphaël Vinot 9f580fc223 chg: Bump datatables 2021-07-14 10:55:52 +02:00
Raphaël Vinot cc2e5a4159 Merge branch 'main' into restx 2021-07-05 09:54:37 +02:00
Raphaël Vinot 6abdd2db9a new: Add referer on details view 2021-06-29 11:18:10 -07:00
felalex 6a7787d8af fix: error in case of invalid proxy + username/password support 2021-06-29 18:00:44 +02:00
felalex 1c89a80c7e fix: expected format & passing proxy in _capture 2021-06-25 16:36:57 +02:00
Raphaël Vinot 26cbf94977 Merge branch 'main' into restx 2021-06-22 16:55:41 -07:00
Raphaël Vinot 20fb7a70f9 new: Add referer on details view 2021-06-22 16:47:03 -07:00
felalex a04d224c79 new: trying toget proxy field from /capture 2021-06-22 18:39:14 +02:00
felalex 046dc4d1d1 new: added field for proxy 2021-06-18 17:12:50 +02:00
Raphaël Vinot 1a9d409d90 Merge branch 'main' into restx 2021-06-17 09:51:57 -07:00
Raphaël Vinot 14a3581fbc chg: Bump 3rd party libs 2021-06-16 18:28:57 -07:00
Raphaël Vinot cd8e76f26b fix: Properly use SRIs 2021-06-16 18:25:40 -07:00
Raphaël Vinot eba913caa0 fix: sort sri dump 2021-06-16 18:21:15 -07:00
Raphaël Vinot c89689374e new: SRI validation on resources 2021-06-16 17:36:01 -07:00
Raphaël Vinot a44b7b30b5 new: Get URLs and Hostnames from captures 2021-06-16 14:57:14 -07:00
Raphaël Vinot 7b32454f9a new: get all the ressources hashes in API 2021-06-15 18:26:41 -07:00
Raphaël Vinot 7053ca7994
Merge branch 'main' into restx 2021-06-11 12:12:59 -07:00
Raphaël Vinot 89026e8f3b chg: move mypy config to pyproject 2021-06-11 10:15:03 -07:00
Raphaël Vinot c52509614e new: get capture info API call 2021-06-09 17:59:24 -07:00
Raphaël Vinot 810cceb263 new: Add more calls to the API, cleanup 2021-06-07 15:37:11 -07:00
Raphaël Vinot e5b76b3a2f chg: Move API into a new file, cleanup 2021-06-07 13:12:23 -07:00
Raphaël Vinot 7bf0b78754 new: Use flask-restx for the API 2021-06-04 17:30:14 -07:00
Raphaël Vinot 387f2d3c3d chg: Improve MISP lookup 2021-06-02 13:27:50 -07:00
Raphaël Vinot 8c62a597e7 fix: Make mypy happy 2021-06-01 16:06:24 -07:00
Raphaël Vinot a1bbffd4ad fix: Missing template for MISP lookup 2021-06-01 15:35:35 -07:00
Raphaël Vinot 3071a1a7c9 new: Add MISP lookup 2021-06-01 15:31:14 -07:00
Raphaël Vinot 53ef253c94 chg: Improve MISP push 2021-05-31 13:27:25 -07:00
Raphaël Vinot 4fd7ee2568 chg: Make parent link more generic 2021-05-27 16:33:17 -07:00
Raphaël Vinot 53bc8324cd new: reference to parent on tree (if exists) 2021-05-27 15:23:04 -07:00
Raphaël Vinot 1117ab6371 chg: add stats, avoid building big trees twice, bump deps 2021-05-26 18:25:06 -07:00
Raphaël Vinot 315a2733c3 new: Tickbox to confirm notification 2021-05-26 12:07:47 -07:00
Raphaël Vinot 8a5b1f263e new: Fake field to avoid dumb bots 2021-05-25 13:20:28 -07:00
Raphaël Vinot 11f05626b5 chg: Improve module auto trigger 2021-05-19 15:03:07 -07:00
Raphaël Vinot 7b4f5bd6d2 new: feature to hide captures with error 2021-05-19 10:18:43 -07:00
Raphaël Vinot 925bb9d48e new: priority for captures 2021-05-18 14:58:56 -07:00
Raphaël Vinot 3614a47275 chg: Make mypy happy 2021-05-17 18:29:46 -07:00
Raphaël Vinot bb214d9e2b chg: Use keywords only paramaters when relevant 2021-05-17 17:08:43 -07:00
Raphaël Vinot 5ee62d157f chg: Properly handle capture parents, avoid duplicates 2021-05-14 14:25:37 -07:00
Raphaël Vinot b735cbeae3 chg: Reorganize styles 2021-05-12 14:13:44 -07:00
Raphaël Vinot 1d5925d755 new: Keep captures parent, use extends in MISP export 2021-05-12 13:30:07 -07:00
Raphaël Vinot a47615fb0a chg: Make mypy happy 2021-05-11 22:09:07 -07:00
Raphaël Vinot c07e3696cd fix: force size of img resources in popup, cleanup 2021-05-11 17:14:28 -07:00
Raphaël Vinot 986e581f5d fix: Move flashed messages around 2021-05-11 10:53:00 -07:00
Raphaël Vinot f5258a9904 fix: Put flashed message below the top menu 2021-05-10 18:28:56 -07:00
Raphaël Vinot 6592531ec1 fix: index search on partial URLs
Fix #202
2021-05-06 11:27:40 +02:00
Raphaël Vinot 64d1ed785b fix: Blur cropped screenshot if too big
Fix #196
2021-04-28 18:13:10 +02:00
Raphaël Vinot 12697964ce chg: Improve tree stats 2021-04-27 17:09:58 +02:00
Raphaël Vinot 3389dcafd6 new: Add support for uwhoisd 2021-04-26 00:52:08 +02:00
Raphaël Vinot 39e894b7b0 fix: body hash page on direct access 2021-04-20 17:32:22 +02:00
Raphaël Vinot 9952e4de32 fix: remove UA - IP mapping from redis 2021-04-18 17:58:16 +02:00
Raphaël Vinot be1ca8d088 new: Disable FLoC 2021-04-18 01:43:50 +02:00
Raphaël Vinot e8b19969fe chg: normalize capture button 2021-04-09 15:45:20 +02:00
Raphaël Vinot dcc7cd1351 fix: Typos 2021-04-09 14:34:20 +02:00
Raphaël Vinot 7707d638cf new: Use async capture for the UI.
Add a method to make sure splash is up before trying to capture.
2021-04-08 19:15:53 +02:00
Raphaël Vinot d78ee5de11 new: Link to UA page on wikipedia 2021-04-08 13:47:58 +02:00
Raphaël Vinot 4d23d7ac36 chg: Remove extra code, return unziped resources 2021-04-08 01:05:11 +02:00
Raphaël Vinot e79387ee0d chg: Remove unused entries in legend and stats 2021-04-08 00:35:10 +02:00
Raphaël Vinot 81acc8e2d5 chg: Improve UI (index and menu) 2021-04-06 23:34:57 +02:00
Raphaël Vinot ca0bd97689 new: Wait page when a capture is queued/ongoing 2021-04-01 18:51:42 +02:00
Raphaël Vinot b6ea1c24db new: Query capture status 2021-03-31 19:25:57 +02:00
Raphaël Vinot 55eab4af9c fix: Disable push button if a MISP event already exists. 2021-03-31 09:25:01 +02:00
Raphaël Vinot a8711a18f1 fix: Make mypy happy 2021-03-30 16:43:57 +02:00
Raphaël Vinot 9f88edd74d new: Check if an event with the same lookyloo capture is already on MISP
Fix #181
2021-03-30 16:32:04 +02:00
Raphaël Vinot 094b69f9b9 fix: Make cookie samesite only 2021-03-30 01:10:18 +02:00
Raphaël Vinot a2d6ff7b49 fix: scroll to screenshot, add margin 2021-03-30 00:45:24 +02:00
Raphaël Vinot 88bd450b59 chg: Improve tree page (menu and stats) 2021-03-25 18:32:48 +01:00
Raphaël Vinot 9845f60322 fix: Pass the UA in bulk captures 2021-03-19 22:29:13 +01:00
Raphaël Vinot 2a55461286 new: Capture an URL on the rendered page, keep the session (WiP) 2021-03-19 17:52:20 +01:00
Raphaël Vinot cd7b050cb0 chg: rename and cleanup methods 2021-03-18 18:47:54 +01:00
Raphaël Vinot 25c1a6dbce chg: Use sorted cache whenever possible 2021-03-18 15:39:55 +01:00
Raphaël Vinot a909346fc1 fix: mypy error 2021-03-18 00:50:42 +01:00
Raphaël Vinot 49ae4490b2 new: (hidden) interface to search for hostnames and URLs 2021-03-18 00:40:14 +01:00
Raphaël Vinot aec6fe2632 fix: Make mypy happy 2021-03-04 18:21:36 +01:00
Raphaël Vinot 3ed7608f16 chg: Improve ressources listing 2021-03-04 18:10:54 +01:00
Raphaël Vinot 4488a9a492 fix: do not fail if MISP is not working 2021-02-12 12:06:26 +01:00
Raphaël Vinot b91bbe5f07 new: Show preview of embeded ressources if it is an image. 2021-02-09 22:01:20 +01:00
Raphaël Vinot d711fd3644 fix: Make get legacy user config quiet 2021-02-05 15:24:11 +01:00