CIRCL
/
lookyloo
mirror of https://github.com/CIRCL/lookyloo
2
0
Fork 0
Code Issues Releases Wiki Activity
1,509 Commits
8 Branches
29 Tags
67 MiB
Commit Graph

458 Commits (lacus)

Author SHA1 Message Date
Raphaël Vinot d222ae04aa new: Keep capture even if we have a network error 2022-05-03 12:23:16 +02:00
Raphaël Vinot bd62d62392 chg: Add timeout for FOX module 2022-05-02 13:57:37 +02:00
Raphaël Vinot eaff6c81ee chg: add logging to FOX module 2022-05-02 13:11:03 +02:00
Raphaël Vinot 463d1d2d1a new: autosubmit to FOX, bump deps 2022-05-02 13:04:55 +02:00
Raphaël Vinot 163c35a614 fix: Avoid exception if bytes in query 2022-04-30 00:08:35 +02:00
Raphaël Vinot 16c40aea29 new: Add quick and dirty deduplicate if the same capture is sent multiple times within 300s 2022-04-29 21:55:59 +02:00
Raphaël Vinot cc17b414ff chg: Remove refs to splash 2022-04-25 14:43:02 +02:00
Raphaël Vinot 8d159ffba0 new: Switch away from splash to use playwright 2022-04-21 14:55:07 +03:00
Raphaël Vinot 756f34022f chg: Make mypy happy 2022-03-31 16:41:58 +02:00
Raphaël Vinot f489938b6b chg: Sync abstract manager with template 2022-03-31 16:34:25 +02:00
Raphaël Vinot cf46dde1ed chg: Add basic pre-hook config 2022-03-31 11:30:53 +02:00
Raphaël Vinot ae9cb3e81c chg: Bump deps 2022-03-29 21:13:02 +02:00
Raphaël Vinot 34d435e999 fix: [urlscan] create cache directory for response 2022-02-03 13:28:48 +01:00
Raphaël Vinot 43a29f5781 fix: [Modules] Missing change in helpers, fix urlscan cache storage 2022-02-03 12:49:16 +01:00
Raphaël Vinot afc77126d4 chg: [Modules] use same directory structure for cache in all modules. 2022-02-03 12:38:44 +01:00
Raphaël Vinot d1314aea0c fix: [Phishtank] Cache issue when the domain is an IP 2022-02-03 11:53:11 +01:00
Raphaël Vinot b80dc1aab3 chg: Make sure publishing the MISP event is done in the background 2021-12-20 16:50:29 +01:00
Raphaël Vinot 8cceb948f3 new: Get any supported hash algorithm over the API 2021-12-02 17:55:02 +01:00
Raphaël Vinot 649d7714cf chg: Bump pyhashlookup, fix typing 2021-12-01 15:44:45 +01:00
Raphaël Vinot 0835bcefa0 fix: remove empty dir with right call 2021-12-01 09:37:57 +01:00
Raphaël Vinot b75540cd7d chg: always return something in merge_hashlookup_tree 2021-11-30 15:51:48 +01:00
Raphaël Vinot fc4c342316 new: Add stats on hits on hashlookup 2021-11-30 15:43:42 +01:00
Raphaël Vinot 5e33ad6d0b chg: make mypy happy 2021-11-30 15:23:48 +01:00
Raphaël Vinot 8f4040b806 new: Hashlookup integration 2021-11-30 14:59:48 +01:00
Raphaël Vinot 5e5206ae00 fix: Catch exception if cookies are invalid 2021-11-26 14:04:07 -05:00
Raphaël Vinot 2a9ddabc5b fix: make mypy happy 2021-11-26 12:47:48 -05:00
Raphaël Vinot a4f238f33d new: Defang URLs in email notifications 
Fix #291
 2021-11-26 12:42:43 -05:00
Raphaël Vinot 4042ee9116 new: Add CORS config to submit 2021-11-08 16:47:39 -08:00
Raphaël Vinot 1f998b457f chg: use template 2021-10-18 13:06:43 +02:00
Raphaël Vinot 7caee08f76 chg: Bump deps, add logging. 2021-10-07 18:33:40 +02:00
Raphaël Vinot 80d2804f6c chg: Bump deps 2021-10-01 14:53:46 +02:00
Raphaël Vinot 8296502707 fix: Avoid exception if the capture happens at 0 microsec. 2021-09-30 15:38:25 +02:00
Raphaël Vinot 9215522563 fix: Avoid exception in case the DNS files are broken 2021-09-29 15:05:31 +02:00
Raphaël Vinot 1a0fe677ab fix: Copy bodies before sending them over. 
send_file in flask close the BytesIO, causing issue if we need to
re-access it.
 2021-09-29 15:00:10 +02:00
Raphaël Vinot 0da28cef28 chg: Move MISP export to module, add phishtank permaURL 2021-09-27 16:04:00 +02:00
Raphaël Vinot 681e136ef4 chg: More cleanup to use the new caching system as it should be. 2021-09-27 11:36:27 +02:00
Raphaël Vinot d05b1edf48 new: Add logging in Indexing 2021-09-24 16:27:46 +02:00
Raphaël Vinot 6e9e3990c4 fix: Indexes not updated on tree rebuild, better handling of tree cache 2021-09-24 16:16:41 +02:00
Raphaël Vinot 48fc807e7d new: Add monitoring for pickle cache status 2021-09-24 12:02:28 +02:00
Raphaël Vinot 1d3bb55343 chg: Rebuild trees when captures are moved. 2021-09-23 16:40:39 +02:00
Raphaël Vinot 87c6925c7b new: IPs lookup against phishtank 2021-09-23 13:58:40 +02:00
Raphaël Vinot e6e61089b6 chg: Speedup cache initialization 2021-09-23 10:29:02 +02:00
Raphaël Vinot aa7173d27d fix: a broken/missing pickle is handled elsewhere 2021-09-23 09:17:34 +02:00
Raphaël Vinot 19c8c54b0b fix: force-rebuild pickle if needed. 2021-09-22 22:23:20 +02:00
Raphaël Vinot 32ee474be2 chg: Improve tree creation and cache 2021-09-22 17:09:04 +02:00
Raphaël Vinot 01fff00cad chg: Slight cleanup 2021-09-21 18:04:22 +02:00
Raphaël Vinot bfcb4feb9a chg: Remove import 2021-09-17 15:10:21 +02:00
Raphaël Vinot 275b4aee93 fix: Avoid None in urlscan.io result. 2021-09-17 12:58:05 +02:00
Raphaël Vinot e7d9c1cb1a fix: return error message if urlscan isn't able to run the capture. 2021-09-17 09:51:52 +02:00
Raphaël Vinot cefb3fddbf new: Phishtank lookup. 2021-09-16 16:33:44 +02:00
Raphaël Vinot 9c552f1032 chg: Move modules into files 2021-09-16 11:22:02 +02:00
Raphaël Vinot 1d40dd44ae new: Add urlscan permanent URL to MISP export 2021-09-15 15:42:02 +02:00
Raphaël Vinot d6982392ef chg: cleanup get_user_agents 
Related: #262
 2021-09-13 22:32:02 +02:00
Raphaël Vinot 22d8672131 chg: Catch timeouts when pushing to MISP 2021-09-10 15:25:37 +02:00
Raphaël Vinot d1f673f3a7 chg: Cleanup passing listing key to and from bool in redis 2021-09-10 14:20:58 +02:00
Raphaël Vinot 5abc264159 fix: Better initialization of cache, avoid missing the error message. 2021-09-08 16:34:54 +02:00
Raphaël Vinot 902c8f81b6 chg: Improve error message if the capture fails 
Fix #257
 2021-09-07 18:16:01 +02:00
Raphaël Vinot 048e44ce91 fix: Avoid exception in _set_capture_cache in case of critical error 2021-09-07 16:00:07 +02:00
Raphaël Vinot dfbe40a52e chg: reorder imports 2021-09-07 16:00:07 +02:00
Raphaël Vinot 7d76d96ff7 fix: Avoid 500 error on the website if unable to dump pickle. 2021-09-06 16:30:43 +02:00
Raphaël Vinot 074bb675f6 fix: Handling of status code 0, export capture UUID 2021-09-06 16:12:41 +02:00
Raphaël Vinot c09adec333 chg: Improve logging. 2021-09-01 14:08:25 +02:00
Raphaël Vinot 72d85c62e3 fix: Use redis ConnectionError to avoid exception on shutdown 2021-09-01 12:44:57 +02:00
Raphaël Vinot 6db708bb93 chg: Avoid exception if the backend is already down 2021-08-31 15:43:55 +02:00
Raphaël Vinot 9042b73e69 chg: Remove dead code, proper use of exceptions. 2021-08-31 13:48:19 +02:00
Raphaël Vinot 117500b777 chg: Make archiver an index generator 2021-08-30 12:48:13 +02:00
Raphaël Vinot 1bff8f1529 fix: cast capture dir to string for redis 2021-08-28 20:36:50 +02:00
Raphaël Vinot 8a51383d7a chg: Move the process managment methods to the proper class 2021-08-27 17:28:26 +02:00
Raphaël Vinot 85e43fc677 chg: Make the website start a normal start script 2021-08-27 16:45:16 +02:00
Raphaël Vinot 8918b11a7f chg: The Tech Blog UAs can only be fetched manually, moving it to tools 2021-08-27 15:22:39 +02:00
Raphaël Vinot d41b7735dd chg: Improve storage, support both modes. 2021-08-26 15:49:19 +02:00
Raphaël Vinot 407e78ae7f chg: More cleanup, support clean shutdown of multiple async captures 2021-08-25 16:40:51 +02:00
Raphaël Vinot bc5e41a1ca fix: Properly re-set the cache if the tree was not built 2021-08-25 15:00:26 +02:00
Raphaël Vinot bf700e7a7b chg: Major refactoring, move capture code to external script. 2021-08-25 13:36:48 +02:00
Raphaël Vinot 81390d5ea0 chg: cleanup in the mail lookyloo class 2021-08-24 18:32:54 +02:00
Raphaël Vinot 8433cbcc1b chg: Cleanup archiver, initialize index captures in start 2021-08-24 17:10:14 +02:00
Raphaël Vinot 53ceb9c329 chg: Cleanup when dir is moved, digit months on 2 values 2021-08-23 14:53:19 +02:00
Raphaël Vinot b8109c6462 fix: Make mypy happy 2021-08-23 14:20:26 +02:00
Raphaël Vinot d359bc7521 chg: Better use of cache, sanity checks 2021-08-23 12:17:44 +02:00
Raphaël Vinot 58b837cb6c new: Archiver, refactoring. 2021-08-20 17:46:22 +02:00
Raphaël Vinot 6be9b69d95 chg: Use connection pool whenever possible 2021-08-18 18:01:04 +02:00
Raphaël Vinot 7495e5b4f1 chg: Use connection pool in indexing 2021-08-18 17:31:17 +02:00
Raphaël Vinot 046387a9f4 chg: Make mypy happy 2021-08-18 16:37:50 +02:00
Raphaël Vinot 3a96e27f8c chg: Try using redis connection pool 2021-08-18 16:07:39 +02:00
Raphaël Vinot 09a69622c0 fix: urlscan dislikes URLs without scheme. 2021-08-16 11:01:00 +02:00
Raphaël Vinot 50dbbd6eba new: [API] Trigger modules for a capture 2021-08-13 13:50:26 +02:00
Raphaël Vinot 3436f5bd4e chg: Improve urlscan support, get results. 2021-08-11 15:26:12 +02:00
Raphaël Vinot 7933670941 new: Integration with urlscan.io 2021-08-10 17:38:47 +02:00
Raphaël Vinot 28e3162187 Merge branch 'restx' 2021-07-21 11:05:25 +02:00
Raphaël Vinot 59f2a510c0 fix: properly catch broken capture, bump deps 2021-07-14 11:34:10 +02:00
Raphaël Vinot cc2e5a4159 Merge branch 'main' into restx 2021-07-05 09:54:37 +02:00
Raphaël Vinot 06aa85cfe8 fix: Do not return the whole database if the list of captures to sort is empty. 
Related #227
 2021-06-29 13:48:13 -07:00
felalex d3e0b7f394 new: proxy parameter in crawl 2021-06-25 19:11:15 +02:00
felalex 1c89a80c7e fix: expected format & passing proxy in _capture 2021-06-25 16:36:57 +02:00
Raphaël Vinot 26cbf94977 Merge branch 'main' into restx 2021-06-22 16:55:41 -07:00
Raphaël Vinot a44b7b30b5 new: Get URLs and Hostnames from captures 2021-06-16 14:57:14 -07:00
Raphaël Vinot 7ad9aa7bbc Merge branch 'main' into restx 2021-06-15 17:24:19 -07:00
Raphaël Vinot 4a8db1fc6e fix: re-cache captures with incomplete redirects, code cleanup 2021-06-15 16:03:33 -07:00
Raphaël Vinot b6a636697f chg: Improve memory use CaptureCache 2021-06-15 15:16:56 -07:00
Raphaël Vinot c52509614e new: get capture info API call 2021-06-09 17:59:24 -07:00