CIRCL
/
lookyloo
mirror of https://github.com/CIRCL/lookyloo
2
0
Fork 0
Code Issues Releases Wiki Activity
1,376 Commits
8 Branches
31 Tags
73 MiB
Commit Graph

411 Commits (942a2e3b38dad98c77785577ddfe126c955345e3)

Author SHA1 Message Date
Raphaël Vinot 275b4aee93 fix: Avoid None in urlscan.io result. 2021-09-17 12:58:05 +02:00
Raphaël Vinot e7d9c1cb1a fix: return error message if urlscan isn't able to run the capture. 2021-09-17 09:51:52 +02:00
Raphaël Vinot cefb3fddbf new: Phishtank lookup. 2021-09-16 16:33:44 +02:00
Raphaël Vinot 9c552f1032 chg: Move modules into files 2021-09-16 11:22:02 +02:00
Raphaël Vinot 1d40dd44ae new: Add urlscan permanent URL to MISP export 2021-09-15 15:42:02 +02:00
Raphaël Vinot d6982392ef chg: cleanup get_user_agents 
Related: #262
 2021-09-13 22:32:02 +02:00
Raphaël Vinot 22d8672131 chg: Catch timeouts when pushing to MISP 2021-09-10 15:25:37 +02:00
Raphaël Vinot d1f673f3a7 chg: Cleanup passing listing key to and from bool in redis 2021-09-10 14:20:58 +02:00
Raphaël Vinot 5abc264159 fix: Better initialization of cache, avoid missing the error message. 2021-09-08 16:34:54 +02:00
Raphaël Vinot 902c8f81b6 chg: Improve error message if the capture fails 
Fix #257
 2021-09-07 18:16:01 +02:00
Raphaël Vinot 048e44ce91 fix: Avoid exception in _set_capture_cache in case of critical error 2021-09-07 16:00:07 +02:00
Raphaël Vinot dfbe40a52e chg: reorder imports 2021-09-07 16:00:07 +02:00
Raphaël Vinot 7d76d96ff7 fix: Avoid 500 error on the website if unable to dump pickle. 2021-09-06 16:30:43 +02:00
Raphaël Vinot 074bb675f6 fix: Handling of status code 0, export capture UUID 2021-09-06 16:12:41 +02:00
Raphaël Vinot c09adec333 chg: Improve logging. 2021-09-01 14:08:25 +02:00
Raphaël Vinot 72d85c62e3 fix: Use redis ConnectionError to avoid exception on shutdown 2021-09-01 12:44:57 +02:00
Raphaël Vinot 6db708bb93 chg: Avoid exception if the backend is already down 2021-08-31 15:43:55 +02:00
Raphaël Vinot 9042b73e69 chg: Remove dead code, proper use of exceptions. 2021-08-31 13:48:19 +02:00
Raphaël Vinot 117500b777 chg: Make archiver an index generator 2021-08-30 12:48:13 +02:00
Raphaël Vinot 1bff8f1529 fix: cast capture dir to string for redis 2021-08-28 20:36:50 +02:00
Raphaël Vinot 8a51383d7a chg: Move the process managment methods to the proper class 2021-08-27 17:28:26 +02:00
Raphaël Vinot 85e43fc677 chg: Make the website start a normal start script 2021-08-27 16:45:16 +02:00
Raphaël Vinot 8918b11a7f chg: The Tech Blog UAs can only be fetched manually, moving it to tools 2021-08-27 15:22:39 +02:00
Raphaël Vinot d41b7735dd chg: Improve storage, support both modes. 2021-08-26 15:49:19 +02:00
Raphaël Vinot 407e78ae7f chg: More cleanup, support clean shutdown of multiple async captures 2021-08-25 16:40:51 +02:00
Raphaël Vinot bc5e41a1ca fix: Properly re-set the cache if the tree was not built 2021-08-25 15:00:26 +02:00
Raphaël Vinot bf700e7a7b chg: Major refactoring, move capture code to external script. 2021-08-25 13:36:48 +02:00
Raphaël Vinot 81390d5ea0 chg: cleanup in the mail lookyloo class 2021-08-24 18:32:54 +02:00
Raphaël Vinot 8433cbcc1b chg: Cleanup archiver, initialize index captures in start 2021-08-24 17:10:14 +02:00
Raphaël Vinot 53ceb9c329 chg: Cleanup when dir is moved, digit months on 2 values 2021-08-23 14:53:19 +02:00
Raphaël Vinot b8109c6462 fix: Make mypy happy 2021-08-23 14:20:26 +02:00
Raphaël Vinot d359bc7521 chg: Better use of cache, sanity checks 2021-08-23 12:17:44 +02:00
Raphaël Vinot 58b837cb6c new: Archiver, refactoring. 2021-08-20 17:46:22 +02:00
Raphaël Vinot 6be9b69d95 chg: Use connection pool whenever possible 2021-08-18 18:01:04 +02:00
Raphaël Vinot 7495e5b4f1 chg: Use connection pool in indexing 2021-08-18 17:31:17 +02:00
Raphaël Vinot 046387a9f4 chg: Make mypy happy 2021-08-18 16:37:50 +02:00
Raphaël Vinot 3a96e27f8c chg: Try using redis connection pool 2021-08-18 16:07:39 +02:00
Raphaël Vinot 09a69622c0 fix: urlscan dislikes URLs without scheme. 2021-08-16 11:01:00 +02:00
Raphaël Vinot 50dbbd6eba new: [API] Trigger modules for a capture 2021-08-13 13:50:26 +02:00
Raphaël Vinot 3436f5bd4e chg: Improve urlscan support, get results. 2021-08-11 15:26:12 +02:00
Raphaël Vinot 7933670941 new: Integration with urlscan.io 2021-08-10 17:38:47 +02:00
Raphaël Vinot 28e3162187 Merge branch 'restx' 2021-07-21 11:05:25 +02:00
Raphaël Vinot 59f2a510c0 fix: properly catch broken capture, bump deps 2021-07-14 11:34:10 +02:00
Raphaël Vinot cc2e5a4159 Merge branch 'main' into restx 2021-07-05 09:54:37 +02:00
Raphaël Vinot 06aa85cfe8 fix: Do not return the whole database if the list of captures to sort is empty. 
Related #227
 2021-06-29 13:48:13 -07:00
felalex d3e0b7f394 new: proxy parameter in crawl 2021-06-25 19:11:15 +02:00
felalex 1c89a80c7e fix: expected format & passing proxy in _capture 2021-06-25 16:36:57 +02:00
Raphaël Vinot 26cbf94977 Merge branch 'main' into restx 2021-06-22 16:55:41 -07:00
Raphaël Vinot a44b7b30b5 new: Get URLs and Hostnames from captures 2021-06-16 14:57:14 -07:00
Raphaël Vinot 7ad9aa7bbc Merge branch 'main' into restx 2021-06-15 17:24:19 -07:00
Raphaël Vinot 4a8db1fc6e fix: re-cache captures with incomplete redirects, code cleanup 2021-06-15 16:03:33 -07:00
Raphaël Vinot b6a636697f chg: Improve memory use CaptureCache 2021-06-15 15:16:56 -07:00
Raphaël Vinot c52509614e new: get capture info API call 2021-06-09 17:59:24 -07:00
Raphaël Vinot 3d2c7420a4 chg: Bump mypy 2021-06-09 12:12:01 -07:00
Raphaël Vinot cf65489cef chg: Bump mypy 2021-06-09 12:04:26 -07:00
Raphaël Vinot 88eb85df46 fix: Avoid exception if rendered page is missing on broken tree 2021-06-03 19:07:02 -07:00
Raphaël Vinot 1ae02e0dea chg: Add body hash and domains in MISP lookup 2021-06-03 18:40:06 -07:00
Raphaël Vinot 387f2d3c3d chg: Improve MISP lookup 2021-06-02 13:27:50 -07:00
Raphaël Vinot f204947afa fix: Missing rendered node in MISP lookup 2021-06-01 16:22:07 -07:00
Raphaël Vinot 8c62a597e7 fix: Make mypy happy 2021-06-01 16:06:24 -07:00
Raphaël Vinot 3071a1a7c9 new: Add MISP lookup 2021-06-01 15:31:14 -07:00
Raphaël Vinot 53ef253c94 chg: Improve MISP push 2021-05-31 13:27:25 -07:00
Raphaël Vinot 17af2df5cd chg: Always make sure the lock file is removed even if building the tree fails 2021-05-28 17:28:26 -07:00
Raphaël Vinot 1117ab6371 chg: add stats, avoid building big trees twice, bump deps 2021-05-26 18:25:06 -07:00
Raphaël Vinot 11f05626b5 chg: Improve module auto trigger 2021-05-19 15:03:07 -07:00
Raphaël Vinot 896f3c5f13 fix: properly check if capture is in the queue 2021-05-18 16:04:13 -07:00
Raphaël Vinot 925bb9d48e new: priority for captures 2021-05-18 14:58:56 -07:00
Raphaël Vinot bb214d9e2b chg: Use keywords only paramaters when relevant 2021-05-17 17:08:43 -07:00
Raphaël Vinot 5ee62d157f chg: Properly handle capture parents, avoid duplicates 2021-05-14 14:25:37 -07:00
Raphaël Vinot 1d5925d755 new: Keep captures parent, use extends in MISP export 2021-05-12 13:30:07 -07:00
Fafner [_KeyZee_] 97207b8f77
solving tor proxy for only_global_lookups 2021-05-04 17:17:21 +02:00
Raphaël Vinot 12697964ce chg: Improve tree stats 2021-04-27 17:09:58 +02:00
Raphaël Vinot 3389dcafd6 new: Add support for uwhoisd 2021-04-26 00:52:08 +02:00
Raphaël Vinot 39e894b7b0 fix: body hash page on direct access 2021-04-20 17:32:22 +02:00
Raphaël Vinot 9952e4de32 fix: remove UA - IP mapping from redis 2021-04-18 17:58:16 +02:00
Raphaël Vinot bf5d15baf0 fix: remove capture from index 2021-04-14 18:05:00 +02:00
Raphaël Vinot d7665e0a5e chg: Make mypy happy 2021-04-09 14:48:42 +02:00
Raphaël Vinot f865ec912a fix: Move set/unset running to abstract 
Avoid issues when a script fails unexpectedly.
 2021-04-09 14:33:42 +02:00
Raphaël Vinot 9470b0c738 fix: remove broken pickles 2021-04-09 10:44:37 +02:00
Raphaël Vinot 7707d638cf new: Use async capture for the UI. 
Add a method to make sure splash is up before trying to capture.
 2021-04-08 19:15:53 +02:00
Raphaël Vinot 4d23d7ac36 chg: Remove extra code, return unziped resources 2021-04-08 01:05:11 +02:00
Raphaël Vinot e492428bf5 fix: Proper call to local cache 2021-04-03 02:24:38 +02:00
Raphaël Vinot 82db729e96 fix: cache captures index in main script 2021-04-03 02:06:32 +02:00
Raphaël Vinot b6ea1c24db new: Query capture status 2021-03-31 19:25:57 +02:00
Raphaël Vinot a8711a18f1 fix: Make mypy happy 2021-03-30 16:43:57 +02:00
Raphaël Vinot 9f88edd74d new: Check if an event with the same lookyloo capture is already on MISP 
Fix #181
 2021-03-30 16:32:04 +02:00
Raphaël Vinot acb1bf2814 fix: MISP export when rendered node is empty 
Fix #182
 2021-03-30 14:06:34 +02:00
Raphaël Vinot 88bd450b59 chg: Improve tree page (menu and stats) 2021-03-25 18:32:48 +01:00
Raphaël Vinot fa6b4701c0 chg: update the cache at the right place. 2021-03-20 21:54:46 +01:00
Raphaël Vinot b1fbf9ff74 fix: Properly type capture_dir in cache 2021-03-20 01:18:28 +01:00
Raphaël Vinot 2a55461286 new: Capture an URL on the rendered page, keep the session (WiP) 2021-03-19 17:52:20 +01:00
Raphaël Vinot cd7b050cb0 chg: rename and cleanup methods 2021-03-18 18:47:54 +01:00
Raphaël Vinot 25c1a6dbce chg: Use sorted cache whenever possible 2021-03-18 15:39:55 +01:00
Raphaël Vinot 49ae4490b2 new: (hidden) interface to search for hostnames and URLs 2021-03-18 00:40:14 +01:00
Raphaël Vinot 99402efa42 fix: allow to dump very big tree 2021-03-16 14:09:30 +01:00
Raphaël Vinot 1f9f5f1a9a chg: Improve url and hostnames search 2021-03-16 13:35:59 +01:00
Raphaël Vinot 8c656062b9 fix: Catch exception when the tree has too many redirects. 2021-03-12 18:20:17 +01:00
Raphaël Vinot 46aea0fe3a chg: Better error handling 2021-03-12 16:52:12 +01:00
Raphaël Vinot e90fc151bd fix: Avoid indexing URLs multiple times 2021-03-12 13:18:48 +01:00
Raphaël Vinot 4533bbdfa2 chg: Make MyPy happy 2021-03-08 14:03:03 +01:00