CIRCL
/
lookyloo
mirror of https://github.com/CIRCL/lookyloo
2
0
Fork 0
Code Issues Releases Wiki Activity
2,279 Commits
8 Branches
29 Tags
68 MiB
Commit Graph

706 Commits (main)

Author SHA1 Message Date
Raphaël Vinot 68204c2686 new: Support for rendered nodes with HTML and downloaded file 2023-09-26 18:08:04 +02:00
Raphaël Vinot 5d5b93266d new: optionally ignore IPs when comparing nodes 
Related https://github.com/Lookyloo/monitoring/issues/23
 2023-09-12 14:03:02 +02:00
Raphaël Vinot 0365c9d6c9 fix: Strip all non-URL characters from capture. 2023-09-11 16:54:50 +02:00
Raphaël Vinot 00bf8fb551 fix: Improve getting MISP instances 2023-08-29 17:30:45 +02:00
Raphaël Vinot 6034d10834 new: (WiP) Add support for multiple MISPs 2023-08-29 14:03:01 +02:00
Raphaël Vinot 72ba77377d chg: Catch empty file hash and return empry file immediately 2023-08-24 15:12:38 +02:00
Raphaël Vinot ff15f3a60a Avoid exceptions when trying to get a datablob 2023-08-24 14:55:32 +02:00
Raphaël Vinot 8b316ad185 fix: Properly search hash in embedded resources 2023-08-24 14:37:03 +02:00
Raphaël Vinot 2a843ac828 chg: Delist less captures, avoid edge case causing an exception for MISP export 2023-08-23 17:22:46 +02:00
Raphaël Vinot 447229ced3 chg: Compress HARs by default, update codebase accordingly 2023-08-11 13:16:59 +02:00
Raphaël Vinot 1f517b1562 chg: Cosmetic changes to support upcomming ete4 2023-08-10 17:31:19 +02:00
Raphaël Vinot 206e5957b5 new: Support for favicons fetching and display 
Related https://github.com/Lookyloo/PlaywrightCapture/issues/45
 2023-08-09 16:50:33 +02:00
Raphaël Vinot e256a7fe6b chg: Proper use of shutil.move, speedup initialization of CaptureCache 2023-08-08 12:41:21 +02:00
Raphaël Vinot 54674f6c5b avoid exception if pickle needs to be rebuild 2023-08-04 13:16:05 +02:00
Raphaël Vinot 35218900ed fix: MISP export when the capture is a file. 2023-07-30 17:48:16 +02:00
Raphaël Vinot 182e7af188 chg: Attempt to keep index up-to-date with cached captures 2023-07-28 14:40:33 +02:00
Raphaël Vinot c0f601e5db chg: remove incomplete_redirects from cache, was always 0/False 2023-07-28 14:05:28 +02:00
Raphaël Vinot ea2ded9beb fix: properly handle missing title in cache 2023-07-27 15:21:06 +02:00
Raphaël Vinot ebfc2f00a5 fix: Exception when a formerly broken capture is re-processed and works 2023-07-27 14:56:39 +02:00
Raphaël Vinot db639d9dde chg: Improve logging for errors in cookies, bump har2tree 2023-07-27 12:43:52 +02:00
Raphaël Vinot 855485984f fix: handle gracefully empty lists in hset, and duplicates UUIDs 2023-07-26 22:16:00 +02:00
Raphaël Vinot fd9325bb0d chg: Improve logging, add lock on indexer. 2023-07-26 12:37:12 +02:00
Raphaël Vinot ebd2b29993 fix: export compress HAR 2023-07-26 11:28:32 +02:00
Raphaël Vinot ef3432cbed fix: Few more improvments on lockfile and broken captures. 2023-07-25 20:16:48 +02:00
Raphaël Vinot 484aec5ddd fix: Properly handle lock file. 2023-07-25 19:29:53 +02:00
Raphaël Vinot 0c7b3d9106 fix: indexer getting stuck when we had more than one at a time 2023-07-25 17:08:00 +02:00
Raphaël Vinot 18b5a51c86 fix: Exception when none of the captures are cached. 2023-07-24 22:41:11 +02:00
Raphaël Vinot 0604ccfdee new: Improve the HHH details page 2023-07-24 14:53:00 +02:00
Raphaël Vinot 8cd6f5490a fix: Properly cache URL, fix mypy issue 
Related https://github.com/Lookyloo/PyLookyloo/issues/32
 2023-07-24 13:15:13 +02:00
Raphaël Vinot 36f9d3dc60 chg: Improve error reporting 
Related https://github.com/Lookyloo/PyLookyloo/issues/32
 2023-07-24 12:48:57 +02:00
Raphaël Vinot 177474e874 new: Basic support for HHHash 2023-07-21 15:48:20 +02:00
Raphaël Vinot b3000d02dd fix: 🤷 2023-07-13 16:59:52 +02:00
Raphaël Vinot f3095c7ccf chg: Add logging when comparing is impossible due to a missing pickle 2023-07-10 12:27:43 +02:00
Raphaël Vinot 3193763c8a fix: logic issue in last commit 2023-07-06 14:59:24 +02:00
Raphaël Vinot 79c9b95faf fix: Avoid issue if a cature is somewhat broken and we want to send a notification 2023-07-06 14:50:38 +02:00
Raphaël Vinot 2c5c30006c fix: index page stuck building captures for no good reason 2023-07-04 11:22:36 +02:00
Raphaël Vinot 6bb660ed82 fix: incorrect pickle rebuild, causing the index to get stuck 2023-07-03 17:04:55 +02:00
Raphaël Vinot 95a17f5770 fix: Handling of captures with errors 
Related https://github.com/Lookyloo/monitoring/issues/15
 2023-06-21 15:41:14 +02:00
Raphaël Vinot 475b424878 chg: retry a few times if remote lacus isn't available immediately 2023-06-16 12:47:50 +02:00
Raphaël Vinot 324c85cac8 new: Support for STARTTLS in the notification 2023-06-14 16:19:46 +02:00
Raphaël Vinot 0505edff0e new: Global proxy setting. 
Overwrites any proxy given by the user. Still uses the tor proxy when
needed.

Related: https://github.com/Lookyloo/lookyloo/issues/318
 2023-06-13 12:07:26 +02:00
Raphaël Vinot 97a668aa91 chg: do not export pickle 2023-06-12 16:01:30 +02:00
Raphaël Vinot 4ff9b7651d new: Support for TZ, Geo, locale, color scheme 2023-06-07 15:05:40 +02:00
Raphaël Vinot 474ab91dcd chg: Use URL from capture settings file 2023-06-05 15:21:22 +02:00
Raphaël Vinot b3b7d4f5c6 new: Add lookyloo permaurls in comparator response 2023-05-17 12:14:24 +02:00
Raphaël Vinot f8acca5e51 fix: properly ignore resources if only hostname matches 2023-05-17 11:14:53 +02:00
Raphaël Vinot 386aa6e91b fix: make the sets into tuples so endswith works 2023-05-17 09:53:20 +02:00
Raphaël Vinot 95f05a3da1 new: Display proxy if passed to the CaptureSettings 
Fix #695
 2023-05-15 16:33:49 +02:00
Raphaël Vinot 582b5956e9 new: Store capture settings, use TypedDict whenever possible. 2023-05-15 16:08:19 +02:00
Raphaël Vinot a504facf5e new: Flag to quickly say if two captures are different or not 2023-05-11 15:07:37 +02:00
Raphaël Vinot 14d4e0f2db fix: MISP lookup from Lookyloo. 2023-05-04 10:20:54 +02:00
Raphaël Vinot 6a9bcc0050 new: Automatic reporting via API 
Related to #678
 2023-04-28 17:19:53 +02:00
Raphaël Vinot c62032fee2 new: Endpoint to get comparables from a capture 2023-04-24 16:25:29 +02:00
Raphaël Vinot b33ef171a4 new: Method to get comparable stuff out of tree 2023-04-21 23:57:23 +02:00
Raphaël Vinot 88997e7c99 fix: Incorrect call to new PSL lib 2023-04-20 16:23:56 +03:00
Raphaël Vinot 6754400e25 fix: [URLHaus] Do not store responses if the URL is invalid 2023-04-13 16:17:58 +02:00
Raphaël Vinot c7d3e54edf fix: Cloudflare module always returning True 2023-04-12 14:05:05 +02:00
Raphaël Vinot 4ceae60db7 chg: Avoid stopping the captures before they're done 2023-04-09 13:58:34 +02:00
Raphaël Vinot 2ceda75eab chg: Fairly big refactoring/cleanup to support LacusCore 1.4.0 2023-04-08 13:49:18 +02:00
Raphaël Vinot 9995371916 chg: Normalize logging on the config file settings 2023-04-05 16:23:46 +02:00
Raphaël Vinot 9281d73675 fix: Properly handle recursion error when storing pickle 2023-03-30 16:30:27 +02:00
Raphaël Vinot 8122b9904c fix: remove gziped pickle too on error. 2023-03-30 15:50:13 +02:00
Raphaël Vinot c48bf2f4de new: Pass compare settings (WiP) 2023-03-24 15:47:41 +01:00
Raphaël Vinot 046417c058 chg: Bump pysecurity.txt, pass Lookyloos UA 2023-03-21 11:28:08 +01:00
Raphaël Vinot a0a23dca20 chg: Bump har2tree 2023-03-16 17:32:14 +01:00
Raphaël Vinot d970c924f8 chg: Simplify tracing the exceptions related to a specific capture. 2023-03-16 13:49:22 +01:00
Raphaël Vinot afd383cfc3 fix: Better error handling with unpickling, improve logging 2023-03-16 12:45:58 +01:00
Raphaël Vinot 9497060028 fix: Cleanup prints, improve archiver. 2023-03-16 12:28:28 +01:00
Raphaël Vinot 0b678e2db0 new: Optimize pickle, store as gzip to reduce space. 2023-03-15 18:02:33 +01:00
Raphaël Vinot 07c365e3bb chg: Improve logging 2023-03-12 19:25:16 +01:00
Raphaël Vinot 25433b5707 chg: always use LookylooException 2023-03-12 19:25:16 +01:00
Raphaël Vinot 3927b97584 fix: let the module figure out the mimetype for the attachment 2023-03-09 13:17:10 +02:00
Raphaël Vinot 53d9a6ba7a new: Add contacts as attachmetn to notification mail 2023-03-09 12:55:45 +02:00
Raphaël Vinot 4a33e10cc9 chg: force protocol 5 for pickles 2023-03-08 16:55:18 +02:00
Raphaël Vinot 7a96ae9767 new: Fetch the exiting collection names from the monitoring interface 2023-03-02 20:51:46 +01:00
Raphaël Vinot cf25da5646 new: Show the monitoring settings on the modal 2023-02-27 16:01:46 +01:00
Raphaël Vinot 09f788d511 Merge branch 'main' into monitoring 2023-02-26 17:32:35 +01:00
Raphaël Vinot 36d39f6076 new: Add PID in lock file, allows to check if the locking process is still there 2023-02-26 17:20:17 +01:00
Raphaël Vinot 11fed3f46a fix: Remove print 2023-02-23 18:49:59 +01:00
Raphaël Vinot 5e72e1104e new: Basic support for monitoring 2023-02-23 18:37:40 +01:00
Raphaël Vinot 370850285d fix: exception when cloudflare lookup is unavailable 2023-02-23 16:55:24 +01:00
Raphaël Vinot 1558f65c8e new: try to get contact info from security.txt file if available. 2023-02-14 13:32:00 +01:00
Raphaël Vinot 2f2fd6024b fix: Some nodes do not have IPs, can't always use it for comparaison 2023-02-09 15:39:40 +01:00
Raphaël Vinot aae3d56ddc new: Aggregate all emails by URL 2023-02-03 11:32:13 +01:00
Raphaël Vinot f6a12d50ea new: very basic takedown information gatherer 2023-02-02 15:18:39 +01:00
Raphaël Vinot 01a4b4d632 chg: Optionally not-defang the URLs in the email notification 2023-02-02 15:18:39 +01:00
Raphaël Vinot 599ae9f9f3 new: Compare ressources URLs 2023-01-31 11:24:16 +01:00
Raphaël Vinot 9273b80602 fix: Cleanup cloudflare module. 2023-01-30 16:26:46 +01:00
Raphaël Vinot ecb4623b86 new: Cloudflare lookup feature to flag IPs appropriately 2023-01-30 16:07:01 +01:00
Raphaël Vinot 0cb58d17d9 new: Add status code in comparator 2023-01-24 15:59:47 +01:00
Raphaël Vinot eab4b17070 fix: Avoid exceptions when the submission is a file 2023-01-24 15:59:06 +01:00
Raphaël Vinot f335b4a58d fix: Use publicsuffixlist instead of publicsuffix2 2023-01-20 15:23:11 +01:00
Raphaël Vinot d449edeb23 new: Allow API user to force re-caching the captures on URL/Hostname request 2023-01-20 11:15:33 +01:00
Raphaël Vinot 190d5a7118 new: Very basic capture comparator, API only. 2023-01-18 16:31:12 +01:00
Raphaël Vinot fdf1069c76 fix: Properly load IPs in old format. 2023-01-16 17:22:23 +01:00
Raphaël Vinot a3c62d1682 fix: Issue when loading IPs 2023-01-16 17:00:11 +01:00
Raphaël Vinot 1e563ff345 chg: Improve DNS resolution 2023-01-16 15:50:10 +01:00
Raphaël Vinot ae59b59071 fix: improper concat for sets 2023-01-16 15:16:40 +01:00
Raphaël Vinot 6624aec1bd new: run DNS lookups on AAAA too 2023-01-16 15:10:10 +01:00
Raphaël Vinot 55d727792a fix: Add logger in capture cache 2022-12-31 12:19:42 +01:00
Raphaël Vinot c7aba15ee2 fix: Avoid exception if the redirects key in cache is empty 2022-12-31 12:16:50 +01:00
Raphaël Vinot 5200402e06 chg: Another quick deps bump, prepare v1.17.0 2022-12-29 14:07:27 +01:00
Raphaël Vinot 01b9f8d1be fix: remove print, replace with logging 2022-12-27 01:55:18 +01:00
Raphaël Vinot 02497e167c fix: Catch QuotaExceededError in VT module 2022-12-26 23:30:00 +01:00
Raphaël Vinot 1abd97f37a new: switch to return emails only on whois query 2022-12-26 23:01:44 +01:00
Raphaël Vinot 479fcb39e5 new: Basic support for IPASN lookup 2022-12-23 16:36:11 +01:00
Raphaël Vinot ca7ec64ffe fix: avoid exception if the tree cannot be build 2022-12-19 17:42:14 +01:00
Raphaël Vinot 0a9003f58e chg: Use cache whenever possible 2022-12-08 11:57:45 +01:00
Raphaël Vinot 1ae76ddea7 chg: Use cache instead of pickle for meta info 2022-12-08 10:49:26 +01:00
Raphaël Vinot 2c6d6b2556 chg: Get UUIDs sorted from most recent to oldest. 2022-12-07 21:18:38 +01:00
Raphaël Vinot 8083c8cabc fix: Major improvement in index display when you break the cache... 2022-12-07 21:05:27 +01:00
Raphaël Vinot 8c77b9372b fix: do not re-cache captures with errors. 2022-12-07 15:30:19 +01:00
Raphaël Vinot d2b5605182 fix: Force UA so we dont try to re-cache in a loop. 2022-12-07 15:14:33 +01:00
Raphaël Vinot 3e7f1c3de9 chg: use cache instead of get_info everywhere. 2022-12-07 14:32:13 +01:00
Raphaël Vinot 94b3b487f3 chg: Add more entries to the cache, use it in the modules. 2022-12-07 13:03:15 +01:00
Raphaël Vinot 52076925b8 fix: Avoid exception on non-empty directory 2022-12-06 11:00:01 +01:00
Raphaël Vinot 6871c36848 fix: Skip urlhaus module if no reports 2022-12-05 18:22:47 +01:00
Raphaël Vinot eecff2cd9b fix: Do not pass the user_agent if a device_name is set 
Fix https://github.com/Lookyloo/lookyloo/issues/556
 2022-12-02 09:57:56 +01:00
Raphaël Vinot 9653fc2cc7 new: Avoid dangling keys in running, sync AbstractManager with pandora 2022-12-01 15:52:21 +01:00
Raphaël Vinot af9515c123 new: URL Haus module 
Related: #505
 2022-11-30 18:38:41 +01:00
Raphaël Vinot 56ed06eb84 new: Show error message in bulk capture 2022-11-29 15:30:16 +01:00
Raphaël Vinot 3092b46f5d chg: More logging improvements 2022-11-28 12:23:40 +01:00
Raphaël Vinot 2d0b7224cc new: Timeout if a tree takes too long to generate 2022-11-27 00:09:40 +01:00
Raphaël Vinot 298c091068 chg: Reduce logging in indexing 2022-11-23 17:19:05 +01:00
Raphaël Vinot 00370291ac new: Logging config in file 2022-11-23 15:54:22 +01:00
Raphaël Vinot 3c1cbd6ece new: Very basic page to submit an existing capture via a HAR file 2022-11-19 01:32:17 +01:00
Raphaël Vinot 9677c4d120 new: Support lacus unreachable by caching locally 
+ initialize lacus globally for consistency.
 2022-11-01 18:10:25 +01:00
Raphaël Vinot 8caa45a74d fix: Avoid passing a referer to None, causing URLscan to fail. 2022-10-29 01:01:23 +02:00
Raphaël Vinot a48c6e0bd6 new: SIGTERM handling (PyLacus and LacusCore) 2022-10-28 12:40:28 +02:00
Raphaël Vinot 83fb88c24e fix: Document upload 2022-10-07 15:39:40 +02:00
Raphaël Vinot 5be0014de7 chg: Improve logging 2022-09-28 12:52:23 +02:00
Raphaël Vinot c067d2cb68 fix: Properly report status is lookyloo is lagging behind lacus 2022-09-28 12:23:40 +02:00
Raphaël Vinot c71a79fb14 chg: clear cache on update 2022-09-28 11:44:15 +02:00
Raphaël Vinot a27683f090 fix: Match compressed HAR as valid for rebuild 2022-09-28 11:23:44 +02:00
Raphaël Vinot dcc6b158b0 fix: exception on broken captures in stats 2022-09-28 02:10:55 +02:00
Raphaël Vinot 0074f33544 fix: do not add already processed UUIDs in the queue 2022-09-27 22:00:47 +02:00
Raphaël Vinot d80d9c50f9 chg: Disable depth on lookyloo enqueue 2022-09-27 15:43:44 +02:00
Raphaël Vinot 23bad9bc4a fix: More exceptions avoidance on broken captures 2022-09-27 12:28:15 +02:00
Raphaël Vinot 33d30a3f4c chg: More cleanup 2022-09-27 02:39:10 +02:00
Raphaël Vinot df4a37ea12 fix: Properly catch broken captures 2022-09-26 17:16:04 +02:00
Raphaël Vinot 8687d8fabf chg: better reporting for broken captures 2022-09-26 16:04:23 +02:00
Raphaël Vinot 8d7e0ecf15 chg: Less logging 2022-09-26 15:51:14 +02:00
Raphaël Vinot 31261e84c2 fix: Better handling of half broken captures without HAR files 2022-09-26 14:58:30 +02:00
Raphaël Vinot 50e59bdf31 fix: Not having a HAR file is valid now. 2022-09-25 22:52:42 +02:00
Raphaël Vinot 862c9e0750 fix: wrong key type 2022-09-23 15:38:23 +02:00
Raphaël Vinot c7ca251e7a chg: make to_capture key a ranked set again 2022-09-23 14:25:01 +02:00
Raphaël Vinot b6428977b7 fix: Exception if there are no lookyloo-specific settings 2022-09-23 13:50:51 +02:00
Raphaël Vinot 86f3c5638e chg: Bump deps, fix typo 2022-09-23 13:45:14 +02:00
Raphaël Vinot ea5debdeaf fix: Typo 2022-09-23 13:13:09 +02:00
Raphaël Vinot da502ee3d6 chg: Implement support for LacusCore *or* PyLacus 2022-09-23 13:13:09 +02:00