CIRCL
/
lookyloo
mirror of https://github.com/CIRCL/lookyloo
2
0
Fork 0
Code Issues Releases Wiki Activity
2,127 Commits
8 Branches
31 Tags
73 MiB
Commit Graph

672 Commits (bf47dc79dcf6eebab38f9349b7c73b912e8899db)

Author SHA1 Message Date
Raphaël Vinot 3af6068dce fix: allow to run the scripts from anywhere 2023-11-20 23:52:01 +01:00
Raphaël Vinot 7791eff842 new: Store directories by day, refactor indexing 2023-11-16 16:54:21 +01:00
Raphaël Vinot 1b987c38b7 chg: avoid issue if ipasnhistory is down, improve IDNA rendering 2023-11-10 21:19:22 +01:00
Raphaël Vinot 8152e3260c fix: Avoid exception if ipasnhistory is loading 2023-11-09 11:10:42 +01:00
Raphaël Vinot 264d94cd2d fix: Pass IPs to whois instead of v4/v6... 2023-11-06 21:17:34 +01:00
Raphaël Vinot 7b142277ed fix: Properly handle unlimited requests in body hash lookup 2023-11-06 16:58:32 +01:00
Raphaël Vinot c8606ff167 fix: Do not update cached UUIDs from hostnode popup 2023-11-06 16:45:37 +01:00
Raphaël Vinot 77d5cab1e8 chg: Speedup hostnode popup 2023-11-06 16:16:09 +01:00
Raphaël Vinot 21965758f2 fix: Avoid encoding issue with titles 2023-10-26 00:05:50 +02:00
Raphaël Vinot a6fff6687d fix: Avoid exception when there is no URL to send to VT 2023-10-21 16:59:23 +02:00
Raphaël Vinot a2ba5c551d fix: allow auto_report to be "True" without any setting. 2023-10-20 15:48:28 +02:00
Raphaël Vinot 4143d5f249 chg: avoid trying to load a tree that is currenly being built 2023-10-13 20:08:58 +02:00
Raphaël Vinot 0daff9ef77 chg: settings tweaks, logging 2023-10-11 15:02:11 +02:00
Raphaël Vinot b4599492f3 fix: Avoid exception killing website if non-responsive 3rd party module. 2023-10-11 14:57:53 +02:00
Raphaël Vinot 2920f796fe fix: Speedup generating pickles in BG 2023-10-09 10:26:37 +02:00
Raphaël Vinot 1220f5926d fix: reduce calls to stat on archived dirs, improve logging 2023-09-29 15:00:40 +02:00
Raphaël Vinot 68204c2686 new: Support for rendered nodes with HTML and downloaded file 2023-09-26 18:08:04 +02:00
Raphaël Vinot 5d5b93266d new: optionally ignore IPs when comparing nodes 
Related https://github.com/Lookyloo/monitoring/issues/23
 2023-09-12 14:03:02 +02:00
Raphaël Vinot 0365c9d6c9 fix: Strip all non-URL characters from capture. 2023-09-11 16:54:50 +02:00
Raphaël Vinot 00bf8fb551 fix: Improve getting MISP instances 2023-08-29 17:30:45 +02:00
Raphaël Vinot 6034d10834 new: (WiP) Add support for multiple MISPs 2023-08-29 14:03:01 +02:00
Raphaël Vinot 72ba77377d chg: Catch empty file hash and return empry file immediately 2023-08-24 15:12:38 +02:00
Raphaël Vinot ff15f3a60a Avoid exceptions when trying to get a datablob 2023-08-24 14:55:32 +02:00
Raphaël Vinot 8b316ad185 fix: Properly search hash in embedded resources 2023-08-24 14:37:03 +02:00
Raphaël Vinot 2a843ac828 chg: Delist less captures, avoid edge case causing an exception for MISP export 2023-08-23 17:22:46 +02:00
Raphaël Vinot 447229ced3 chg: Compress HARs by default, update codebase accordingly 2023-08-11 13:16:59 +02:00
Raphaël Vinot 1f517b1562 chg: Cosmetic changes to support upcomming ete4 2023-08-10 17:31:19 +02:00
Raphaël Vinot 206e5957b5 new: Support for favicons fetching and display 
Related https://github.com/Lookyloo/PlaywrightCapture/issues/45
 2023-08-09 16:50:33 +02:00
Raphaël Vinot e256a7fe6b chg: Proper use of shutil.move, speedup initialization of CaptureCache 2023-08-08 12:41:21 +02:00
Raphaël Vinot 54674f6c5b avoid exception if pickle needs to be rebuild 2023-08-04 13:16:05 +02:00
Raphaël Vinot 35218900ed fix: MISP export when the capture is a file. 2023-07-30 17:48:16 +02:00
Raphaël Vinot 182e7af188 chg: Attempt to keep index up-to-date with cached captures 2023-07-28 14:40:33 +02:00
Raphaël Vinot c0f601e5db chg: remove incomplete_redirects from cache, was always 0/False 2023-07-28 14:05:28 +02:00
Raphaël Vinot ea2ded9beb fix: properly handle missing title in cache 2023-07-27 15:21:06 +02:00
Raphaël Vinot ebfc2f00a5 fix: Exception when a formerly broken capture is re-processed and works 2023-07-27 14:56:39 +02:00
Raphaël Vinot db639d9dde chg: Improve logging for errors in cookies, bump har2tree 2023-07-27 12:43:52 +02:00
Raphaël Vinot 855485984f fix: handle gracefully empty lists in hset, and duplicates UUIDs 2023-07-26 22:16:00 +02:00
Raphaël Vinot fd9325bb0d chg: Improve logging, add lock on indexer. 2023-07-26 12:37:12 +02:00
Raphaël Vinot ebd2b29993 fix: export compress HAR 2023-07-26 11:28:32 +02:00
Raphaël Vinot ef3432cbed fix: Few more improvments on lockfile and broken captures. 2023-07-25 20:16:48 +02:00
Raphaël Vinot 484aec5ddd fix: Properly handle lock file. 2023-07-25 19:29:53 +02:00
Raphaël Vinot 0c7b3d9106 fix: indexer getting stuck when we had more than one at a time 2023-07-25 17:08:00 +02:00
Raphaël Vinot 18b5a51c86 fix: Exception when none of the captures are cached. 2023-07-24 22:41:11 +02:00
Raphaël Vinot 0604ccfdee new: Improve the HHH details page 2023-07-24 14:53:00 +02:00
Raphaël Vinot 8cd6f5490a fix: Properly cache URL, fix mypy issue 
Related https://github.com/Lookyloo/PyLookyloo/issues/32
 2023-07-24 13:15:13 +02:00
Raphaël Vinot 36f9d3dc60 chg: Improve error reporting 
Related https://github.com/Lookyloo/PyLookyloo/issues/32
 2023-07-24 12:48:57 +02:00
Raphaël Vinot 177474e874 new: Basic support for HHHash 2023-07-21 15:48:20 +02:00
Raphaël Vinot b3000d02dd fix: 🤷 2023-07-13 16:59:52 +02:00
Raphaël Vinot f3095c7ccf chg: Add logging when comparing is impossible due to a missing pickle 2023-07-10 12:27:43 +02:00
Raphaël Vinot 3193763c8a fix: logic issue in last commit 2023-07-06 14:59:24 +02:00
Raphaël Vinot 79c9b95faf fix: Avoid issue if a cature is somewhat broken and we want to send a notification 2023-07-06 14:50:38 +02:00
Raphaël Vinot 2c5c30006c fix: index page stuck building captures for no good reason 2023-07-04 11:22:36 +02:00
Raphaël Vinot 6bb660ed82 fix: incorrect pickle rebuild, causing the index to get stuck 2023-07-03 17:04:55 +02:00
Raphaël Vinot 95a17f5770 fix: Handling of captures with errors 
Related https://github.com/Lookyloo/monitoring/issues/15
 2023-06-21 15:41:14 +02:00
Raphaël Vinot 475b424878 chg: retry a few times if remote lacus isn't available immediately 2023-06-16 12:47:50 +02:00
Raphaël Vinot 324c85cac8 new: Support for STARTTLS in the notification 2023-06-14 16:19:46 +02:00
Raphaël Vinot 0505edff0e new: Global proxy setting. 
Overwrites any proxy given by the user. Still uses the tor proxy when
needed.

Related: https://github.com/Lookyloo/lookyloo/issues/318
 2023-06-13 12:07:26 +02:00
Raphaël Vinot 97a668aa91 chg: do not export pickle 2023-06-12 16:01:30 +02:00
Raphaël Vinot 4ff9b7651d new: Support for TZ, Geo, locale, color scheme 2023-06-07 15:05:40 +02:00
Raphaël Vinot 474ab91dcd chg: Use URL from capture settings file 2023-06-05 15:21:22 +02:00
Raphaël Vinot b3b7d4f5c6 new: Add lookyloo permaurls in comparator response 2023-05-17 12:14:24 +02:00
Raphaël Vinot f8acca5e51 fix: properly ignore resources if only hostname matches 2023-05-17 11:14:53 +02:00
Raphaël Vinot 386aa6e91b fix: make the sets into tuples so endswith works 2023-05-17 09:53:20 +02:00
Raphaël Vinot 95f05a3da1 new: Display proxy if passed to the CaptureSettings 
Fix #695
 2023-05-15 16:33:49 +02:00
Raphaël Vinot 582b5956e9 new: Store capture settings, use TypedDict whenever possible. 2023-05-15 16:08:19 +02:00
Raphaël Vinot a504facf5e new: Flag to quickly say if two captures are different or not 2023-05-11 15:07:37 +02:00
Raphaël Vinot 14d4e0f2db fix: MISP lookup from Lookyloo. 2023-05-04 10:20:54 +02:00
Raphaël Vinot 6a9bcc0050 new: Automatic reporting via API 
Related to #678
 2023-04-28 17:19:53 +02:00
Raphaël Vinot c62032fee2 new: Endpoint to get comparables from a capture 2023-04-24 16:25:29 +02:00
Raphaël Vinot b33ef171a4 new: Method to get comparable stuff out of tree 2023-04-21 23:57:23 +02:00
Raphaël Vinot 88997e7c99 fix: Incorrect call to new PSL lib 2023-04-20 16:23:56 +03:00
Raphaël Vinot 6754400e25 fix: [URLHaus] Do not store responses if the URL is invalid 2023-04-13 16:17:58 +02:00
Raphaël Vinot c7d3e54edf fix: Cloudflare module always returning True 2023-04-12 14:05:05 +02:00
Raphaël Vinot 4ceae60db7 chg: Avoid stopping the captures before they're done 2023-04-09 13:58:34 +02:00
Raphaël Vinot 2ceda75eab chg: Fairly big refactoring/cleanup to support LacusCore 1.4.0 2023-04-08 13:49:18 +02:00
Raphaël Vinot 9995371916 chg: Normalize logging on the config file settings 2023-04-05 16:23:46 +02:00
Raphaël Vinot 9281d73675 fix: Properly handle recursion error when storing pickle 2023-03-30 16:30:27 +02:00
Raphaël Vinot 8122b9904c fix: remove gziped pickle too on error. 2023-03-30 15:50:13 +02:00
Raphaël Vinot c48bf2f4de new: Pass compare settings (WiP) 2023-03-24 15:47:41 +01:00
Raphaël Vinot 046417c058 chg: Bump pysecurity.txt, pass Lookyloos UA 2023-03-21 11:28:08 +01:00
Raphaël Vinot a0a23dca20 chg: Bump har2tree 2023-03-16 17:32:14 +01:00
Raphaël Vinot d970c924f8 chg: Simplify tracing the exceptions related to a specific capture. 2023-03-16 13:49:22 +01:00
Raphaël Vinot afd383cfc3 fix: Better error handling with unpickling, improve logging 2023-03-16 12:45:58 +01:00
Raphaël Vinot 9497060028 fix: Cleanup prints, improve archiver. 2023-03-16 12:28:28 +01:00
Raphaël Vinot 0b678e2db0 new: Optimize pickle, store as gzip to reduce space. 2023-03-15 18:02:33 +01:00
Raphaël Vinot 07c365e3bb chg: Improve logging 2023-03-12 19:25:16 +01:00
Raphaël Vinot 25433b5707 chg: always use LookylooException 2023-03-12 19:25:16 +01:00
Raphaël Vinot 3927b97584 fix: let the module figure out the mimetype for the attachment 2023-03-09 13:17:10 +02:00
Raphaël Vinot 53d9a6ba7a new: Add contacts as attachmetn to notification mail 2023-03-09 12:55:45 +02:00
Raphaël Vinot 4a33e10cc9 chg: force protocol 5 for pickles 2023-03-08 16:55:18 +02:00
Raphaël Vinot 7a96ae9767 new: Fetch the exiting collection names from the monitoring interface 2023-03-02 20:51:46 +01:00
Raphaël Vinot cf25da5646 new: Show the monitoring settings on the modal 2023-02-27 16:01:46 +01:00
Raphaël Vinot 09f788d511 Merge branch 'main' into monitoring 2023-02-26 17:32:35 +01:00
Raphaël Vinot 36d39f6076 new: Add PID in lock file, allows to check if the locking process is still there 2023-02-26 17:20:17 +01:00
Raphaël Vinot 11fed3f46a fix: Remove print 2023-02-23 18:49:59 +01:00
Raphaël Vinot 5e72e1104e new: Basic support for monitoring 2023-02-23 18:37:40 +01:00
Raphaël Vinot 370850285d fix: exception when cloudflare lookup is unavailable 2023-02-23 16:55:24 +01:00
Raphaël Vinot 1558f65c8e new: try to get contact info from security.txt file if available. 2023-02-14 13:32:00 +01:00
Raphaël Vinot 2f2fd6024b fix: Some nodes do not have IPs, can't always use it for comparaison 2023-02-09 15:39:40 +01:00
Raphaël Vinot aae3d56ddc new: Aggregate all emails by URL 2023-02-03 11:32:13 +01:00