CIRCL
/
lookyloo
mirror of https://github.com/CIRCL/lookyloo
2
0
Fork 0
Code Issues Releases Wiki Activity
1,982 Commits
8 Branches
29 Tags
68 MiB
Commit Graph

221 Commits (refactor_indexing)

Author SHA1 Message Date
Raphaël Vinot f67a24f5c2 new: Store directories by day, refactor indexing (WiP) 2023-11-16 00:38:36 +01:00
Raphaël Vinot 1c5c178d20 fix: s3fs support was broken. 2023-10-23 15:59:14 +02:00
Raphaël Vinot fcaeda8f7f new: Use S3FS in archiving script instead, remove python 3.12 uspport 
Also remove standalone script for updating archived indexes.
 2023-10-23 13:57:44 +02:00
Raphaël Vinot db9ca0ea2b fix: Properly match 0/1 as string 2023-10-20 15:55:50 +02:00
Raphaël Vinot a2ba5c551d fix: allow auto_report to be "True" without any setting. 2023-10-20 15:48:28 +02:00
Raphaël Vinot 0daff9ef77 chg: settings tweaks, logging 2023-10-11 15:02:11 +02:00
Raphaël Vinot b4599492f3 fix: Avoid exception killing website if non-responsive 3rd party module. 2023-10-11 14:57:53 +02:00
Raphaël Vinot 5ca7c5cb1d fix: Typo in last commit 2023-10-10 21:40:49 +02:00
Raphaël Vinot 3e4eb572a0 chg: auto-restart webservers after 1000 requests 2023-10-10 21:32:26 +02:00
Raphaël Vinot 2920f796fe fix: Speedup generating pickles in BG 2023-10-09 10:26:37 +02:00
Raphaël Vinot f2c9647a9e new: Don't attempt to initialize indexes if they're on a s3fs mount 2023-10-04 11:06:02 +02:00
Raphaël Vinot e3b85508f1 fix: Attempt to check if a directory is empty faster. 2023-10-02 16:16:22 +02:00
Raphaël Vinot f250cba632 chg: yet another attempt to improve checking archived captures 2023-10-02 15:50:46 +02:00
Raphaël Vinot 1220f5926d fix: reduce calls to stat on archived dirs, improve logging 2023-09-29 15:00:40 +02:00
Raphaël Vinot 3b5e45a1e7 fix: Properly stop when there is nothing to archive 2023-09-23 11:38:25 +02:00
Raphaël Vinot d4b9ca13af chg: Avoid setting the lock and quitting, cleanup 2023-09-19 14:26:35 +02:00
Raphaël Vinot 1fcd4cfa3f fix: Make sure we only get valid dirs out of the month directory 2023-09-19 12:51:34 +02:00
Raphaël Vinot 50406a921c fix: Attempt to avoid listing non-existing directories 2023-09-19 12:14:00 +02:00
Raphaël Vinot 29fb60e9b7 fix: Avois error when capture is in a weird state 2023-09-18 15:02:33 +02:00
Raphaël Vinot 1533e33ede fix: remove locks from the archived directories 2023-09-18 10:38:05 +02:00
Raphaël Vinot 3fb9da8480 fix: Try to avoid exceptions and race conditions between archiving and build pickle 2023-09-18 01:31:28 +02:00
Raphaël Vinot 532b68dd07 fix: Avoid exception when attempting to move a capture 2023-09-18 00:33:59 +02:00
Raphaël Vinot 84175a944c fix: Update index of the right directory 2023-09-15 15:38:23 +02:00
Raphaël Vinot ae484cc14f chg: Update index right after archiving 2023-09-15 13:25:37 +02:00
Raphaël Vinot 8c707b364f fix: Missing f in f-string... 2023-09-15 11:42:33 +02:00
Raphaël Vinot 6c3f64e78a chg: Avoid waiting when many captures need to be archived 2023-09-15 11:30:04 +02:00
Raphaël Vinot 416ca7224e fix: Avoid race condition when re-enqueuing 2023-09-01 16:00:45 +02:00
Raphaël Vinot 2ec34e2b9b chg: Improve archiver 2023-08-20 16:21:33 +02:00
Raphaël Vinot d0d08b5882 fix: Avoid exception on empty dir in archiver. 2023-08-16 11:15:00 +02:00
Raphaël Vinot 447229ced3 chg: Compress HARs by default, update codebase accordingly 2023-08-11 13:16:59 +02:00
Raphaël Vinot 206e5957b5 new: Support for favicons fetching and display 
Related https://github.com/Lookyloo/PlaywrightCapture/issues/45
 2023-08-09 16:50:33 +02:00
Raphaël Vinot e256a7fe6b chg: Proper use of shutil.move, speedup initialization of CaptureCache 2023-08-08 12:41:21 +02:00
Raphaël Vinot fa1d353834 chg: Avoid a few more disk access whenever possible. 2023-08-07 13:13:57 +02:00
Raphaël Vinot 212ba9a0d9 chg: Reduce disk usage 2023-08-06 21:34:20 +02:00
Raphaël Vinot 27b10f2c05 chg: Speedup indexes update 2023-08-05 20:47:08 +02:00
Raphaël Vinot 14df52a623 chg: Many improvments in archiver 2023-08-05 13:36:56 +02:00
Raphaël Vinot e9dad5de61 chg: Attempt to reduce disk use 2023-08-04 15:03:58 +02:00
Raphaël Vinot c203aa91b9 chg: Avoid directory listing as much as possible in archiver, allow shutdown 2023-08-04 14:02:45 +02:00
Raphaël Vinot 5fca6b13ea chg: Show stacktrace when we cannot build the pickle 2023-08-04 13:15:39 +02:00
Raphaël Vinot 959b7ca96d fix: use glob with path instead of rglob (faster)) 2023-08-04 13:15:03 +02:00
Raphaël Vinot 4be8186cc6 chg: Improve readability of the background indexer 2023-07-30 16:59:41 +02:00
Raphaël Vinot ea2ded9beb fix: properly handle missing title in cache 2023-07-27 15:21:06 +02:00
Raphaël Vinot ebfc2f00a5 fix: Exception when a formerly broken capture is re-processed and works 2023-07-27 14:56:39 +02:00
Raphaël Vinot 855485984f fix: handle gracefully empty lists in hset, and duplicates UUIDs 2023-07-26 22:16:00 +02:00
Raphaël Vinot fd9325bb0d chg: Improve logging, add lock on indexer. 2023-07-26 12:37:12 +02:00
Raphaël Vinot f60457a484 fix: Put the max captures counter at the right place... 2023-07-26 11:45:22 +02:00
Raphaël Vinot fc5850e147 chg: Avoid building old pickles forever 2023-07-26 11:38:40 +02:00
Raphaël Vinot a18f8f9675 chg: do not discard capture without HAR files 
They are often just captures with an error file.
 2023-07-25 20:29:30 +02:00
Raphaël Vinot ef3432cbed fix: Few more improvments on lockfile and broken captures. 2023-07-25 20:16:48 +02:00
Raphaël Vinot 484aec5ddd fix: Properly handle lock file. 2023-07-25 19:29:53 +02:00