Raphaël Vinot
|
926c0da23e
|
chg: Disable index cache for backgroupd processes
|
2024-03-12 12:02:10 +01:00 |
Raphaël Vinot
|
e40796c92e
|
chg: merge index and reindex methods.
|
2024-03-11 00:14:07 +01:00 |
Raphaël Vinot
|
d2df33aa5c
|
fix: use a more direct way to index
|
2024-03-09 15:33:10 +01:00 |
Raphaël Vinot
|
24cc00fe96
|
chg: clear cache when it is not needed
|
2024-03-08 15:50:47 +01:00 |
Raphaël Vinot
|
07a6fe8f49
|
chg: Bump deps
|
2024-03-08 10:36:04 +01:00 |
Raphaël Vinot
|
caab916ff1
|
fix: Missing file
|
2024-03-05 21:03:36 +01:00 |
Raphaël Vinot
|
e45b7c4346
|
new: Indexer for *all* the captures
|
2024-03-05 20:51:21 +01:00 |
Raphaël Vinot
|
9e302a9b14
|
new: Add shodan hash on favicon views
|
2024-02-26 19:09:48 +01:00 |
Raphaël Vinot
|
decf887b63
|
new: Shodan MM3H indexing
|
2024-02-26 17:07:23 +01:00 |
Raphaël Vinot
|
7e25747d82
|
fix: properly check zscore (can be 0)
|
2024-02-24 21:18:57 +01:00 |
Raphaël Vinot
|
4153138644
|
new: Add favicons in indexer
|
2024-02-19 16:15:52 +01:00 |
Raphaël Vinot
|
e02b7392a6
|
chg: Only search for finished captures in the top 50, properly march CaptureStatusCore.DONE
|
2024-02-15 02:36:16 +01:00 |
Raphaël Vinot
|
c67f01c775
|
chg: Improve strict typing
|
2024-01-26 15:03:36 +01:00 |
Raphaël Vinot
|
fcfe9751f3
|
fix: potential race condition when checking if a capture is ongoing or not
|
2024-01-25 14:21:57 +01:00 |
Raphaël Vinot
|
86dfb20122
|
chg; Bump PyLacus
|
2024-01-16 00:27:43 +01:00 |
Raphaël Vinot
|
60c8d7e78d
|
fix: Do not set priority to None
|
2024-01-14 02:18:21 +01:00 |
Raphaël Vinot
|
bd6a0f2d22
|
chg: cleanup with annotations
|
2024-01-13 01:45:45 +01:00 |
Raphaël Vinot
|
ee1ad48b25
|
chg: Use new annotations
|
2024-01-12 17:15:41 +01:00 |
Raphaël Vinot
|
d06da1aa52
|
fix: Avoid exception if a file is missing on s3
|
2024-01-08 21:02:54 +01:00 |
Raphaël Vinot
|
f7c45b5039
|
fix: Add proper path in set to check
|
2024-01-08 16:50:48 +01:00 |
Raphaël Vinot
|
79f5b728d0
|
fix: do not attempt to remove a directory too early
|
2024-01-08 16:37:48 +01:00 |
Raphaël Vinot
|
d60a4e56db
|
chg: Update indexes only when needed
|
2024-01-08 16:27:12 +01:00 |
Raphaël Vinot
|
89dbef8683
|
chg: Avoid to discard the index lock too soon
|
2023-11-21 16:50:15 +01:00 |
Raphaël Vinot
|
9031141b61
|
chg: Remove empty dirs when everything has been archived
|
2023-11-21 11:50:09 +01:00 |
Raphaël Vinot
|
6d61645d97
|
chg: remove index when all the captures are archived
|
2023-11-20 23:48:56 +01:00 |
Raphaël Vinot
|
efe2124753
|
fix: Quit BG indexer when shutdown is requested. Improve exceptions handling in archiver
|
2023-11-20 11:45:41 +01:00 |
Raphaël Vinot
|
11a3b6b2f9
|
chg: Improve indexes cleanup
|
2023-11-18 03:20:49 +01:00 |
Raphaël Vinot
|
ff27808320
|
fix: Path in index may be the full path (old format)
|
2023-11-18 02:47:43 +01:00 |
Raphaël Vinot
|
cd11df7ac4
|
fix: Update index files to remove archived (or simply gone) captures
|
2023-11-18 02:39:21 +01:00 |
Raphaël Vinot
|
9a9c4464ed
|
fix: Update index for recent captures on every archive
|
2023-11-17 15:47:12 +01:00 |
Raphaël Vinot
|
ce76218657
|
fix: build backlog pickles in reverse order
|
2023-11-16 23:58:07 +01:00 |
Raphaël Vinot
|
096d7c6fb5
|
chg: clear old UUIDs found when archiving
|
2023-11-16 23:22:04 +01:00 |
Raphaël Vinot
|
f209ef22f1
|
fix: skip root directory when scanning on s3fs
|
2023-11-16 22:55:44 +01:00 |
Raphaël Vinot
|
7791eff842
|
new: Store directories by day, refactor indexing
|
2023-11-16 16:54:21 +01:00 |
Raphaël Vinot
|
1c5c178d20
|
fix: s3fs support was broken.
|
2023-10-23 15:59:14 +02:00 |
Raphaël Vinot
|
fcaeda8f7f
|
new: Use S3FS in archiving script instead, remove python 3.12 uspport
Also remove standalone script for updating archived indexes.
|
2023-10-23 13:57:44 +02:00 |
Raphaël Vinot
|
db9ca0ea2b
|
fix: Properly match 0/1 as string
|
2023-10-20 15:55:50 +02:00 |
Raphaël Vinot
|
a2ba5c551d
|
fix: allow auto_report to be "True" without any setting.
|
2023-10-20 15:48:28 +02:00 |
Raphaël Vinot
|
0daff9ef77
|
chg: settings tweaks, logging
|
2023-10-11 15:02:11 +02:00 |
Raphaël Vinot
|
b4599492f3
|
fix: Avoid exception killing website if non-responsive 3rd party module.
|
2023-10-11 14:57:53 +02:00 |
Raphaël Vinot
|
5ca7c5cb1d
|
fix: Typo in last commit
|
2023-10-10 21:40:49 +02:00 |
Raphaël Vinot
|
3e4eb572a0
|
chg: auto-restart webservers after 1000 requests
|
2023-10-10 21:32:26 +02:00 |
Raphaël Vinot
|
2920f796fe
|
fix: Speedup generating pickles in BG
|
2023-10-09 10:26:37 +02:00 |
Raphaël Vinot
|
f2c9647a9e
|
new: Don't attempt to initialize indexes if they're on a s3fs mount
|
2023-10-04 11:06:02 +02:00 |
Raphaël Vinot
|
e3b85508f1
|
fix: Attempt to check if a directory is empty faster.
|
2023-10-02 16:16:22 +02:00 |
Raphaël Vinot
|
f250cba632
|
chg: yet another attempt to improve checking archived captures
|
2023-10-02 15:50:46 +02:00 |
Raphaël Vinot
|
1220f5926d
|
fix: reduce calls to stat on archived dirs, improve logging
|
2023-09-29 15:00:40 +02:00 |
Raphaël Vinot
|
3b5e45a1e7
|
fix: Properly stop when there is nothing to archive
|
2023-09-23 11:38:25 +02:00 |
Raphaël Vinot
|
d4b9ca13af
|
chg: Avoid setting the lock and quitting, cleanup
|
2023-09-19 14:26:35 +02:00 |
Raphaël Vinot
|
1fcd4cfa3f
|
fix: Make sure we only get valid dirs out of the month directory
|
2023-09-19 12:51:34 +02:00 |