Commit Graph

1235 Commits (4042ee911639ad35179a7166ff52283381076275)

Author SHA1 Message Date
Raphaël Vinot 2e5a5f3aff fix: unlink indexes pointing to unknown directories 2021-08-30 14:45:44 +02:00
Raphaël Vinot 4c4b01547a chg: Bump deps 2021-08-30 13:55:07 +02:00
Raphaël Vinot e56c70d1a1 chg: out of safety, do not remove a capture dir. 2021-08-30 12:54:17 +02:00
Raphaël Vinot 117500b777 chg: Make archiver an index generator 2021-08-30 12:48:13 +02:00
Raphaël Vinot 1bff8f1529 fix: cast capture dir to string for redis 2021-08-28 20:36:50 +02:00
Raphaël Vinot 324736f62c fix: Use proper exception on redis start 2021-08-27 18:08:34 +02:00
Raphaël Vinot ae76cb77be fix: Uncomment website start 2021-08-27 17:49:27 +02:00
Raphaël Vinot 8a51383d7a chg: Move the process managment methods to the proper class 2021-08-27 17:28:26 +02:00
Raphaël Vinot 85e43fc677 chg: Make the website start a normal start script 2021-08-27 16:45:16 +02:00
Raphaël Vinot 8918b11a7f chg: The Tech Blog UAs can only be fetched manually, moving it to tools 2021-08-27 15:22:39 +02:00
Raphaël Vinot 9ee6771158 new: bump top user agents 2021-08-27 15:07:27 +02:00
Raphaël Vinot 5bd0ae850f chg: improve lookup_dirs cleanup 2021-08-27 14:27:38 +02:00
Raphaël Vinot 11b3654b10 chg: Bump deps 2021-08-27 14:26:51 +02:00
Raphaël Vinot 6edb0c5d56 chg: Bump deps 2021-08-26 15:54:30 +02:00
Raphaël Vinot d41b7735dd chg: Improve storage, support both modes. 2021-08-26 15:49:19 +02:00
Raphaël Vinot 407e78ae7f chg: More cleanup, support clean shutdown of multiple async captures 2021-08-25 16:40:51 +02:00
Raphaël Vinot bc5e41a1ca fix: Properly re-set the cache if the tree was not built 2021-08-25 15:00:26 +02:00
Raphaël Vinot 83969ab8bb chg: Bump deps 2021-08-25 13:39:04 +02:00
Raphaël Vinot bf700e7a7b chg: Major refactoring, move capture code to external script. 2021-08-25 13:36:48 +02:00
Raphaël Vinot d8416f0f47 fix: Make python as python3 in docker 2021-08-24 18:50:05 +02:00
Raphaël Vinot c732e38395 chg: Add logging in BG processing 2021-08-24 18:44:00 +02:00
Raphaël Vinot 81390d5ea0 chg: cleanup in the mail lookyloo class 2021-08-24 18:32:54 +02:00
Raphaël Vinot 8433cbcc1b chg: Cleanup archiver, initialize index captures in start 2021-08-24 17:10:14 +02:00
Raphaël Vinot ece30a33eb chg: Fix typo in archiver 2021-08-23 16:56:17 +02:00
Raphaël Vinot fb1685cedc add: reset recent captures in archiving process 2021-08-23 16:19:50 +02:00
Raphaël Vinot 8f28335010 fix: properly match cut time 2021-08-23 15:51:06 +02:00
Raphaël Vinot 2c1971311a chg: Make the cut-off date for archiving the 1st of the month 2021-08-23 15:36:59 +02:00
Raphaël Vinot 5c9b88a3ca fix: Make sure all the archived UUIDs are removed 2021-08-23 15:29:21 +02:00
Raphaël Vinot 67e6571145 chg: Force init the archived indexes 2021-08-23 15:14:08 +02:00
Raphaël Vinot 53ceb9c329 chg: Cleanup when dir is moved, digit months on 2 values 2021-08-23 14:53:19 +02:00
Raphaël Vinot b8109c6462 fix: Make mypy happy 2021-08-23 14:20:26 +02:00
Raphaël Vinot d359bc7521 chg: Better use of cache, sanity checks 2021-08-23 12:17:44 +02:00
Raphaël Vinot 9eb5076133 fix: typo in config file 2021-08-20 17:53:26 +02:00
Raphaël Vinot 1b14c248dd chg: Add config for archive 2021-08-20 17:50:57 +02:00
Raphaël Vinot 8bd63a1efa chg: Bump deps 2021-08-20 17:47:19 +02:00
Raphaël Vinot 58b837cb6c new: Archiver, refactoring. 2021-08-20 17:46:22 +02:00
Raphaël Vinot 6be9b69d95 chg: Use connection pool whenever possible 2021-08-18 18:01:04 +02:00
Raphaël Vinot 7495e5b4f1 chg: Use connection pool in indexing 2021-08-18 17:31:17 +02:00
Raphaël Vinot 046387a9f4 chg: Make mypy happy 2021-08-18 16:37:50 +02:00
Raphaël Vinot f2376ebf0a fix: broken poetry.lock 2021-08-18 16:35:41 +02:00
Raphaël Vinot 08aeb1f83f new: Add use hiredis 2021-08-18 16:29:47 +02:00
Raphaël Vinot 3a96e27f8c chg: Try using redis connection pool 2021-08-18 16:07:39 +02:00
Raphaël Vinot 725a43cc44 new: Trigger a capture with a GET request from a browser.
Fix #248
2021-08-18 10:53:32 +02:00
Raphaël Vinot 48d81815a0 chg: Bump d3js 2021-08-17 12:22:10 +02:00
Raphaël Vinot 248e1f5fb2 chg: Bump deps 2021-08-17 12:14:03 +02:00
Raphaël Vinot 021f53716b new: Submit a capture in a HTTP GET request
Fix #246
2021-08-17 12:12:23 +02:00
Raphaël Vinot 09a69622c0 fix: urlscan dislikes URLs without scheme. 2021-08-16 11:01:00 +02:00
Raphaël Vinot 39979c8708 chg: Bump har2tree, bugfix 2021-08-14 23:45:21 +02:00
Raphaël Vinot 9ff2bf8157 fix: avoid exception if submission is missing 2021-08-13 16:24:34 +02:00
Raphaël Vinot 50dbbd6eba new: [API] Trigger modules for a capture 2021-08-13 13:50:26 +02:00