Commit Graph

113 Commits (1e0b2c16d75e9b069586e8a83dba0ee01d276847)

Author SHA1 Message Date
Raphaël Vinot 6ba019ec83 chg: Improve somewhat the useragents available for capturing
Fix #416
2022-06-09 18:58:17 +02:00
Raphaël Vinot 1817a3e13b chg: sunday cleanup 2022-05-23 00:15:52 +02:00
Raphaël Vinot d222ae04aa new: Keep capture even if we have a network error 2022-05-03 12:23:16 +02:00
Raphaël Vinot 463d1d2d1a new: autosubmit to FOX, bump deps 2022-05-02 13:04:55 +02:00
Raphaël Vinot ef1094a331 chg: Bump deps, fix cookie issue
Fix  #404
2022-04-29 00:44:03 +02:00
Raphaël Vinot 1679ccf90f chg: Improve capture, ignore ssl issues. 2022-04-26 13:49:24 +02:00
Raphaël Vinot 77fbf47e73 fix: capture cleanup 2022-04-26 10:25:11 +02:00
Raphaël Vinot 147bc65992 fix: Mypy, docker 2022-04-26 00:59:57 +02:00
Raphaël Vinot 41c7e87458 fix: docker, improve error catching 2022-04-26 00:33:50 +02:00
Raphaël Vinot 5af278f84d fix: issue in playwrightcapture module 2022-04-25 15:20:05 +02:00
Raphaël Vinot 4ad898a375 chg: Use packaged playwright capture module 2022-04-25 13:34:01 +02:00
Raphaël Vinot c93a6c307d chg: properly set cookies 2022-04-24 20:17:54 +03:00
Raphaël Vinot 680eb1b309 fix: better handling if capture fails. 2022-04-21 15:48:28 +03:00
Raphaël Vinot 8d159ffba0 new: Switch away from splash to use playwright 2022-04-21 14:55:07 +03:00
Raphaël Vinot 83fc0bd8f4 fix: shutil.move wants str (not Path) for python<3.9 2022-04-10 12:43:56 +02:00
Kimmo Linnavuo a80b6a31e4 Use shutil.move instead of path rename when moving discarded captures 2022-04-08 15:28:06 +03:00
Raphaël Vinot cf46dde1ed chg: Add basic pre-hook config 2022-03-31 11:30:53 +02:00
Raphaël Vinot ae9cb3e81c chg: Bump deps 2022-03-29 21:13:02 +02:00
Raphaël Vinot c9307b5159 chg: Improve start/stop for DBs 2021-12-02 14:39:32 +01:00
Raphaël Vinot a55fb5380a chg: Sync stop script with template 2021-11-26 14:16:22 -05:00
Raphaël Vinot d7c9892957 fix: Wait for DBs to be down before returning in stop script 2021-11-26 13:48:46 -05:00
Raphaël Vinot daca988f3f chg: better handling of broken indexes in archiver 2021-11-26 12:36:35 -05:00
Raphaël Vinot cef1088984 chg: programmatically shutdown DBs 2021-11-26 12:35:15 -05:00
Raphaël Vinot 58b50f2b24 new: Pass optional arbitrary HTTP headers to capture 2021-11-23 12:59:56 -08:00
Raphaël Vinot bfb1e6b181 fix: Use default_public for all capture, including if submitted via the API 2021-11-02 14:58:31 -07:00
Raphaël Vinot 1f998b457f chg: use template 2021-10-18 13:06:43 +02:00
Raphaël Vinot 6e9e3990c4 fix: Indexes not updated on tree rebuild, better handling of tree cache 2021-09-24 16:16:41 +02:00
Raphaël Vinot 48fc807e7d new: Add monitoring for pickle cache status 2021-09-24 12:02:28 +02:00
Raphaël Vinot 32ee474be2 chg: Improve tree creation and cache 2021-09-22 17:09:04 +02:00
Raphaël Vinot d1f673f3a7 chg: Cleanup passing listing key to and from bool in redis 2021-09-10 14:20:58 +02:00
Raphaël Vinot 9c7929569e fix: The captures are visible on the index by default. 2021-09-08 20:43:56 +02:00
Raphaël Vinot 48b632aa1e fix: Incorrect matching for listing key in capture (always false) 2021-09-08 10:53:31 +02:00
Raphaël Vinot 902c8f81b6 chg: Improve error message if the capture fails
Fix #257
2021-09-07 18:16:01 +02:00
Raphaël Vinot dfbe40a52e chg: reorder imports 2021-09-07 16:00:07 +02:00
Raphaël Vinot c09adec333 chg: Improve logging. 2021-09-01 14:08:25 +02:00
Raphaël Vinot 797de9ddb3 fix: remove datefmt from logging.basicConfig, it was a bad idea. 2021-09-01 10:40:59 +02:00
Raphaël Vinot 2e5a5f3aff fix: unlink indexes pointing to unknown directories 2021-08-30 14:45:44 +02:00
Raphaël Vinot e56c70d1a1 chg: out of safety, do not remove a capture dir. 2021-08-30 12:54:17 +02:00
Raphaël Vinot 117500b777 chg: Make archiver an index generator 2021-08-30 12:48:13 +02:00
Raphaël Vinot 324736f62c fix: Use proper exception on redis start 2021-08-27 18:08:34 +02:00
Raphaël Vinot ae76cb77be fix: Uncomment website start 2021-08-27 17:49:27 +02:00
Raphaël Vinot 8a51383d7a chg: Move the process managment methods to the proper class 2021-08-27 17:28:26 +02:00
Raphaël Vinot 85e43fc677 chg: Make the website start a normal start script 2021-08-27 16:45:16 +02:00
Raphaël Vinot d41b7735dd chg: Improve storage, support both modes. 2021-08-26 15:49:19 +02:00
Raphaël Vinot 407e78ae7f chg: More cleanup, support clean shutdown of multiple async captures 2021-08-25 16:40:51 +02:00
Raphaël Vinot bf700e7a7b chg: Major refactoring, move capture code to external script. 2021-08-25 13:36:48 +02:00
Raphaël Vinot c732e38395 chg: Add logging in BG processing 2021-08-24 18:44:00 +02:00
Raphaël Vinot 81390d5ea0 chg: cleanup in the mail lookyloo class 2021-08-24 18:32:54 +02:00
Raphaël Vinot 8433cbcc1b chg: Cleanup archiver, initialize index captures in start 2021-08-24 17:10:14 +02:00
Raphaël Vinot ece30a33eb chg: Fix typo in archiver 2021-08-23 16:56:17 +02:00