Raphaël Vinot
|
79c9b95faf
|
fix: Avoid issue if a cature is somewhat broken and we want to send a notification
|
2023-07-06 14:50:38 +02:00 |
Raphaël Vinot
|
2c5c30006c
|
fix: index page stuck building captures for no good reason
|
2023-07-04 11:22:36 +02:00 |
Raphaël Vinot
|
6bb660ed82
|
fix: incorrect pickle rebuild, causing the index to get stuck
|
2023-07-03 17:04:55 +02:00 |
Raphaël Vinot
|
95a17f5770
|
fix: Handling of captures with errors
Related https://github.com/Lookyloo/monitoring/issues/15
|
2023-06-21 15:41:14 +02:00 |
Raphaël Vinot
|
475b424878
|
chg: retry a few times if remote lacus isn't available immediately
|
2023-06-16 12:47:50 +02:00 |
Raphaël Vinot
|
324c85cac8
|
new: Support for STARTTLS in the notification
|
2023-06-14 16:19:46 +02:00 |
Raphaël Vinot
|
0505edff0e
|
new: Global proxy setting.
Overwrites any proxy given by the user. Still uses the tor proxy when
needed.
Related: https://github.com/Lookyloo/lookyloo/issues/318
|
2023-06-13 12:07:26 +02:00 |
Raphaël Vinot
|
97a668aa91
|
chg: do not export pickle
|
2023-06-12 16:01:30 +02:00 |
Raphaël Vinot
|
4ff9b7651d
|
new: Support for TZ, Geo, locale, color scheme
|
2023-06-07 15:05:40 +02:00 |
Raphaël Vinot
|
474ab91dcd
|
chg: Use URL from capture settings file
|
2023-06-05 15:21:22 +02:00 |
Raphaël Vinot
|
b3b7d4f5c6
|
new: Add lookyloo permaurls in comparator response
|
2023-05-17 12:14:24 +02:00 |
Raphaël Vinot
|
f8acca5e51
|
fix: properly ignore resources if only hostname matches
|
2023-05-17 11:14:53 +02:00 |
Raphaël Vinot
|
386aa6e91b
|
fix: make the sets into tuples so endswith works
|
2023-05-17 09:53:20 +02:00 |
Raphaël Vinot
|
95f05a3da1
|
new: Display proxy if passed to the CaptureSettings
Fix #695
|
2023-05-15 16:33:49 +02:00 |
Raphaël Vinot
|
582b5956e9
|
new: Store capture settings, use TypedDict whenever possible.
|
2023-05-15 16:08:19 +02:00 |
Raphaël Vinot
|
a504facf5e
|
new: Flag to quickly say if two captures are different or not
|
2023-05-11 15:07:37 +02:00 |
Raphaël Vinot
|
14d4e0f2db
|
fix: MISP lookup from Lookyloo.
|
2023-05-04 10:20:54 +02:00 |
Raphaël Vinot
|
6a9bcc0050
|
new: Automatic reporting via API
Related to #678
|
2023-04-28 17:19:53 +02:00 |
Raphaël Vinot
|
c62032fee2
|
new: Endpoint to get comparables from a capture
|
2023-04-24 16:25:29 +02:00 |
Raphaël Vinot
|
b33ef171a4
|
new: Method to get comparable stuff out of tree
|
2023-04-21 23:57:23 +02:00 |
Raphaël Vinot
|
88997e7c99
|
fix: Incorrect call to new PSL lib
|
2023-04-20 16:23:56 +03:00 |
Raphaël Vinot
|
6754400e25
|
fix: [URLHaus] Do not store responses if the URL is invalid
|
2023-04-13 16:17:58 +02:00 |
Raphaël Vinot
|
c7d3e54edf
|
fix: Cloudflare module always returning True
|
2023-04-12 14:05:05 +02:00 |
Raphaël Vinot
|
4ceae60db7
|
chg: Avoid stopping the captures before they're done
|
2023-04-09 13:58:34 +02:00 |
Raphaël Vinot
|
2ceda75eab
|
chg: Fairly big refactoring/cleanup to support LacusCore 1.4.0
|
2023-04-08 13:49:18 +02:00 |
Raphaël Vinot
|
9995371916
|
chg: Normalize logging on the config file settings
|
2023-04-05 16:23:46 +02:00 |
Raphaël Vinot
|
9281d73675
|
fix: Properly handle recursion error when storing pickle
|
2023-03-30 16:30:27 +02:00 |
Raphaël Vinot
|
8122b9904c
|
fix: remove gziped pickle too on error.
|
2023-03-30 15:50:13 +02:00 |
Raphaël Vinot
|
c48bf2f4de
|
new: Pass compare settings (WiP)
|
2023-03-24 15:47:41 +01:00 |
Raphaël Vinot
|
046417c058
|
chg: Bump pysecurity.txt, pass Lookyloos UA
|
2023-03-21 11:28:08 +01:00 |
Raphaël Vinot
|
a0a23dca20
|
chg: Bump har2tree
|
2023-03-16 17:32:14 +01:00 |
Raphaël Vinot
|
d970c924f8
|
chg: Simplify tracing the exceptions related to a specific capture.
|
2023-03-16 13:49:22 +01:00 |
Raphaël Vinot
|
afd383cfc3
|
fix: Better error handling with unpickling, improve logging
|
2023-03-16 12:45:58 +01:00 |
Raphaël Vinot
|
9497060028
|
fix: Cleanup prints, improve archiver.
|
2023-03-16 12:28:28 +01:00 |
Raphaël Vinot
|
0b678e2db0
|
new: Optimize pickle, store as gzip to reduce space.
|
2023-03-15 18:02:33 +01:00 |
Raphaël Vinot
|
07c365e3bb
|
chg: Improve logging
|
2023-03-12 19:25:16 +01:00 |
Raphaël Vinot
|
25433b5707
|
chg: always use LookylooException
|
2023-03-12 19:25:16 +01:00 |
Raphaël Vinot
|
3927b97584
|
fix: let the module figure out the mimetype for the attachment
|
2023-03-09 13:17:10 +02:00 |
Raphaël Vinot
|
53d9a6ba7a
|
new: Add contacts as attachmetn to notification mail
|
2023-03-09 12:55:45 +02:00 |
Raphaël Vinot
|
4a33e10cc9
|
chg: force protocol 5 for pickles
|
2023-03-08 16:55:18 +02:00 |
Raphaël Vinot
|
7a96ae9767
|
new: Fetch the exiting collection names from the monitoring interface
|
2023-03-02 20:51:46 +01:00 |
Raphaël Vinot
|
cf25da5646
|
new: Show the monitoring settings on the modal
|
2023-02-27 16:01:46 +01:00 |
Raphaël Vinot
|
09f788d511
|
Merge branch 'main' into monitoring
|
2023-02-26 17:32:35 +01:00 |
Raphaël Vinot
|
36d39f6076
|
new: Add PID in lock file, allows to check if the locking process is still there
|
2023-02-26 17:20:17 +01:00 |
Raphaël Vinot
|
11fed3f46a
|
fix: Remove print
|
2023-02-23 18:49:59 +01:00 |
Raphaël Vinot
|
5e72e1104e
|
new: Basic support for monitoring
|
2023-02-23 18:37:40 +01:00 |
Raphaël Vinot
|
370850285d
|
fix: exception when cloudflare lookup is unavailable
|
2023-02-23 16:55:24 +01:00 |
Raphaël Vinot
|
1558f65c8e
|
new: try to get contact info from security.txt file if available.
|
2023-02-14 13:32:00 +01:00 |
Raphaël Vinot
|
2f2fd6024b
|
fix: Some nodes do not have IPs, can't always use it for comparaison
|
2023-02-09 15:39:40 +01:00 |
Raphaël Vinot
|
aae3d56ddc
|
new: Aggregate all emails by URL
|
2023-02-03 11:32:13 +01:00 |
Raphaël Vinot
|
f6a12d50ea
|
new: very basic takedown information gatherer
|
2023-02-02 15:18:39 +01:00 |
Raphaël Vinot
|
01a4b4d632
|
chg: Optionally not-defang the URLs in the email notification
|
2023-02-02 15:18:39 +01:00 |
Raphaël Vinot
|
599ae9f9f3
|
new: Compare ressources URLs
|
2023-01-31 11:24:16 +01:00 |
Raphaël Vinot
|
9273b80602
|
fix: Cleanup cloudflare module.
|
2023-01-30 16:26:46 +01:00 |
Raphaël Vinot
|
ecb4623b86
|
new: Cloudflare lookup feature to flag IPs appropriately
|
2023-01-30 16:07:01 +01:00 |
Raphaël Vinot
|
0cb58d17d9
|
new: Add status code in comparator
|
2023-01-24 15:59:47 +01:00 |
Raphaël Vinot
|
eab4b17070
|
fix: Avoid exceptions when the submission is a file
|
2023-01-24 15:59:06 +01:00 |
Raphaël Vinot
|
f335b4a58d
|
fix: Use publicsuffixlist instead of publicsuffix2
|
2023-01-20 15:23:11 +01:00 |
Raphaël Vinot
|
d449edeb23
|
new: Allow API user to force re-caching the captures on URL/Hostname request
|
2023-01-20 11:15:33 +01:00 |
Raphaël Vinot
|
190d5a7118
|
new: Very basic capture comparator, API only.
|
2023-01-18 16:31:12 +01:00 |
Raphaël Vinot
|
fdf1069c76
|
fix: Properly load IPs in old format.
|
2023-01-16 17:22:23 +01:00 |
Raphaël Vinot
|
a3c62d1682
|
fix: Issue when loading IPs
|
2023-01-16 17:00:11 +01:00 |
Raphaël Vinot
|
1e563ff345
|
chg: Improve DNS resolution
|
2023-01-16 15:50:10 +01:00 |
Raphaël Vinot
|
ae59b59071
|
fix: improper concat for sets
|
2023-01-16 15:16:40 +01:00 |
Raphaël Vinot
|
6624aec1bd
|
new: run DNS lookups on AAAA too
|
2023-01-16 15:10:10 +01:00 |
Raphaël Vinot
|
55d727792a
|
fix: Add logger in capture cache
|
2022-12-31 12:19:42 +01:00 |
Raphaël Vinot
|
c7aba15ee2
|
fix: Avoid exception if the redirects key in cache is empty
|
2022-12-31 12:16:50 +01:00 |
Raphaël Vinot
|
5200402e06
|
chg: Another quick deps bump, prepare v1.17.0
|
2022-12-29 14:07:27 +01:00 |
Raphaël Vinot
|
01b9f8d1be
|
fix: remove print, replace with logging
|
2022-12-27 01:55:18 +01:00 |
Raphaël Vinot
|
02497e167c
|
fix: Catch QuotaExceededError in VT module
|
2022-12-26 23:30:00 +01:00 |
Raphaël Vinot
|
1abd97f37a
|
new: switch to return emails only on whois query
|
2022-12-26 23:01:44 +01:00 |
Raphaël Vinot
|
479fcb39e5
|
new: Basic support for IPASN lookup
|
2022-12-23 16:36:11 +01:00 |
Raphaël Vinot
|
ca7ec64ffe
|
fix: avoid exception if the tree cannot be build
|
2022-12-19 17:42:14 +01:00 |
Raphaël Vinot
|
0a9003f58e
|
chg: Use cache whenever possible
|
2022-12-08 11:57:45 +01:00 |
Raphaël Vinot
|
1ae76ddea7
|
chg: Use cache instead of pickle for meta info
|
2022-12-08 10:49:26 +01:00 |
Raphaël Vinot
|
2c6d6b2556
|
chg: Get UUIDs sorted from most recent to oldest.
|
2022-12-07 21:18:38 +01:00 |
Raphaël Vinot
|
8083c8cabc
|
fix: Major improvement in index display when you break the cache...
|
2022-12-07 21:05:27 +01:00 |
Raphaël Vinot
|
8c77b9372b
|
fix: do not re-cache captures with errors.
|
2022-12-07 15:30:19 +01:00 |
Raphaël Vinot
|
d2b5605182
|
fix: Force UA so we dont try to re-cache in a loop.
|
2022-12-07 15:14:33 +01:00 |
Raphaël Vinot
|
3e7f1c3de9
|
chg: use cache instead of get_info everywhere.
|
2022-12-07 14:32:13 +01:00 |
Raphaël Vinot
|
94b3b487f3
|
chg: Add more entries to the cache, use it in the modules.
|
2022-12-07 13:03:15 +01:00 |
Raphaël Vinot
|
52076925b8
|
fix: Avoid exception on non-empty directory
|
2022-12-06 11:00:01 +01:00 |
Raphaël Vinot
|
6871c36848
|
fix: Skip urlhaus module if no reports
|
2022-12-05 18:22:47 +01:00 |
Raphaël Vinot
|
eecff2cd9b
|
fix: Do not pass the user_agent if a device_name is set
Fix https://github.com/Lookyloo/lookyloo/issues/556
|
2022-12-02 09:57:56 +01:00 |
Raphaël Vinot
|
9653fc2cc7
|
new: Avoid dangling keys in running, sync AbstractManager with pandora
|
2022-12-01 15:52:21 +01:00 |
Raphaël Vinot
|
af9515c123
|
new: URL Haus module
Related: #505
|
2022-11-30 18:38:41 +01:00 |
Raphaël Vinot
|
56ed06eb84
|
new: Show error message in bulk capture
|
2022-11-29 15:30:16 +01:00 |
Raphaël Vinot
|
3092b46f5d
|
chg: More logging improvements
|
2022-11-28 12:23:40 +01:00 |
Raphaël Vinot
|
2d0b7224cc
|
new: Timeout if a tree takes too long to generate
|
2022-11-27 00:09:40 +01:00 |
Raphaël Vinot
|
298c091068
|
chg: Reduce logging in indexing
|
2022-11-23 17:19:05 +01:00 |
Raphaël Vinot
|
00370291ac
|
new: Logging config in file
|
2022-11-23 15:54:22 +01:00 |
Raphaël Vinot
|
3c1cbd6ece
|
new: Very basic page to submit an existing capture via a HAR file
|
2022-11-19 01:32:17 +01:00 |
Raphaël Vinot
|
9677c4d120
|
new: Support lacus unreachable by caching locally
+ initialize lacus globally for consistency.
|
2022-11-01 18:10:25 +01:00 |
Raphaël Vinot
|
8caa45a74d
|
fix: Avoid passing a referer to None, causing URLscan to fail.
|
2022-10-29 01:01:23 +02:00 |
Raphaël Vinot
|
a48c6e0bd6
|
new: SIGTERM handling (PyLacus and LacusCore)
|
2022-10-28 12:40:28 +02:00 |
Raphaël Vinot
|
83fb88c24e
|
fix: Document upload
|
2022-10-07 15:39:40 +02:00 |
Raphaël Vinot
|
5be0014de7
|
chg: Improve logging
|
2022-09-28 12:52:23 +02:00 |
Raphaël Vinot
|
c067d2cb68
|
fix: Properly report status is lookyloo is lagging behind lacus
|
2022-09-28 12:23:40 +02:00 |
Raphaël Vinot
|
c71a79fb14
|
chg: clear cache on update
|
2022-09-28 11:44:15 +02:00 |
Raphaël Vinot
|
a27683f090
|
fix: Match compressed HAR as valid for rebuild
|
2022-09-28 11:23:44 +02:00 |
Raphaël Vinot
|
dcc6b158b0
|
fix: exception on broken captures in stats
|
2022-09-28 02:10:55 +02:00 |
Raphaël Vinot
|
0074f33544
|
fix: do not add already processed UUIDs in the queue
|
2022-09-27 22:00:47 +02:00 |
Raphaël Vinot
|
d80d9c50f9
|
chg: Disable depth on lookyloo enqueue
|
2022-09-27 15:43:44 +02:00 |
Raphaël Vinot
|
23bad9bc4a
|
fix: More exceptions avoidance on broken captures
|
2022-09-27 12:28:15 +02:00 |
Raphaël Vinot
|
33d30a3f4c
|
chg: More cleanup
|
2022-09-27 02:39:10 +02:00 |
Raphaël Vinot
|
df4a37ea12
|
fix: Properly catch broken captures
|
2022-09-26 17:16:04 +02:00 |
Raphaël Vinot
|
8687d8fabf
|
chg: better reporting for broken captures
|
2022-09-26 16:04:23 +02:00 |
Raphaël Vinot
|
8d7e0ecf15
|
chg: Less logging
|
2022-09-26 15:51:14 +02:00 |
Raphaël Vinot
|
31261e84c2
|
fix: Better handling of half broken captures without HAR files
|
2022-09-26 14:58:30 +02:00 |
Raphaël Vinot
|
50e59bdf31
|
fix: Not having a HAR file is valid now.
|
2022-09-25 22:52:42 +02:00 |
Raphaël Vinot
|
862c9e0750
|
fix: wrong key type
|
2022-09-23 15:38:23 +02:00 |
Raphaël Vinot
|
c7ca251e7a
|
chg: make to_capture key a ranked set again
|
2022-09-23 14:25:01 +02:00 |
Raphaël Vinot
|
b6428977b7
|
fix: Exception if there are no lookyloo-specific settings
|
2022-09-23 13:50:51 +02:00 |
Raphaël Vinot
|
86f3c5638e
|
chg: Bump deps, fix typo
|
2022-09-23 13:45:14 +02:00 |
Raphaël Vinot
|
ea5debdeaf
|
fix: Typo
|
2022-09-23 13:13:09 +02:00 |
Raphaël Vinot
|
da502ee3d6
|
chg: Implement support for LacusCore *or* PyLacus
|
2022-09-23 13:13:09 +02:00 |
Raphaël Vinot
|
d38b612c37
|
chg: Bump lacuscore
|
2022-09-23 13:13:09 +02:00 |
Raphaël Vinot
|
623813167e
|
chg: Add missing bits
|
2022-09-23 13:13:09 +02:00 |
Raphaël Vinot
|
dcbae777a1
|
fix: working integration of lacus.
|
2022-09-23 13:13:09 +02:00 |
Raphaël Vinot
|
318f554db3
|
chg: move to lacus, WiP
|
2022-09-23 13:13:09 +02:00 |
Raphaël Vinot
|
812c63b0f2
|
fix: error in UAs, typing
|
2022-09-05 18:58:45 +02:00 |
Raphaël Vinot
|
742a0b1799
|
fix: Exception when we try to update the UAs from within a PW context
|
2022-09-01 18:20:29 +02:00 |
Raphaël Vinot
|
c6464936fc
|
chg: Bump to poetry v1.2, remove dep on setuptools
|
2022-08-31 16:33:13 +02:00 |
Raphaël Vinot
|
31514b79f9
|
fix: Exception in VT module
|
2022-08-25 16:09:30 +02:00 |
Raphaël Vinot
|
ec07429e65
|
chg: Much better handling of downloaded files
|
2022-08-25 13:28:02 +02:00 |
Raphaël Vinot
|
cbe73fa13f
|
fix: better handling of downloaded content from URL
|
2022-08-24 14:06:48 +02:00 |
Raphaël Vinot
|
53b4297dc8
|
chg: Bump deps
|
2022-08-24 13:28:37 +02:00 |
Raphaël Vinot
|
4bde0b1f5d
|
fix: Missing file...
|
2022-08-23 17:55:19 +02:00 |
Raphaël Vinot
|
f4dfa0343e
|
new: Submit to Pandora button for file URLs
|
2022-08-23 17:48:36 +02:00 |
Raphaël Vinot
|
f232eba662
|
chg: Improve UA rendering
|
2022-08-23 17:44:48 +02:00 |
Raphaël Vinot
|
ebbe6e3ce9
|
new: Pick mobile devices on capture page
|
2022-08-22 17:34:00 +02:00 |
Raphaël Vinot
|
0a07412877
|
fix: Make sure the view is consistent across gunicorn processes
|
2022-08-18 12:07:15 +02:00 |
Raphaël Vinot
|
998ef12b06
|
new: Add support for playwright devices and browser name (API only)
|
2022-08-18 11:19:32 +02:00 |
Raphaël Vinot
|
d6dc30b260
|
chg: cleanup formating
|
2022-08-14 17:14:48 +02:00 |
Raphaël Vinot
|
7dc27d48b4
|
fix: properly handle listing configuration, clear None from queries before pasing to redis
|
2022-08-14 16:48:32 +02:00 |
Raphaël Vinot
|
c9a5adc066
|
fix: Avoid invalid URL chars in filename
|
2022-08-14 16:41:37 +02:00 |
Raphaël Vinot
|
72c4e43474
|
new: Upload a file instead of submitting a URL.
|
2022-08-14 16:41:34 +02:00 |
Raphaël Vinot
|
fdfa2e25cb
|
new: Initial listing of all resources loaded on the capture
|
2022-08-14 16:31:49 +02:00 |
Raphaël Vinot
|
0c81f6b218
|
fix: Do not try to load archived captures in sorted_capture_cache
|
2022-08-14 16:15:53 +02:00 |
Arhamyss
|
c589933a40
|
fix: changed the way things are displayed and fixed errors to make the download working
|
2022-08-10 11:40:11 +02:00 |
Arhamyss
|
5a77044530
|
new: first try to add an url in the hostnode to dl the file if there is one to be downloaded
|
2022-08-09 18:23:11 +02:00 |
Raphaël Vinot
|
137c05648b
|
fix: Always make sure to build the tree when we set the cache
|
2022-07-29 19:31:56 +02:00 |
Raphaël Vinot
|
a80a5425fa
|
fix: wrong delete call
|
2022-07-29 19:08:56 +02:00 |
Raphaël Vinot
|
ac4b81e880
|
chg: That wil teach me not to use mypy
|
2022-07-29 19:05:18 +02:00 |
Raphaël Vinot
|
8b5ce7986b
|
chg: properly clear cache when needed
|
2022-07-29 19:02:06 +02:00 |
Raphaël Vinot
|
80541f4d35
|
chg: Some more cleanup in the caching system
|
2022-07-29 13:40:15 +02:00 |
Raphaël Vinot
|
08897d4d77
|
fix: Python 3.8 compatibility
|
2022-07-29 13:15:45 +02:00 |
Raphaël Vinot
|
5cc51b3e3e
|
chg: Few more improvements in cache.
|
2022-07-29 13:08:42 +02:00 |
Raphaël Vinot
|
b1d30cadcf
|
fix: Attempt to fix the stats page
|
2022-07-29 11:02:51 +02:00 |
Raphaël Vinot
|
9b4d56a0b6
|
fix: Do not display all the recent captures with the same hash in popup
|
2022-07-28 16:14:47 +02:00 |