Raphaël Vinot
|
4a8db1fc6e
|
fix: re-cache captures with incomplete redirects, code cleanup
|
2021-06-15 16:03:33 -07:00 |
Raphaël Vinot
|
b6a636697f
|
chg: Improve memory use CaptureCache
|
2021-06-15 15:16:56 -07:00 |
Raphaël Vinot
|
c52509614e
|
new: get capture info API call
|
2021-06-09 17:59:24 -07:00 |
Raphaël Vinot
|
3d2c7420a4
|
chg: Bump mypy
|
2021-06-09 12:12:01 -07:00 |
Raphaël Vinot
|
cf65489cef
|
chg: Bump mypy
|
2021-06-09 12:04:26 -07:00 |
Raphaël Vinot
|
88eb85df46
|
fix: Avoid exception if rendered page is missing on broken tree
|
2021-06-03 19:07:02 -07:00 |
Raphaël Vinot
|
1ae02e0dea
|
chg: Add body hash and domains in MISP lookup
|
2021-06-03 18:40:06 -07:00 |
Raphaël Vinot
|
387f2d3c3d
|
chg: Improve MISP lookup
|
2021-06-02 13:27:50 -07:00 |
Raphaël Vinot
|
f204947afa
|
fix: Missing rendered node in MISP lookup
|
2021-06-01 16:22:07 -07:00 |
Raphaël Vinot
|
8c62a597e7
|
fix: Make mypy happy
|
2021-06-01 16:06:24 -07:00 |
Raphaël Vinot
|
3071a1a7c9
|
new: Add MISP lookup
|
2021-06-01 15:31:14 -07:00 |
Raphaël Vinot
|
53ef253c94
|
chg: Improve MISP push
|
2021-05-31 13:27:25 -07:00 |
Raphaël Vinot
|
17af2df5cd
|
chg: Always make sure the lock file is removed even if building the tree fails
|
2021-05-28 17:28:26 -07:00 |
Raphaël Vinot
|
1117ab6371
|
chg: add stats, avoid building big trees twice, bump deps
|
2021-05-26 18:25:06 -07:00 |
Raphaël Vinot
|
11f05626b5
|
chg: Improve module auto trigger
|
2021-05-19 15:03:07 -07:00 |
Raphaël Vinot
|
896f3c5f13
|
fix: properly check if capture is in the queue
|
2021-05-18 16:04:13 -07:00 |
Raphaël Vinot
|
925bb9d48e
|
new: priority for captures
|
2021-05-18 14:58:56 -07:00 |
Raphaël Vinot
|
bb214d9e2b
|
chg: Use keywords only paramaters when relevant
|
2021-05-17 17:08:43 -07:00 |
Raphaël Vinot
|
5ee62d157f
|
chg: Properly handle capture parents, avoid duplicates
|
2021-05-14 14:25:37 -07:00 |
Raphaël Vinot
|
1d5925d755
|
new: Keep captures parent, use extends in MISP export
|
2021-05-12 13:30:07 -07:00 |
Fafner [_KeyZee_]
|
97207b8f77
|
solving tor proxy for only_global_lookups
|
2021-05-04 17:17:21 +02:00 |
Raphaël Vinot
|
12697964ce
|
chg: Improve tree stats
|
2021-04-27 17:09:58 +02:00 |
Raphaël Vinot
|
3389dcafd6
|
new: Add support for uwhoisd
|
2021-04-26 00:52:08 +02:00 |
Raphaël Vinot
|
39e894b7b0
|
fix: body hash page on direct access
|
2021-04-20 17:32:22 +02:00 |
Raphaël Vinot
|
9952e4de32
|
fix: remove UA - IP mapping from redis
|
2021-04-18 17:58:16 +02:00 |
Raphaël Vinot
|
bf5d15baf0
|
fix: remove capture from index
|
2021-04-14 18:05:00 +02:00 |
Raphaël Vinot
|
d7665e0a5e
|
chg: Make mypy happy
|
2021-04-09 14:48:42 +02:00 |
Raphaël Vinot
|
f865ec912a
|
fix: Move set/unset running to abstract
Avoid issues when a script fails unexpectedly.
|
2021-04-09 14:33:42 +02:00 |
Raphaël Vinot
|
9470b0c738
|
fix: remove broken pickles
|
2021-04-09 10:44:37 +02:00 |
Raphaël Vinot
|
7707d638cf
|
new: Use async capture for the UI.
Add a method to make sure splash is up before trying to capture.
|
2021-04-08 19:15:53 +02:00 |
Raphaël Vinot
|
4d23d7ac36
|
chg: Remove extra code, return unziped resources
|
2021-04-08 01:05:11 +02:00 |
Raphaël Vinot
|
e492428bf5
|
fix: Proper call to local cache
|
2021-04-03 02:24:38 +02:00 |
Raphaël Vinot
|
82db729e96
|
fix: cache captures index in main script
|
2021-04-03 02:06:32 +02:00 |
Raphaël Vinot
|
b6ea1c24db
|
new: Query capture status
|
2021-03-31 19:25:57 +02:00 |
Raphaël Vinot
|
a8711a18f1
|
fix: Make mypy happy
|
2021-03-30 16:43:57 +02:00 |
Raphaël Vinot
|
9f88edd74d
|
new: Check if an event with the same lookyloo capture is already on MISP
Fix #181
|
2021-03-30 16:32:04 +02:00 |
Raphaël Vinot
|
acb1bf2814
|
fix: MISP export when rendered node is empty
Fix #182
|
2021-03-30 14:06:34 +02:00 |
Raphaël Vinot
|
88bd450b59
|
chg: Improve tree page (menu and stats)
|
2021-03-25 18:32:48 +01:00 |
Raphaël Vinot
|
fa6b4701c0
|
chg: update the cache at the right place.
|
2021-03-20 21:54:46 +01:00 |
Raphaël Vinot
|
b1fbf9ff74
|
fix: Properly type capture_dir in cache
|
2021-03-20 01:18:28 +01:00 |
Raphaël Vinot
|
2a55461286
|
new: Capture an URL on the rendered page, keep the session (WiP)
|
2021-03-19 17:52:20 +01:00 |
Raphaël Vinot
|
cd7b050cb0
|
chg: rename and cleanup methods
|
2021-03-18 18:47:54 +01:00 |
Raphaël Vinot
|
25c1a6dbce
|
chg: Use sorted cache whenever possible
|
2021-03-18 15:39:55 +01:00 |
Raphaël Vinot
|
49ae4490b2
|
new: (hidden) interface to search for hostnames and URLs
|
2021-03-18 00:40:14 +01:00 |
Raphaël Vinot
|
99402efa42
|
fix: allow to dump very big tree
|
2021-03-16 14:09:30 +01:00 |
Raphaël Vinot
|
1f9f5f1a9a
|
chg: Improve url and hostnames search
|
2021-03-16 13:35:59 +01:00 |
Raphaël Vinot
|
8c656062b9
|
fix: Catch exception when the tree has too many redirects.
|
2021-03-12 18:20:17 +01:00 |
Raphaël Vinot
|
46aea0fe3a
|
chg: Better error handling
|
2021-03-12 16:52:12 +01:00 |
Raphaël Vinot
|
e90fc151bd
|
fix: Avoid indexing URLs multiple times
|
2021-03-12 13:18:48 +01:00 |
Raphaël Vinot
|
4533bbdfa2
|
chg: Make MyPy happy
|
2021-03-08 14:03:03 +01:00 |
Raphaël Vinot
|
e35ceab3dd
|
chg: Better logging and error handling
|
2021-03-08 13:57:55 +01:00 |
Raphaël Vinot
|
e467f1d1eb
|
chg: Catch exceptions when the capture fails bad.
|
2021-03-08 11:44:29 +01:00 |
Raphaël Vinot
|
984a930aa8
|
chg: Avoid 500 on missing UUID
|
2021-03-07 23:57:46 +01:00 |
Raphaël Vinot
|
b09aec5b84
|
fix: un-break json dump
|
2021-02-12 12:06:26 +01:00 |
Raphaël Vinot
|
d711fd3644
|
fix: Make get legacy user config quiet
|
2021-02-05 15:24:11 +01:00 |
Raphaël Vinot
|
39dd2021dd
|
chg: Complete rework of the login system, add UI for MISP Push
|
2021-02-04 19:51:43 +01:00 |
Raphaël Vinot
|
610ffef43a
|
new: Add VT objects to MISP export
|
2021-02-02 15:23:38 +01:00 |
Raphaël Vinot
|
a68400dac7
|
chg: Add comments in MISP export
|
2021-02-02 15:23:38 +01:00 |
Raphaël Vinot
|
fca3290ef0
|
fix: Make sure to skip last year's final week in the stats
|
2021-02-02 15:23:38 +01:00 |
Raphaël Vinot
|
a7484e6cc4
|
chg: Improve MISP export
* IPs of redirects
* default tags
* auto publish
|
2021-02-01 18:09:32 +01:00 |
Raphaël Vinot
|
54ba67bb87
|
chg: Make lookyloo link org only in MISP export for private instances
|
2021-02-01 14:58:06 +01:00 |
Raphaël Vinot
|
f017ef5bef
|
chg: Add support to timeout in the MISP module
|
2021-02-01 13:01:44 +01:00 |
Raphaël Vinot
|
60ad0dc7ce
|
fix: add warning if MISP module is enabled, but instance unreachable.
|
2021-01-29 16:33:56 +01:00 |
Raphaël Vinot
|
c358c08640
|
fix: Make mypy happy.
|
2021-01-28 19:28:54 +01:00 |
Raphaël Vinot
|
acfb0d1c26
|
new: Add push to MISP feature
|
2021-01-28 18:37:47 +01:00 |
Raphaël Vinot
|
d8d2c59f8b
|
chg: Make mypy happy.
|
2021-01-25 13:14:33 +01:00 |
Raphaël Vinot
|
f9d8081efa
|
fix: hidden view
|
2021-01-18 13:26:02 +01:00 |
Raphaël Vinot
|
8bd874e2b0
|
new: Add variable width thumbnail method
|
2021-01-18 12:30:07 +01:00 |
Raphaël Vinot
|
72463cc898
|
chg: Slight cleanups and improvments
|
2021-01-14 17:28:59 +01:00 |
Raphaël Vinot
|
6149df06eb
|
chg: Make the cache entries a dataclass
Fix #99
|
2021-01-14 17:12:23 +01:00 |
Raphaël Vinot
|
61baab1fbe
|
fix: make mypy happy
|
2021-01-13 15:16:36 +01:00 |
Raphaël Vinot
|
4686b560dc
|
chg: Better handling of insanely long webpages.
|
2021-01-13 14:33:20 +01:00 |
Raphaël Vinot
|
0d68844c90
|
chg: Refactoring and documenting
|
2021-01-12 17:22:51 +01:00 |
Raphaël Vinot
|
33326c2463
|
new: Extract URLs from rendered pages
|
2021-01-08 13:03:23 +01:00 |
Raphaël Vinot
|
2e0671fd96
|
fix: Properly handle new year in week stats
|
2021-01-04 12:34:00 +01:00 |
Raphaël Vinot
|
7241ae2ea1
|
fix: Do not fail on MISP export if there are no redirects
|
2020-12-24 15:43:56 +01:00 |
Raphaël Vinot
|
9b18370fe1
|
fix: Properly generate meta keys related to the User Agents
|
2020-12-22 19:16:46 +01:00 |
Raphaël Vinot
|
fcae1d42d0
|
new: Show images in tooltip in hostnode popup
|
2020-12-22 18:23:39 +01:00 |
Raphaël Vinot
|
62cb5a9eb7
|
fix: Failure on new install
|
2020-12-11 10:26:42 +01:00 |
Raphaël Vinot
|
ed6a53d211
|
chg: small visual improvments
|
2020-12-09 21:08:34 +01:00 |
Raphaël Vinot
|
3c6eca3567
|
new: Add screenshot thumbnail on tree, move links to the end of the node
|
2020-12-09 19:11:19 +01:00 |
Raphaël Vinot
|
b73a3b7533
|
chg: Add attachements to MISP export
|
2020-12-08 14:59:34 +01:00 |
Raphaël Vinot
|
fbbfa3082d
|
chg: Fix typing
|
2020-12-07 20:54:33 +01:00 |
Raphaël Vinot
|
8eab287721
|
new: Initial implementation of MISP export
|
2020-12-07 20:50:46 +01:00 |
Raphaël Vinot
|
9b3d338d3c
|
chg: Rename analisys->submission, add unique urls on stats graph
|
2020-12-07 13:55:03 +01:00 |
Raphaël Vinot
|
7bba854e0a
|
fix: Properly call logger
|
2020-12-04 01:33:58 +01:00 |
Raphaël Vinot
|
5897f8024d
|
chg: Do not fail when sanejs is too slow to answer
|
2020-12-04 00:31:53 +01:00 |
Raphaël Vinot
|
d353463548
|
chg: Simplify modules activation
|
2020-12-03 12:33:35 +01:00 |
Raphaël Vinot
|
820c7c5a4d
|
fix: remove broken pickles if needed
|
2020-12-01 17:00:44 +01:00 |
Raphaël Vinot
|
1a7a74cccd
|
chg: Cleanup stats page
|
2020-11-27 16:27:29 +01:00 |
Fafner [_KeyZee_]
|
0e3d007bf5
|
fixing typo (again)
|
2020-11-25 12:22:45 +01:00 |
Fafner [_KeyZee_]
|
7bf13db3f2
|
fixing ttypo
|
2020-11-25 12:05:08 +01:00 |
Raphaël Vinot
|
d964549e56
|
fix: Typing, dump to json
|
2020-11-24 21:01:30 +01:00 |
Fafner [_KeyZee_]
|
d77dfb1cac
|
updating output for stats
|
2020-11-24 18:55:13 +01:00 |
Raphaël Vinot
|
c54d29601d
|
chg: Fix typing
|
2020-11-24 18:19:53 +01:00 |
Fafner [_KeyZee_]
|
bedbb8eb45
|
adding get_stats()
|
2020-11-24 16:46:01 +01:00 |
Fafner [_KeyZee_]
|
2ae7d87420
|
Adding uniq_domains
|
2020-11-24 16:44:43 +01:00 |
Raphaël Vinot
|
9c382890f5
|
fix: Improve lokking on error in capture
|
2020-11-10 20:27:01 +01:00 |
Raphaël Vinot
|
354f269218
|
new: Integrate categorization in indexing
|
2020-11-09 16:02:54 +01:00 |
Raphaël Vinot
|
9243f84295
|
fix: No exception if body_hash is not indexed
|
2020-11-06 15:59:13 +01:00 |
Raphaël Vinot
|
99c939fd7f
|
chg: Rename scrape -> capture everywhere
Related to #118
|
2020-11-03 16:34:52 +01:00 |
Raphaël Vinot
|
69d34b803c
|
fix: Duplicates on index.
|
2020-11-03 12:21:14 +01:00 |
Raphaël Vinot
|
8b1e3585ea
|
chg: Improve initial caching.
|
2020-10-29 23:25:20 +01:00 |
Raphaël Vinot
|
69f65c9a87
|
chg: Update cache on tree visualization
|
2020-10-29 17:55:43 +01:00 |
Raphaël Vinot
|
5f4a97e11b
|
fix: Typing meta
|
2020-10-29 14:19:17 +01:00 |
Raphaël Vinot
|
06a4c2cc87
|
fix: Speedup sorted cache
|
2020-10-29 14:06:38 +01:00 |
Raphaël Vinot
|
da85aa0fb7
|
fix: Pre-sort entries on index
|
2020-10-29 13:29:13 +01:00 |
Raphaël Vinot
|
2802cfd46c
|
new: Add captures categorization
|
2020-10-28 18:49:15 +01:00 |
Raphaël Vinot
|
451553e60d
|
fix: Do not overwrite the urlnodes
|
2020-10-27 01:52:28 +01:00 |
Raphaël Vinot
|
068c92d5a4
|
new: API to query hostnames
|
2020-10-27 01:42:00 +01:00 |
Raphaël Vinot
|
39f88e9121
|
new: API to query URLs
|
2020-10-27 00:02:18 +01:00 |
Raphaël Vinot
|
bdc0488e38
|
new: Hash lookup method
|
2020-10-23 20:51:15 +02:00 |
Raphaël Vinot
|
c6c4da981c
|
chg: Improve start/stop
|
2020-10-22 16:41:00 +02:00 |
Raphaël Vinot
|
f062895801
|
fix: Add max size to lru_cache
|
2020-10-13 13:03:57 +02:00 |
Raphaël Vinot
|
4f52804361
|
chg: Make new version of mypy happy
|
2020-10-12 12:15:07 +02:00 |
Raphaël Vinot
|
90a9ff9bb5
|
chg: Refactoring, add get_hashes
|
2020-10-09 18:05:25 +02:00 |
Raphaël Vinot
|
be2cd18d2b
|
chg: Cleanup mypy config
|
2020-10-01 11:48:00 +02:00 |
Raphaël Vinot
|
9ce06b8821
|
chg: Add resolved IP on hostnode page
|
2020-09-29 14:24:18 +02:00 |
Raphaël Vinot
|
ef6eb1c253
|
new: Initial support for CNAME lookups
|
2020-09-25 17:23:10 +02:00 |
Raphaël Vinot
|
3ce4f85118
|
fix: Avoid 500 when sanejs is unreachable
|
2020-09-25 12:02:40 +02:00 |
Raphaël Vinot
|
9663c5de87
|
fix: Properly load the config files when needed.
|
2020-09-21 16:54:07 +02:00 |
Raphaël Vinot
|
a7ecaee8c4
|
chg: If apikey is None in the modules, break immediately.
|
2020-09-21 16:47:49 +02:00 |
Raphaël Vinot
|
7a34095d9c
|
new: Config option for Flask IP and Port, reorganize config loading
|
2020-09-21 16:41:30 +02:00 |
Raphaël Vinot
|
b06f02e101
|
new: Expose rendered content on hostnode page
|
2020-09-09 15:04:21 +02:00 |
Raphaël Vinot
|
2b237a7416
|
new: Use rendered HTML
|
2020-09-08 19:18:55 +02:00 |
Raphaël Vinot
|
457829a23f
|
chg: cleanup and improve views
|
2020-09-04 18:40:51 +02:00 |
Raphaël Vinot
|
9f4c77d5d2
|
chg: Cleanups, allow to add context from ressources page
|
2020-09-03 16:32:53 +02:00 |
Raphaël Vinot
|
407a9a5511
|
new: Download ressource from ressources page
|
2020-09-03 14:39:38 +02:00 |
Raphaël Vinot
|
6d22085ceb
|
fix: do not crash on empty malicious file
|
2020-09-01 18:39:49 +02:00 |
Raphaël Vinot
|
ecbbceb781
|
fix: Typing
|
2020-09-01 17:54:54 +02:00 |
Raphaël Vinot
|
b16a5768ea
|
chg: Refactoring and normalizing the known hashes lookups
|
2020-09-01 17:33:36 +02:00 |
Raphaël Vinot
|
aee62136ab
|
fix: cache unknown hashes in sanejs
|
2020-09-01 00:51:44 +02:00 |
Raphaël Vinot
|
4a9a50e289
|
chg: Add icon for hostnodes with empty content.
|
2020-08-31 15:21:41 +02:00 |
Raphaël Vinot
|
a5c210a5de
|
new: Add checkmark for legitimate nodes.
|
2020-08-28 19:11:19 +02:00 |
Raphaël Vinot
|
6866b49c87
|
chg: Add support for legitimate content context
|
2020-08-28 18:24:14 +02:00 |
Raphaël Vinot
|
97b5b2d77f
|
chg: Working prototype for user-defined context
|
2020-08-28 18:03:52 +02:00 |
Raphaël Vinot
|
611b2a8ff5
|
fix: Crash if there are no ressources
|
2020-08-27 16:15:07 +02:00 |
Raphaël Vinot
|
9eb08beb3c
|
chg: Add load/dump of known legit captures
|
2020-08-27 15:48:56 +02:00 |
Raphaël Vinot
|
60b767d08f
|
chg: Normalize ressources details display, refactoring
|
2020-08-27 12:57:10 +02:00 |
Raphaël Vinot
|
11598fc3ba
|
chg: Major refactoring of the contextualization of the body contents
|
2020-08-26 16:52:59 +02:00 |
Raphaël Vinot
|
1b0cdde84e
|
chg: Normalize known content lookup
|
2020-08-25 18:00:16 +02:00 |
Raphaël Vinot
|
4d6237aae5
|
fix: Make mypy happy
|
2020-08-25 15:52:54 +02:00 |
Raphaël Vinot
|
c75b9e65fe
|
chg: Remove known content from SaneJS module
|
2020-08-25 15:51:08 +02:00 |
Raphaël Vinot
|
bdb726ca9e
|
new: Add malicious listing on hostnode
|
2020-08-24 15:31:53 +02:00 |
Raphaël Vinot
|
c5aabcf4a3
|
new: Initial commit to mark body responses as legitimate/malicious
|
2020-08-20 19:39:03 +02:00 |
Raphaël Vinot
|
fa57435433
|
chg: Sanity check in email send
|
2020-08-20 15:05:27 +02:00 |
Raphaël Vinot
|
339d0dd7e0
|
fix: Make mypy happy
|
2020-08-10 20:11:26 +02:00 |
Raphaël Vinot
|
83d8f49be9
|
fix: Missing change for last commit
|
2020-08-10 20:05:47 +02:00 |
Raphaël Vinot
|
2873773ee7
|
new: trigger to hide a capture from the front page (admin only)
|
2020-08-10 12:35:16 +02:00 |
Raphaël Vinot
|
f17acbce60
|
fix: Referer cannot be None
|
2020-07-20 23:04:11 +02:00 |