CIRCL
/
lookyloo
mirror of https://github.com/CIRCL/lookyloo
2
0
Fork 0
Code Issues Releases Wiki Activity
1,370 Commits
8 Branches
29 Tags
68 MiB
Commit Graph

283 Commits (1817a3e13b8897b731144b8fc0af5113a4328734)

Author SHA1 Message Date
Raphaël Vinot 3436f5bd4e chg: Improve urlscan support, get results. 2021-08-11 15:26:12 +02:00
Raphaël Vinot 7933670941 new: Integration with urlscan.io 2021-08-10 17:38:47 +02:00
Raphaël Vinot 28e3162187 Merge branch 'restx' 2021-07-21 11:05:25 +02:00
Raphaël Vinot 59f2a510c0 fix: properly catch broken capture, bump deps 2021-07-14 11:34:10 +02:00
Raphaël Vinot cc2e5a4159 Merge branch 'main' into restx 2021-07-05 09:54:37 +02:00
Raphaël Vinot 06aa85cfe8 fix: Do not return the whole database if the list of captures to sort is empty. 
Related #227
 2021-06-29 13:48:13 -07:00
felalex d3e0b7f394 new: proxy parameter in crawl 2021-06-25 19:11:15 +02:00
felalex 1c89a80c7e fix: expected format & passing proxy in _capture 2021-06-25 16:36:57 +02:00
Raphaël Vinot 26cbf94977 Merge branch 'main' into restx 2021-06-22 16:55:41 -07:00
Raphaël Vinot a44b7b30b5 new: Get URLs and Hostnames from captures 2021-06-16 14:57:14 -07:00
Raphaël Vinot 7ad9aa7bbc Merge branch 'main' into restx 2021-06-15 17:24:19 -07:00
Raphaël Vinot 4a8db1fc6e fix: re-cache captures with incomplete redirects, code cleanup 2021-06-15 16:03:33 -07:00
Raphaël Vinot c52509614e new: get capture info API call 2021-06-09 17:59:24 -07:00
Raphaël Vinot 3d2c7420a4 chg: Bump mypy 2021-06-09 12:12:01 -07:00
Raphaël Vinot cf65489cef chg: Bump mypy 2021-06-09 12:04:26 -07:00
Raphaël Vinot 387f2d3c3d chg: Improve MISP lookup 2021-06-02 13:27:50 -07:00
Raphaël Vinot f204947afa fix: Missing rendered node in MISP lookup 2021-06-01 16:22:07 -07:00
Raphaël Vinot 3071a1a7c9 new: Add MISP lookup 2021-06-01 15:31:14 -07:00
Raphaël Vinot 17af2df5cd chg: Always make sure the lock file is removed even if building the tree fails 2021-05-28 17:28:26 -07:00
Raphaël Vinot 1117ab6371 chg: add stats, avoid building big trees twice, bump deps 2021-05-26 18:25:06 -07:00
Raphaël Vinot 11f05626b5 chg: Improve module auto trigger 2021-05-19 15:03:07 -07:00
Raphaël Vinot 896f3c5f13 fix: properly check if capture is in the queue 2021-05-18 16:04:13 -07:00
Raphaël Vinot 925bb9d48e new: priority for captures 2021-05-18 14:58:56 -07:00
Raphaël Vinot bb214d9e2b chg: Use keywords only paramaters when relevant 2021-05-17 17:08:43 -07:00
Raphaël Vinot 5ee62d157f chg: Properly handle capture parents, avoid duplicates 2021-05-14 14:25:37 -07:00
Raphaël Vinot 1d5925d755 new: Keep captures parent, use extends in MISP export 2021-05-12 13:30:07 -07:00
Fafner [_KeyZee_] 97207b8f77
solving tor proxy for only_global_lookups 2021-05-04 17:17:21 +02:00
Raphaël Vinot 12697964ce chg: Improve tree stats 2021-04-27 17:09:58 +02:00
Raphaël Vinot 3389dcafd6 new: Add support for uwhoisd 2021-04-26 00:52:08 +02:00
Raphaël Vinot 39e894b7b0 fix: body hash page on direct access 2021-04-20 17:32:22 +02:00
Raphaël Vinot 9952e4de32 fix: remove UA - IP mapping from redis 2021-04-18 17:58:16 +02:00
Raphaël Vinot bf5d15baf0 fix: remove capture from index 2021-04-14 18:05:00 +02:00
Raphaël Vinot 7707d638cf new: Use async capture for the UI. 
Add a method to make sure splash is up before trying to capture.
 2021-04-08 19:15:53 +02:00
Raphaël Vinot 4d23d7ac36 chg: Remove extra code, return unziped resources 2021-04-08 01:05:11 +02:00
Raphaël Vinot e492428bf5 fix: Proper call to local cache 2021-04-03 02:24:38 +02:00
Raphaël Vinot 82db729e96 fix: cache captures index in main script 2021-04-03 02:06:32 +02:00
Raphaël Vinot b6ea1c24db new: Query capture status 2021-03-31 19:25:57 +02:00
Raphaël Vinot acb1bf2814 fix: MISP export when rendered node is empty 
Fix #182
 2021-03-30 14:06:34 +02:00
Raphaël Vinot 88bd450b59 chg: Improve tree page (menu and stats) 2021-03-25 18:32:48 +01:00
Raphaël Vinot fa6b4701c0 chg: update the cache at the right place. 2021-03-20 21:54:46 +01:00
Raphaël Vinot 2a55461286 new: Capture an URL on the rendered page, keep the session (WiP) 2021-03-19 17:52:20 +01:00
Raphaël Vinot cd7b050cb0 chg: rename and cleanup methods 2021-03-18 18:47:54 +01:00
Raphaël Vinot 25c1a6dbce chg: Use sorted cache whenever possible 2021-03-18 15:39:55 +01:00
Raphaël Vinot 49ae4490b2 new: (hidden) interface to search for hostnames and URLs 2021-03-18 00:40:14 +01:00
Raphaël Vinot 99402efa42 fix: allow to dump very big tree 2021-03-16 14:09:30 +01:00
Raphaël Vinot 1f9f5f1a9a chg: Improve url and hostnames search 2021-03-16 13:35:59 +01:00
Raphaël Vinot 8c656062b9 fix: Catch exception when the tree has too many redirects. 2021-03-12 18:20:17 +01:00
Raphaël Vinot 46aea0fe3a chg: Better error handling 2021-03-12 16:52:12 +01:00
Raphaël Vinot e35ceab3dd chg: Better logging and error handling 2021-03-08 13:57:55 +01:00
Raphaël Vinot e467f1d1eb chg: Catch exceptions when the capture fails bad. 2021-03-08 11:44:29 +01:00
Raphaël Vinot 984a930aa8 chg: Avoid 500 on missing UUID 2021-03-07 23:57:46 +01:00
Raphaël Vinot 39dd2021dd chg: Complete rework of the login system, add UI for MISP Push 2021-02-04 19:51:43 +01:00
Raphaël Vinot 610ffef43a new: Add VT objects to MISP export 2021-02-02 15:23:38 +01:00
Raphaël Vinot a68400dac7 chg: Add comments in MISP export 2021-02-02 15:23:38 +01:00
Raphaël Vinot fca3290ef0 fix: Make sure to skip last year's final week in the stats 2021-02-02 15:23:38 +01:00
Raphaël Vinot a7484e6cc4 chg: Improve MISP export 
* IPs of redirects
* default tags
* auto publish
 2021-02-01 18:09:32 +01:00
Raphaël Vinot 54ba67bb87 chg: Make lookyloo link org only in MISP export for private instances 2021-02-01 14:58:06 +01:00
Raphaël Vinot c358c08640 fix: Make mypy happy. 2021-01-28 19:28:54 +01:00
Raphaël Vinot acfb0d1c26 new: Add push to MISP feature 2021-01-28 18:37:47 +01:00
Raphaël Vinot d8d2c59f8b chg: Make mypy happy. 2021-01-25 13:14:33 +01:00
Raphaël Vinot 8bd874e2b0 new: Add variable width thumbnail method 2021-01-18 12:30:07 +01:00
Raphaël Vinot 72463cc898 chg: Slight cleanups and improvments 2021-01-14 17:28:59 +01:00
Raphaël Vinot 6149df06eb chg: Make the cache entries a dataclass 
Fix #99
 2021-01-14 17:12:23 +01:00
Raphaël Vinot 61baab1fbe fix: make mypy happy 2021-01-13 15:16:36 +01:00
Raphaël Vinot 4686b560dc chg: Better handling of insanely long webpages. 2021-01-13 14:33:20 +01:00
Raphaël Vinot 0d68844c90 chg: Refactoring and documenting 2021-01-12 17:22:51 +01:00
Raphaël Vinot 33326c2463 new: Extract URLs from rendered pages 2021-01-08 13:03:23 +01:00
Raphaël Vinot 2e0671fd96 fix: Properly handle new year in week stats 2021-01-04 12:34:00 +01:00
Raphaël Vinot 7241ae2ea1 fix: Do not fail on MISP export if there are no redirects 2020-12-24 15:43:56 +01:00
Raphaël Vinot 9b18370fe1 fix: Properly generate meta keys related to the User Agents 2020-12-22 19:16:46 +01:00
Raphaël Vinot fcae1d42d0 new: Show images in tooltip in hostnode popup 2020-12-22 18:23:39 +01:00
Raphaël Vinot 62cb5a9eb7 fix: Failure on new install 2020-12-11 10:26:42 +01:00
Raphaël Vinot ed6a53d211 chg: small visual improvments 2020-12-09 21:08:34 +01:00
Raphaël Vinot 3c6eca3567 new: Add screenshot thumbnail on tree, move links to the end of the node 2020-12-09 19:11:19 +01:00
Raphaël Vinot b73a3b7533 chg: Add attachements to MISP export 2020-12-08 14:59:34 +01:00
Raphaël Vinot fbbfa3082d chg: Fix typing 2020-12-07 20:54:33 +01:00
Raphaël Vinot 8eab287721 new: Initial implementation of MISP export 2020-12-07 20:50:46 +01:00
Raphaël Vinot 9b3d338d3c chg: Rename analisys->submission, add unique urls on stats graph 2020-12-07 13:55:03 +01:00
Raphaël Vinot d353463548 chg: Simplify modules activation 2020-12-03 12:33:35 +01:00
Raphaël Vinot 1a7a74cccd chg: Cleanup stats page 2020-11-27 16:27:29 +01:00
Fafner [_KeyZee_] 0e3d007bf5
fixing typo (again) 2020-11-25 12:22:45 +01:00
Fafner [_KeyZee_] 7bf13db3f2
fixing ttypo 2020-11-25 12:05:08 +01:00
Raphaël Vinot d964549e56 fix: Typing, dump to json 2020-11-24 21:01:30 +01:00
Fafner [_KeyZee_] d77dfb1cac
updating output for stats 2020-11-24 18:55:13 +01:00
Raphaël Vinot c54d29601d chg: Fix typing 2020-11-24 18:19:53 +01:00
Fafner [_KeyZee_] bedbb8eb45
adding get_stats() 2020-11-24 16:46:01 +01:00
Raphaël Vinot 9c382890f5 fix: Improve lokking on error in capture 2020-11-10 20:27:01 +01:00
Raphaël Vinot 354f269218 new: Integrate categorization in indexing 2020-11-09 16:02:54 +01:00
Raphaël Vinot 99c939fd7f chg: Rename scrape -> capture everywhere 
Related to #118
 2020-11-03 16:34:52 +01:00
Raphaël Vinot 69d34b803c fix: Duplicates on index. 2020-11-03 12:21:14 +01:00
Raphaël Vinot 8b1e3585ea chg: Improve initial caching. 2020-10-29 23:25:20 +01:00
Raphaël Vinot 69f65c9a87 chg: Update cache on tree visualization 2020-10-29 17:55:43 +01:00
Raphaël Vinot 5f4a97e11b fix: Typing meta 2020-10-29 14:19:17 +01:00
Raphaël Vinot 06a4c2cc87 fix: Speedup sorted cache 2020-10-29 14:06:38 +01:00
Raphaël Vinot da85aa0fb7 fix: Pre-sort entries on index 2020-10-29 13:29:13 +01:00
Raphaël Vinot 2802cfd46c new: Add captures categorization 2020-10-28 18:49:15 +01:00
Raphaël Vinot 451553e60d fix: Do not overwrite the urlnodes 2020-10-27 01:52:28 +01:00
Raphaël Vinot 068c92d5a4 new: API to query hostnames 2020-10-27 01:42:00 +01:00
Raphaël Vinot 39f88e9121 new: API to query URLs 2020-10-27 00:02:18 +01:00
Raphaël Vinot bdc0488e38 new: Hash lookup method 2020-10-23 20:51:15 +02:00