CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
1,959 Commits
7 Branches
43 Tags
110 MiB
Commit Graph

53 Commits (9974823464dce4d344f6beadee0f14ce9dccef64)

Author SHA1 Message Date
Terrtia 8754350d39
fix: [crawler] user agent + splash restart 2021-03-26 11:30:06 +01:00
Terrtia 38a63c69f6
fix: [crawler] typo 2021-03-05 18:56:31 +01:00
Terrtia ad3b6c44bc
fix: [crawler] typo 2021-03-05 18:54:21 +01:00
Terrtia 6daa750e3b
fix: [Crawler] faup 2021-03-05 18:47:38 +01:00
Terrtia 263d3e7bca
fix: [cralers] remove debug 2021-03-05 18:03:15 +01:00
Terrtia 1f94c1c693
chg: [splash manager] update enpoints + use Splash name to restart docker 2021-03-04 09:26:28 +01:00
Terrtia d8b7ab4de5
chg: [crawler_manager] UI edit config + fix crawler queues 2020-08-24 22:31:41 +02:00
Terrtia 3ea14b29b8
chg: [crawler] show all crawlers type on dashboard 2020-08-17 21:52:57 +02:00
Terrtia 39c3918d09
chg: [crawler] manage crawlers 2020-07-27 15:46:09 +02:00
Terrtia c31aae4efc
chg: [crawler] crawler queue + restart docker on error 2020-07-24 08:54:54 +02:00
Terrtia 41cacf7129
chg: [crawler manager] get all splash dockers, proxies and launch all crawlers 2020-06-09 18:33:41 +02:00
Terrtia 5f289f04f3
chg: [Crawler core + UI] crawler lua: handle retry + fix cookie loader and selector 2020-03-30 18:43:50 +02:00
Terrtia 1c45571042
chg: [crawler] add cookies list by user/global, save cookies from file + dict(name, value), TODO: API + handle errors 2020-03-23 18:00:09 +01:00
Terrtia bb03ef532b
chg: [Correlation UI] add correlation blueprint + UI graph correlation 2019-11-14 17:05:58 +01:00
Terrtia 09ecc4d93f
chg: [Crawler] add default crawler config + update default user_agent 2019-07-24 10:18:10 +02:00
Terrtia 26a4c7fd2c
fix: [Crawler] incorrect config 2019-07-10 09:42:20 +02:00
kovacsbalu 7765ab92e0 Hopp, single quote :) 2019-05-15 10:00:51 +02:00
kovacsbalu 6092f482e6 Fix crawler rotation 
Before this, crawler processed prioritized onions and after all starts prioritized regular.
 2019-05-15 09:57:18 +02:00
Terrtia a4c03b4ba4
fix: [Crawler] force domains/subdomains lower case (rfc4343) 2019-05-06 11:46:20 +02:00
Terrtia fc2c1422ff
fix: [Crawler] unpack_url 2019-04-25 13:54:06 +02:00
Terrtia 2a1cd4a009
chg: [Onion, crawler config] auto crawler: add config by url, fix onions tagging + filter subdomains 2019-04-23 11:15:34 +02:00
Terrtia 6fdf7c2123
chg: [UI crawler] status/remove auto crawler 2019-04-18 16:57:51 +02:00
Terrtia f64c385343
chg: [Crawler] handle port: crawling + history 2019-03-22 16:48:07 +01:00
Terrtia c0d72e7d2a
chg: [Crawler UI] Crawler major refractor (end) + basic UI for manual crawler 2019-02-26 14:50:48 +01:00
Terrtia 7b32d7f34e
chg: [Crawler] major refractor 2019-02-25 16:38:50 +01:00
Terrtia 60f7645ac1
chg: [Crawler] refractor 2019-02-22 17:00:24 +01:00
Terrtia e5dca268a8
chg: [Crawler] refractor 2019-02-21 09:54:43 +01:00
Terrtia da78d0552d
chg: [Crawler UI Tags] add tag by day + add crawler status + UI onion blacklist 2019-02-19 11:41:45 +01:00
Terrtia c2885589cf
chg: [UI] basic navbar + sidebar + refractor 2019-02-07 17:22:44 +01:00
Terrtia 516238025f
chg: [Crawler] add bootsrap4 src + refractor crawler 2019-02-05 17:16:44 +01:00
Terrtia 92d192238b
fix: [Crawler] change max page crawled 2019-01-29 17:04:45 +01:00
Terrtia 6c7086f4eb
fix: [Crawler] first_seen 2019-01-29 16:54:39 +01:00
Terrtia 88eaaeae93
chg: [Crawler] add priority queue, fix #263 2019-01-29 16:08:59 +01:00
Terrtia c1b34bd99c
fix: [Crawler] limit max crawled pages 2019-01-29 15:38:00 +01:00
Terrtia 2dc0eca4a9
fix: [Crawler] fix crawler cache info 2019-01-29 12:09:19 +01:00
Terrtia bb301a870c
fix: [Crawler] fix onion blacklist + add crawler info 2019-01-29 12:00:14 +01:00
Terrtia f842194c57
fix: [Crawler] retry when splash is not available 2018-12-17 16:04:12 +01:00
Terrtia 6328cc22b7
chg: [Crawler] add domains blacklist 2018-09-28 16:29:09 +02:00
Terrtia 82e6df4b94
chg: [Crawler] domains stats + logs + clean 2018-09-28 15:23:27 +02:00
Terrtia e357dce59b
fix: [Crawler] detect splash connection to proxy error 2018-09-27 15:43:03 +02:00
Terrtia c49e871ba8
chg: [crawler] add infos 2018-09-26 16:34:27 +02:00
Terrtia 874824a589
fix: [Crawler] clean 2018-09-24 16:28:55 +02:00
Terrtia 8eca0e0778
fix: [Crawler] clean 2018-09-24 16:24:30 +02:00
Terrtia 50c81773e9
chg: [Crawler] add launcher and install 2018-09-24 16:23:14 +02:00
Terrtia 5b31b6e853
fix: [Crawler] save domain to crawl on splash error 2018-09-18 16:20:13 +02:00
Terrtia f5b648d72a
pixelate paste screenshot 2018-09-18 11:03:40 +02:00
Terrtia 6f0817365a
chg: [Crawler UI] display domain information 2018-09-12 09:55:49 +02:00
Terrtia ced0b1e350
chg: [I2P] add default config 2018-08-24 10:24:03 +02:00
Terrtia 7e24943537
chg: [Crawler] crawler accept all kind of domains 2018-08-24 10:13:56 +02:00
Terrtia e9580d6775
chg: [Crawler] change BDD, save i2p links 2018-08-21 15:54:53 +02:00