CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
415 Commits
7 Branches
44 Tags
110 MiB
Commit Graph

188 Commits (f9bf00f02255c28e7688f27b7e082e79488e72bd)

Author SHA1 Message Date
Mokaddem feb1717dc6 fixed size bug, which in some cases display the total size instead of the average + few cleanup + commented out log scale in moduleStats.js and added large number division in case of div by 0 2016-07-29 16:52:50 +02:00
Mokaddem e8fae17cfd Fixed small race condition bug in ModuleStat.py and Added highlight of bar belonging to the same serie in moduleStats.js 2016-07-29 10:36:44 +02:00
Mokaddem 240bdd9cca Added number of paste by provider in module stats 2016-07-27 09:27:32 +02:00
Mokaddem baa042827d Removed useless comments + prevent trying to add an already present element in the set 2016-07-26 11:06:31 +02:00
Mokaddem cc1b49baaf Fixed variable bug in moduleStats and added draft of credential stats 2016-07-26 10:45:02 +02:00
Mokaddem 06be1f129a Added Draft of ModuleStats + Paste size average 2016-07-25 16:38:57 +02:00
Raphaël Vinot 4aa484a91f Merge pull request #63 from mokaddem/uptodate-duplicate-module 
Uptodate duplicate module
 2016-07-25 09:44:49 +02:00
Mokaddem 34a00abdc2 Merge branch 'uptodate-duplicate-module' into production 2016-07-22 10:31:10 +02:00
Mokaddem c686f69ca6 Clean not compatible ssdeep hash encountered 2016-07-22 10:04:58 +02:00
Raphaël Vinot ea8703e608 Fix path to tldsfile 2016-07-22 09:32:13 +02:00
Mokaddem dad05073fd Fixed bug introduced in merge-conflict 2016-07-22 09:25:05 +02:00
Mokaddem 7c4f4a2f79 Merge branch 'statistics-v2' into uptodate-statistics-v2 2016-07-21 15:16:03 +02:00
Mokaddem b685f81752 Merge branch 'duplicate-module' into uptodate-duplicate-module 2016-07-21 14:59:52 +02:00
Mokaddem 0ab7560477 Added top_progression chart for tld, domain and scheme + Small modification in config file. 2016-07-21 13:44:22 +02:00
Raphaël Vinot ede46ec799 Fix default config. 2016-07-21 09:19:13 +02:00
Raphaël Vinot 781b441bb8 Fix path in config 2016-07-20 14:12:50 +02:00
Raphaël Vinot 620d69467f Add WebStats in launcher 2016-07-20 14:12:37 +02:00
Raphaël Vinot 62eef44ca8 Fix pep8 2016-07-20 14:12:18 +02:00
Raphaël Vinot 8a69110c8e Merge branch 'webpage-splitting' of https://github.com/mokaddem/AIL-framework into mokaddem-webpage-splitting 2016-07-19 16:35:31 +02:00
Mokaddem a6996c0b23 Added related functions and display of duplicated paste in search.py 2016-07-19 10:48:44 +02:00
Mokaddem 996c0e02de Duplicate module takes its messages from other modules and no more from Global. 2016-07-18 16:22:33 +02:00
Mokaddem 6805ed6488 Added default config 2016-07-18 15:52:53 +02:00
Mokaddem 4f6813350b Added two new version of duplicate module. 
One with hashes are saved in json on disk
The other with only leveldb
 2016-07-18 15:50:41 +02:00
Mokaddem 14e9850dd6 Added new module for Duplicate paste. Seems working but has some small bug (re-check same paste twice) 2016-07-15 16:58:48 +02:00
Mokaddem 2383db022f Added default configuration 2016-07-15 09:10:44 +02:00
Mokaddem 9a9e07f600 Added default configuration 2016-07-15 09:08:38 +02:00
Mokaddem 0332f23579 Added SimHash library 2016-07-15 08:56:16 +02:00
Mokaddem fba14bfb4b In index: Added number of processed pastes chart 2016-07-14 11:16:07 +02:00
Mokaddem ab61e32399 Commented out get_language because it adds too much overhead 2016-07-14 11:15:15 +02:00
Mokaddem 594d2def35 In index: Added number of processed pastes chart 2016-07-13 15:57:33 +02:00
Mokaddem 56b6659d8b Commented out get_language because it adds too much overhead 2016-07-13 08:59:48 +02:00
Mokaddem c51bdec8aa Merge branch 'mokaddem-testing' 2016-07-12 11:53:24 +02:00
Mokaddem 8a1247cf5d modified variable name str 2016-07-12 11:52:19 +02:00
Mokaddem 7e5ce0f17f Optimized create_plot and removed test commemts 2016-07-12 11:47:51 +02:00
Mokaddem 465244e1ce Added dynamic table sorting in search page. (Still need to add dependencies) 2016-07-07 16:38:00 +02:00
Mokaddem 7ff9b9a583 Added DomainTrending seems working. 
Started search features with related html pages, not finish yet.
 2016-07-05 16:53:03 +02:00
Mokaddem 5a9eca9291 Added few comments 2016-07-04 09:18:23 +02:00
Mokaddem 4b3101b7b6 Added template tld. Modified URL using Faup and refactored WebStats. 2016-07-01 16:59:08 +02:00
Mokaddem beeeb76de9 Added new modules and started WebTrending web interface 2016-06-30 14:38:28 +02:00
Mokaddem 3dc356dc5e Getting Started: Initial configuration working on the laptop 2016-06-30 14:36:47 +02:00
MaximeStor ab66cd255a Improve SourceCode, keywords and add description in /doc 2016-03-12 12:30:38 +01:00
MaximeStor 701d771aa5 Add first version of Source Code 2016-03-12 11:21:29 +01:00
Raphaël Vinot be86737ca7 Because 0MQ fails. 2016-03-11 16:16:53 +01:00
Raphaël Vinot f6e4ea2270 Fix logging, fix URL regex 2016-02-11 12:19:03 +01:00
Raphaël Vinot d160e4a2c8 Add Credential in the scripts to launch 2016-02-10 17:33:16 +01:00
Raphaël Vinot 90e1b25426 Split filepath and count in credential module 2016-02-10 17:31:52 +01:00
Raphaël Vinot 4895ee9fa2 Add new category (Credential) 2016-02-10 16:39:56 +01:00
Raphaël Vinot 1da8675750 Refactoring on Credential, Phone and Release 2016-02-10 16:39:06 +01:00
Alexandre Dulaunoy c68136b04b Merge branch 'pgp' of https://github.com/Rafiot/AIL-framework 
Conflicts:
	bin/packages/modules.cfg
 2016-02-08 10:13:44 +01:00
MaximeStor b7d2b64a86 Merge branch 'master' of https://github.com/CIRCL/AIL-framework into module 2016-02-06 11:28:48 +01:00
Alexandre Dulaunoy 192ee7f0ec Merge pull request #49 from Alainfou/master 
Phone module added (regex needs optimization)
 2016-02-06 11:20:46 +01:00
Alain ea52fd1068 Phone regex updated 
Might still need to be fixed / optimized, in case of maths or random numbers starting with a 0. Do not capture dates, hours, coordinates anymore. Captured formats are: e.g. +331234567890 ; 09 12 34 56 78 ; +4177/123.45.69 ; +352(0)6-23-23-23...
 2016-02-05 20:58:02 +01:00
Alain fabbfd8ae9 Update module.cfg (adding Keys and Phone section) 2016-02-05 14:00:41 -05:00
Alain 43b3556588 Starting Phone number recognition 2016-02-05 13:58:21 -05:00
MaximeStor 07513a5b37 Add modules Credential and Release 2016-02-05 16:15:09 +01:00
Raphaël Vinot 9171d5b118 Add module to find PGP encrypted blobs 2016-02-05 16:03:37 +01:00
Raphaël Vinot aef8ab0411 Listen locally for 0MQ 2016-02-04 15:32:50 +01:00
Raphaël Vinot 5ca13c42eb Launch redis and leveldb from local directory 2016-02-04 15:24:39 +01:00
Raphaël Vinot 12aca6b760 Add script to import from local directory, use local python from env 2016-02-04 15:22:51 +01:00
Raphaël Vinot 315cb48117 Add template file for writing a new module 2016-02-03 10:33:42 +01:00
Raphaël Vinot 0d6adc2063 Add initial Travis file 2016-01-19 11:43:34 +01:00
Alexandre Dulaunoy cdd0725e88 -v option added to list the path 2015-12-22 21:37:05 +00:00
Alexandre Dulaunoy e3971ac93a Onion fetching loop deactivated by default 2014-12-22 16:06:38 +00:00
Raphaël Vinot 08ceefc375 Re-add config option 2014-12-22 16:50:25 +01:00
Raphaël Vinot 50369c6706 Revert changes on the config file due to merging messup 2014-12-22 16:29:05 +01:00
Raphaël Vinot f717f9fe89 Merge branch 'master' of github.com:CIRCL/AIL-framework 2014-12-22 15:32:48 +01:00
Raphaël Vinot 9ee61db2cf Add hotfixes 2014-12-22 15:27:02 +01:00
Raphaël Vinot 8803c8447a Publish the fetched onions on a ZMQ feed. 2014-09-30 16:55:16 +02:00
Alexandre Dulaunoy 25757b0fff A simple feeder script feeding data from pystemon to AIL. 
The configuration matches the default Redis parameters used
in the pystemon configuration.

https://github.com/cvandeplas/pystemon/blob/master/pystemon.yaml#L16
 2014-09-19 14:03:05 +02:00
Raphaël Vinot 65b9a01644 Add config file for DomainClassifier, proper reporting 2014-09-17 17:22:56 +02:00
Alexandre Dulaunoy 27b134ec03 Add proper publisher for classified domains/hostnames 2014-09-10 09:27:47 +02:00
Raphaël Vinot f017680365 fix onions, cc and domain classifier modules 2014-09-08 16:51:43 +02:00
Alexandre Dulaunoy de6e21d5a7 DomainClassifier sample configuration added 2014-09-08 16:44:05 +02:00
Alexandre Dulaunoy 246621f663 First version of the DomainClassifier 2014-09-08 16:43:21 +02:00
Alexandre Dulaunoy 1397db9691 Global queue for DomainClassifier 2014-09-08 11:07:45 +02:00
Raphaël Vinot e983c839ad Categ now listen to the Global queue 2014-09-05 17:05:45 +02:00
Raphaël Vinot 46f27ada4e More cleanup 2014-09-05 10:42:01 +02:00
Raphaël Vinot fca00beed9 Add Domain Classifier module. 
Cleanup in the config files.
 2014-09-05 10:41:00 +02:00
Raphaël Vinot b7c9e489c9 Fix the exceptions 2014-09-04 11:46:07 +02:00
Raphaël Vinot 9e8611a42d stop killing the disk when creating the word curve 2014-09-02 18:20:28 +02:00
Raphaël Vinot 7542eaf739 Update starting script. 2014-09-02 15:21:36 +02:00
Raphaël Vinot 0c6b09f379 Fix the onion module, log the valid onions. 2014-09-01 16:18:06 +02:00
Raphaël Vinot f4b89669fc The onion module now fetches the URLs it finds. 2014-08-31 22:42:12 +02:00
Raphaël Vinot abfe13436b Big refactoring, make the queues more flexible 2014-08-29 19:37:56 +02:00
Raphaël Vinot 623e876f3b Cleanup. 
* Remove useless subscriber
* Fix typo in the config file
* Update Helper accordingly
 2014-08-26 17:36:57 +02:00
Alexandre Dulaunoy 3b499a2ec8 ZMQ Publisher removed 
ZMQ Publisher removed to allow concurrent use of the scripts.
In short term, we would replace all publishing part within AIL
into pub-sub Redis to avoid ZMQ limitation.
 2014-08-26 14:38:49 +02:00
Alexandre Dulaunoy f070ac2005 cymruwhois uses dotted decimal format 2014-08-25 10:05:36 +02:00
Raphaël Vinot 3886d1b834 Small fixes to make the refactoring production ready 
* the port for the logging is 6380
* use os.environ properly
* fix typos
 2014-08-22 17:35:40 +02:00
Raphaël Vinot 78125db4ea Use env variables everywhere 2014-08-22 14:52:02 +02:00
Raphaël Vinot 277d138a5d cleanup, add FIXME 2014-08-21 14:39:17 +02:00
Raphaël Vinot 63b29176c1 move Redis_Data_Merging to Paste 2014-08-21 12:22:07 +02:00
Raphaël Vinot 50cfac857e Update config 
Make all paths in the config file relative to the home directory.
 2014-08-20 16:00:56 +02:00
Raphaël Vinot a68f5b6a0e fix subscriber names, update default config 2014-08-20 15:54:21 +02:00
Raphaël Vinot 2485ba5df2 Merge remote-tracking branch 'origin/master' into testing 
Conflicts:
	bin/ZMQ_Sub_Urls.py
 2014-08-20 15:24:10 +02:00
Raphaël Vinot 99c8cc7941 completely remove ZMQ_PubSub.py 2014-08-20 15:14:57 +02:00
Alexandre Dulaunoy 1d64dc44c8 MIME type guessing - removed one duplicate call to libmagic 2014-08-20 10:22:33 +02:00
Raphaël Vinot 8d9ffbaa53 Do not create a ZMQ sub if it is not required. 2014-08-19 19:53:33 +02:00
Raphaël Vinot 45b0bf3983 Improve the cleanup. Still some to do. 2014-08-19 19:07:07 +02:00
Raphaël Vinot f1753d67c6 Cleanup the queues. 2014-08-19 16:05:37 +02:00
Alexandre Dulaunoy e8fcea6cd6 Remove undeclared variable 2014-08-18 16:17:36 +02:00