CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
701 Commits
7 Branches
43 Tags
110 MiB
Commit Graph

43 Commits (a8dc9231c6e53d83b7d404f19f674e2d44a6ee2d)

Author SHA1 Message Date
= 64ec44085d Dynamically choose correct port for LevelDB in script 2017-08-22 17:52:15 +02:00
Mokaddem 882ab88b00 Draft module + web link 2017-07-18 16:57:15 +02:00
Mokaddem e225090e07 Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample 2017-03-15 16:36:51 +01:00
Mokaddem adc37ab50e Indexer max size now in config 2017-03-15 12:14:41 +01:00
Olivier MEDOC 16044d4d36 pystemon-feeder: add the option bind in configuration file instead of address 
This is essentially because multiple feeds can be used in the address
configuration variable.
 2017-01-13 14:58:18 +01:00
Olivier MEDOC 9f9c265cb0 SentimentAnalysis: use lexicon file path from the ail configuration file 2017-01-11 11:07:08 +01:00
ptitdoc c0fac82031 Add pystemon path in configuration sample 2017-01-10 18:33:46 +01:00
Mokaddem 0e39f516a9 Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed. 2017-01-09 14:12:26 +01:00
Mokaddem e70b9cd15c Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them. 2016-12-23 10:31:26 +01:00
Mokaddem 7438f16c63 Added new config 2016-12-22 10:06:35 +01:00
Mokaddem 697812dbff Added config.sample 2016-08-23 09:59:39 +02:00
Mokaddem a4595010c0 Added whole config in config.sample 2016-08-09 14:23:36 +02:00
Mokaddem 1c40ad522f Merge branch 'master' into production 2016-08-08 09:26:58 +02:00
Raphaël Vinot 4aa484a91f Merge pull request #63 from mokaddem/uptodate-duplicate-module 
Uptodate duplicate module
 2016-07-25 09:44:49 +02:00
Raphaël Vinot ea8703e608 Fix path to tldsfile 2016-07-22 09:32:13 +02:00
Mokaddem 7c4f4a2f79 Merge branch 'statistics-v2' into uptodate-statistics-v2 2016-07-21 15:16:03 +02:00
Mokaddem b685f81752 Merge branch 'duplicate-module' into uptodate-duplicate-module 2016-07-21 14:59:52 +02:00
Mokaddem 0ab7560477 Added top_progression chart for tld, domain and scheme + Small modification in config file. 2016-07-21 13:44:22 +02:00
Raphaël Vinot ede46ec799 Fix default config. 2016-07-21 09:19:13 +02:00
Raphaël Vinot 781b441bb8 Fix path in config 2016-07-20 14:12:50 +02:00
Raphaël Vinot 8a69110c8e Merge branch 'webpage-splitting' of https://github.com/mokaddem/AIL-framework into mokaddem-webpage-splitting 2016-07-19 16:35:31 +02:00
Mokaddem 6805ed6488 Added default config 2016-07-18 15:52:53 +02:00
Mokaddem 4f6813350b Added two new version of duplicate module. 
One with hashes are saved in json on disk
The other with only leveldb
 2016-07-18 15:50:41 +02:00
Mokaddem 2383db022f Added default configuration 2016-07-15 09:10:44 +02:00
Mokaddem 9a9e07f600 Added default configuration 2016-07-15 09:08:38 +02:00
Raphaël Vinot aef8ab0411 Listen locally for 0MQ 2016-02-04 15:32:50 +01:00
Raphaël Vinot 08ceefc375 Re-add config option 2014-12-22 16:50:25 +01:00
Raphaël Vinot 50369c6706 Revert changes on the config file due to merging messup 2014-12-22 16:29:05 +01:00
Raphaël Vinot f717f9fe89 Merge branch 'master' of github.com:CIRCL/AIL-framework 2014-12-22 15:32:48 +01:00
Raphaël Vinot 8803c8447a Publish the fetched onions on a ZMQ feed. 2014-09-30 16:55:16 +02:00
Alexandre Dulaunoy 25757b0fff A simple feeder script feeding data from pystemon to AIL. 
The configuration matches the default Redis parameters used
in the pystemon configuration.

https://github.com/cvandeplas/pystemon/blob/master/pystemon.yaml#L16
 2014-09-19 14:03:05 +02:00
Raphaël Vinot 65b9a01644 Add config file for DomainClassifier, proper reporting 2014-09-17 17:22:56 +02:00
Alexandre Dulaunoy de6e21d5a7 DomainClassifier sample configuration added 2014-09-08 16:44:05 +02:00
Raphaël Vinot fca00beed9 Add Domain Classifier module. 
Cleanup in the config files.
 2014-09-05 10:41:00 +02:00
Raphaël Vinot abfe13436b Big refactoring, make the queues more flexible 2014-08-29 19:37:56 +02:00
Raphaël Vinot 623e876f3b Cleanup. 
* Remove useless subscriber
* Fix typo in the config file
* Update Helper accordingly
 2014-08-26 17:36:57 +02:00
Raphaël Vinot 50cfac857e Update config 
Make all paths in the config file relative to the home directory.
 2014-08-20 16:00:56 +02:00
Raphaël Vinot a68f5b6a0e fix subscriber names, update default config 2014-08-20 15:54:21 +02:00
Raphaël Vinot 45b0bf3983 Improve the cleanup. Still some to do. 2014-08-19 19:07:07 +02:00
Alexandre Dulaunoy 762def3a23 Log where URLs are hosted - cc_critical option added 
It logs where the hostname of the URL is hosted (ASN and geographic location).
A simple option cc_critical added to set the country code to log as critical.
 2014-08-14 14:22:11 +02:00
Starow 079db6f80c Hardcoded path from ZMQ_Curve are now referring correctly in config.cfg.sample fix #6 2014-08-11 11:33:18 +02:00
Alexandre Dulaunoy 7bdd4a41a5 Indexer module added - initial version with Whoosh full-text indexer 
The indexer module indexes all the pastes using Whoosh. The module
can be extended to support additional full-text indexers in the future.
 2014-08-11 11:04:09 +02:00
Starow 1379ef705a Initial import of AIL framework - Analysis Information Leak framework 
AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past
ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen
sitive information
 2014-08-06 11:43:40 +02:00