CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
41 Commits
7 Branches
43 Tags
110 MiB
Commit Graph

41 Commits (0a6664ffbab58919f6766e2f3ac9d1e3ee49cd66)

Author SHA1 Message Date
Alexandre Dulaunoy 0a6664ffba Indexer: Some index statistics added 
usage: indexer_lookup.py [-h] [-q Q] [-n] [-t] [-l]

Fulltext search for AIL

optional arguments:
  -h, --help  show this help message and exit
  -q Q        query to lookup (one or more)
  -n          return number of indexed documents
  -t          dump top 500 terms
  -l          dump all terms encountered in indexed documents
 2014-08-11 15:07:12 +02:00
Alexandre Dulaunoy f65a94d47b -l added -> dumping all terms indexed 2014-08-11 14:56:15 +02:00
Alexandre Dulaunoy f3d1ca052e Return the number of indexed documents 2014-08-11 14:50:35 +02:00
Jules 12dd16daf9 Merge pull request #10 from adulau/master 
Whoosh requirements added for Indexer modules
 2014-08-11 14:27:43 +02:00
Alexandre Dulaunoy 611d2a466f Configuration that should not be there... 2014-08-11 14:24:27 +02:00
Alexandre Dulaunoy 38860db284 Whoosh requirements added 2014-08-11 14:13:36 +02:00
Alexandre Dulaunoy 72b232366b Whoosh added for the indexer modules 2014-08-11 14:13:09 +02:00
Alexandre Dulaunoy 2b8f2689bf Indexer queue and script added to "BBS-like" LAUNCH script 2014-08-11 14:06:52 +02:00
Alexandre Dulaunoy 006d4b130e Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2014-08-11 13:48:32 +02:00
Jules d48ad07145 Merge pull request #9 from adulau/master 
Indexer module: script to query the index
 2014-08-11 13:47:34 +02:00
Alexandre Dulaunoy 9657c6bf80 Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2014-08-11 13:46:37 +02:00
Starow 80fe77da26 Making bigger the graph 2014-08-11 13:46:02 +02:00
Starow a7679787b7 Removing "Dashboard" Title to win some place for displaying infos.. 2014-08-11 13:41:58 +02:00
Alexandre Dulaunoy b1053af3cd Indexer module: script to query the index 
Test script to query the index generated from the Indexer module.

python indexer_lookup.py -q Visa -q Mastercard
 2014-08-11 12:03:27 +02:00
Starow 079db6f80c Hardcoded path from ZMQ_Curve are now referring correctly in config.cfg.sample fix #6 2014-08-11 11:33:18 +02:00
Jules 30640ab3c6 Merge pull request #8 from adulau/master 
Indexer module added - initial version with Whoosh full-text indexer
 2014-08-11 11:16:44 +02:00
Starow edd4ea9624 Queue name tab moved to left side of the dashboard 2014-08-11 11:15:44 +02:00
Alexandre Dulaunoy 7bdd4a41a5 Indexer module added - initial version with Whoosh full-text indexer 
The indexer module indexes all the pastes using Whoosh. The module
can be extended to support additional full-text indexers in the future.
 2014-08-11 11:04:09 +02:00
Starow 2306783882 Updating dashboard navigation menu list 2014-08-11 10:48:03 +02:00
Starow f3eb02c5b7 Updating installing script + adding another required package to the list 2014-08-11 10:41:50 +02:00
Starow 5bc0f154f9 Update of installing_deps.sh and pip_requirement.txt 2014-08-11 09:36:28 +02:00
Starow d1d4b2ebe0 Importing dns.exeption fix #4 fix #7 2014-08-11 09:27:50 +02:00
Starow 192074e569 Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2014-08-11 09:21:09 +02:00
Starow a5c1d59d29 Catching the exception dns.exception.Timeout fix #7 2014-08-11 09:18:55 +02:00
Starow 54091a2174 Catching the exception dns.exception.Timeout fix #4 2014-08-11 09:08:28 +02:00
Starow eb603e8762 Fixing a bug about caching paste inside Redis :) 2014-08-08 17:23:51 +02:00
Starow 7a1db94f9e Adding a letter (s) 2014-08-08 17:19:42 +02:00
Starow 043800287a adding a . 2014-08-08 17:18:03 +02:00
Starow bf682c4b44 Fixing last commit ... 2014-08-08 17:13:18 +02:00
Starow 503c23ca3b Fixing last commit 2014-08-08 17:08:41 +02:00
Starow c9e1eaf182 Improving cache code 2014-08-08 17:04:25 +02:00
Starow 0a503db815 Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2014-08-08 16:52:42 +02:00
Starow 44addf1afe Redis cache added fix #5 
The paste will be add in Redis during 5min and also saved on disk.
Now if a module want to get the paste for further processing, it will first try to get it in the cache
instead of getting it directly on the disk and wasting I/O.
 2014-08-08 16:48:02 +02:00
Alexandre Dulaunoy b21e7ea3c8 Merge branch 'master' of github.com:CIRCL/AIL-framework 2014-08-08 14:03:30 +02:00
Alexandre Dulaunoy 81ea9cfb3c Clarification for the env variable 2014-08-08 14:03:00 +02:00
Starow af827a3402 Removing the hardcoded IP adress 2014-08-08 13:49:38 +02:00
Alexandre Dulaunoy dd0739be43 README fixed for bare bone install 2014-08-08 11:42:51 +02:00
Starow 0fe6c4d805 Editing redis .conf file 
I change the dir as it was originally to avoid editing these 3 files during the installation
 2014-08-08 09:39:08 +02:00
Starow 97f3a3df9e update pubsublogger with the last version 2014-08-07 14:49:34 +02:00
Starow c10003a630 Changing ZMQ Curve Module comment 2014-08-07 14:46:43 +02:00
Starow 1379ef705a Initial import of AIL framework - Analysis Information Leak framework 
AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past
ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen
sitive information
 2014-08-06 11:43:40 +02:00