CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
1,635 Commits
6 Branches
43 Tags
110 MiB
Commit Graph

801 Commits (4a732ea9f3fa0cd13f3af1289bc309897b4e7521)

Author SHA1 Message Date
Mokaddem da081bae42 Merge branch 'regexInTerms' into production 2017-04-18 16:59:40 +02:00
Mokaddem 2d907ae784 Merge branch 'newIndexer' into auto-conf 2017-04-18 16:20:00 +02:00
Mokaddem b0f1cf84b7 remplaced usage of global variable by a function param 2017-04-18 16:02:22 +02:00
Mokaddem 6e8509a0ba Removed wierd line 2017-04-18 15:57:07 +02:00
Mokaddem c8baabd882 Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...) 2017-04-18 15:28:21 +02:00
Mokaddem 2da4c572c7 New feature in Terms: regex and set of words support (draft) 2017-03-28 17:42:44 +02:00
Mokaddem 6e80eb0954 Typo 2017-03-28 09:55:59 +02:00
Mokaddem c79d4d65c8 Keep original section order while updating 2017-03-28 09:54:24 +02:00
Mokaddem e225090e07 Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample 2017-03-15 16:36:51 +01:00
Mokaddem be93af6f5a Merge branch 'terms-improvements' 2017-03-15 14:55:44 +01:00
Mokaddem 3261eca836 Fixed all_index newline bug + added print of what's happening 2017-03-15 14:29:49 +01:00
Mokaddem 5b7d047622 Changed os.system by os.mkdir + fixed bug during rotation 2017-03-15 14:05:13 +01:00
Mokaddem adc37ab50e Indexer max size now in config 2017-03-15 12:14:41 +01:00
Mokaddem 4005b72180 Added backward support + renamed index with timestamp 2017-03-15 11:51:35 +01:00
Mokaddem 248469d61e Indexer now create an index_dir when it became too large (search in index not updated yet) 2017-03-14 10:37:31 +01:00
Alexandre Dulaunoy 1540b719ed Merge pull request #108 from mokaddem/newModuleInformation 
New version of ModuleInformation
 2017-03-14 08:29:09 +01:00
Mokaddem 3c9cdac39a Merge branch 'terms-improvements' into production 2017-02-28 16:51:39 +01:00
Mokaddem dadc872f9f Merge branch 'newModuleInformation' into production 2017-02-28 16:51:08 +01:00
Mokaddem a2849c1bcd Added a catch for psutils if process get stopped by user 2017-02-28 16:13:01 +01:00
Mokaddem 4071bf2e91 fixed a bug in manage_term (usage of loop.index0) + fixed bug in top_set where graph was not displaying the correct per_paste value 2017-02-28 15:01:48 +01:00
Mokaddem a1c47d7a13 Merge branch 'production' into terms-improvements 2017-02-28 11:15:56 +01:00
Mokaddem fd4feb374f Merge branch 'newModuleInformation' into production 2017-02-28 11:12:06 +01:00
Mokaddem 75b4037519 Fixed bug - If we move the cursor from a line that has dissapear 2017-02-28 11:11:17 +01:00
Mokaddem d6e02591af Merge branch 'newModuleInformation' into production 2017-02-28 10:23:45 +01:00
Mokaddem 975c1a37d9 ModuleInformation create the all_modules files if missing 2017-02-28 10:23:22 +01:00
Mokaddem ccdf256383 Added possibility to show or not in dashboard text that is logged 2017-02-28 10:21:27 +01:00
Mokaddem 39104dea5e Merge branch 'bugfix_categ' into production 2017-02-28 09:56:25 +01:00
Mokaddem 26eb5fdff0 Fixed bug closing parenthesis in categ + changed behavior of sending to browseWarningPaste in Mail 2017-02-28 09:23:32 +01:00
Mokaddem 4a0dac1fae Merge branch 'production' into terms-improvements 2017-02-15 16:31:22 +01:00
Mokaddem a696ccf83b Merge branch 'newModuleInformation' into production 2017-02-15 16:30:44 +01:00
Mokaddem 85a6f6fa63 Fixed a bug (str+float) 2017-02-15 16:30:31 +01:00
Mokaddem 25adae0104 Added possibility to select in top_terms the data per_paste or not (need module testing) 2017-02-15 16:29:02 +01:00
Mokaddem e98e92b0d4 Merge branch 'fix-linkPaste-debug' into production 2017-02-15 13:43:03 +01:00
Mokaddem c552aced42 Merge branch 'newModuleInformation' into production 2017-02-15 13:42:46 +01:00
Mokaddem 1b0d9d9507 Fixed str and int concat + Added sorting by queue name in queue not running 2017-02-15 13:42:21 +01:00
Mokaddem 74e0bc7acb Merge branch 'newModuleInformation' into production 2017-02-15 11:22:50 +01:00
Mokaddem 7fc9ba18d7 Added support of ENTER for navigation 2017-02-15 11:01:11 +01:00
Mokaddem b84194542d Moved logging into a function 2017-02-15 10:57:41 +01:00
Mokaddem 0fce3f5d65 Renamed variables + added comments 2017-02-15 10:37:59 +01:00
Mokaddem 9ca3e79e70 Removed useless comments + commented 2017-02-15 10:19:03 +01:00
Mokaddem 9199ea3f0e Added comments 2017-02-15 10:04:51 +01:00
Mokaddem ff1df07b5d Added comments and moved portion of code for slightly better visibility 2017-02-15 09:43:18 +01:00
Mokaddem b272b1b07b Added a way to quit application with 'q' more or less properly 2017-02-15 09:35:03 +01:00
Mokaddem 0254c51164 Added end of paste marker + handling screen resize 2017-02-15 08:26:48 +01:00
Mokaddem 0b9609e4a9 Merge branch 'newModuleInformation' into production 2017-02-14 17:22:06 +01:00
Mokaddem 444f6071b7 Fixed screen bug by adding an input before starting asciimatics + improved path localisation in the logs from modules 2017-02-14 17:21:52 +01:00
Mokaddem b04bac4731 Merge branch 'newModuleInformation' into production 2017-02-14 16:08:42 +01:00
Mokaddem d520c45a15 Added support of display of the last paste processed by a module 2017-02-14 16:05:59 +01:00
Mokaddem d3b4409156 Added path for debug in SQLInjection 2017-02-14 11:23:34 +01:00
Mokaddem 135a1457e0 Merge branch 'fix-linkPaste-debug' into production 2017-02-14 11:12:22 +01:00
Mokaddem d4da3a316f Added path of the paste in the log of Categ.py 2017-02-14 10:59:47 +01:00
Mokaddem 303575b2a1 Merge remote-tracking branch 'upstream/master' into production 2017-02-14 09:31:18 +01:00
Alexandre Dulaunoy 2f4e5b9016 Merge branch 'master' of github.com:CIRCL/AIL-framework 2017-01-26 07:16:44 +00:00
Alexandre Dulaunoy cf903cc212 (partially fix #90 too) using a simple Alarm (SIGNAL) when exec-timeout 
Introducing a timer (in this case 60 seconds) to ensure that the
execution time of the analysis takes less than 60 seconds. This
is a simple and standard POSIX signal handler. If the timeout
is reached, the module will process the next one.

This approach fixes the specific issues we have currently
with some inputs where the sentiment analysis takes too much time. This
fix should be improved and be more generic:

 - Introducing statistics of content which timeouts.
 - Keeping a list/queue to further process those files using a different
   analysis approach. Maybe a set of "dirty" processes to handle the edge cases
   and to not impact the overall processing and analysis.
 - Make the timer configurable per module (at least for this one).
 2017-01-26 07:11:18 +00:00
Mokaddem c422db6e6d Merge remote-tracking branch 'upstream/master' into production 2017-01-17 16:38:48 +01:00
mokaddem b3a3616b0b Merge pull request #102 from ptitdoc/bugfix1 
Bug fixes in Workers and improvement of pystemon-feeder
 2017-01-17 16:35:32 +01:00
Mokaddem ea423fde0f Added requirement, better behaviors and confirmation box 2017-01-17 15:58:16 +01:00
Mokaddem 692f13d18b Improved module + Added support of click and keyboard navigation for killing and starting modules 2017-01-17 14:31:06 +01:00
Mokaddem 3b8149d840 Added action button and support of logs 2017-01-16 17:08:48 +01:00
Mokaddem 0ecfaf00f5 Added color in front of queue related to its state 2017-01-16 15:30:33 +01:00
Mokaddem af3241c008 Overwritted listbox update function to add formating 2017-01-16 14:41:02 +01:00
Mokaddem 30f303cb99 Added new ModuleInformation 2017-01-16 12:18:23 +01:00
Olivier MEDOC 16044d4d36 pystemon-feeder: add the option bind in configuration file instead of address 
This is essentially because multiple feeds can be used in the address
configuration variable.
 2017-01-13 14:58:18 +01:00
Alexandre Dulaunoy 3b101ea8f5 (partially) Fix #91 using a simple Alarm (SIGNAL) when exec-timeout 
Introducing a timer (in this case 5 seconds) to ensure that the
execution time of the tokenizer takes less than 5 seconds. This
is a simple and standard POSIX signal handler.

This approach fixes the specific issues we have currently
with some inputs where the tokenization takes too much time. This
fix should be improved and be more generic:

 - Introducing statistics of content which timeouts.
 - Keeping a list/queue to further process those files using a different
   tokenizer approach. Maybe a set of "dirty" processes to handle the edge cases
   and to not impact the overall processing and analysis.
 - Make the timer configurable per module (at least for this one).
 2017-01-12 07:32:55 +00:00
Olivier MEDOC 9f9c265cb0 SentimentAnalysis: use lexicon file path from the ail configuration file 2017-01-11 11:07:08 +01:00
ptitdoc c0fac82031 Add pystemon path in configuration sample 2017-01-10 18:33:46 +01:00
Olivier MEDOC 83db40104c import pystemon-feeder configuration from the central configuration file 2017-01-10 18:26:36 +01:00
Olivier MEDOC 3dc014dad9 fix invalid variable propagation 2017-01-10 18:26:24 +01:00
Olivier MEDOC 68ca20db8f fix to use the new pycountry API 2017-01-10 18:26:17 +01:00
Alexandre Dulaunoy 1950a2dc0e Merge pull request #99 from mokaddem/more-feed 
Usage of sha1 to filter paste content in Mixer.py
 2017-01-09 17:06:37 +01:00
Mokaddem 500608d0a1 Usage of sha1 to filter paste content in Mixer.py 2017-01-09 17:01:10 +01:00
Alexandre Dulaunoy f5adb65cb3 Merge pull request #98 from mokaddem/more-feed 
Config 2017
 2017-01-09 15:39:53 +01:00
Mokaddem 409c5b8b42 Added 2017 2017-01-09 15:36:50 +01:00
Alexandre Dulaunoy a96ba40d94 Merge pull request #96 from mokaddem/more-feed 
Handling of multiple feeders
 2017-01-09 15:29:28 +01:00
Mokaddem 0e39f516a9 Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed. 2017-01-09 14:12:26 +01:00
Mokaddem 3a298f0c7c Commented Mixer.py for further multiple feeds support. 2016-12-26 16:23:18 +01:00
Mokaddem 1c2169d3bc Improved interface for multiple feeds and refresh_script. 2016-12-26 16:16:44 +01:00
Mokaddem 97292e0899 Updated web interface to handle new mixer module. and fixed one dependency bug. 2016-12-23 15:44:46 +01:00
Mokaddem e70b9cd15c Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them. 2016-12-23 10:31:26 +01:00
Alexandre Dulaunoy ddfe3cf66a Merge pull request #95 from mokaddem/module_handling 
Module handling
 2016-12-22 22:18:31 +01:00
Alexandre Dulaunoy 3ce850e093 Merge pull request #94 from mokaddem/website-refacto 
Website refacto, support and fixes
 2016-12-22 22:17:58 +01:00
Mokaddem 9758579753 Merge branch 'module_handling' into production 2016-12-22 10:24:30 +01:00
Mokaddem 7438f16c63 Added new config 2016-12-22 10:06:35 +01:00
Mokaddem 73938cf41b Prepared moduleInfo to handle module click with curses (will come back later). 2016-12-22 08:45:53 +01:00
Mokaddem d37b243e1d Added legend + added more info if restarting disabled 2016-12-09 15:13:47 +01:00
Mokaddem d3ed1bbf40 Merge branch 'website-improvement' into production 2016-12-08 10:05:55 +01:00
Mokaddem 73d4f9e082 Webstats should correctly updates top_progression_zset (Not fully tested because not enough data. Will be tested latter) 2016-12-08 10:05:07 +01:00
Mokaddem 570324060e terms top_sets correctly supports blacklisted terms 2016-12-08 09:13:31 +01:00
Alexandre Dulaunoy 412b80c44d Merge pull request #88 from mokaddem/Issue-Feeding-AIL 
Pystemon tuto
 2016-11-26 12:45:28 +01:00
Mokaddem d831a79614 Do not try to restart not running queue if autokill is not set 2016-11-25 16:00:46 +01:00
Mokaddem 4bf45eea36 ModuleInformation sort by name as well as elapsed time 2016-11-25 14:44:27 +01:00
Mokaddem 866be4e884 Added row coloring in ModuleInformation 2016-11-25 14:12:09 +01:00
Mokaddem a3255d168c ModuleInfo correctly handle CurveManageTopSets, 
Changed queue coloring in dashboard,
ModuleInformation now have a history of executed command and better handle module killing.
 2016-11-25 11:54:16 +01:00
Mokaddem 224fbc8084 Better handle stuck modules. Differentiate between not running and no info and tries to restart stuck ones. 2016-11-24 16:58:32 +01:00
Mokaddem 5d269ea1ee Added date in Duplicate for better distinguish duplicate. Also, added a timeout for the dataTable in duplicate inside the modal. 2016-11-24 15:05:29 +01:00
Mokaddem ac254e0e63 Added support of browsing concerned paste in dashboard for all modules + Show in the page what was the concerned module 2016-10-27 11:50:24 +02:00
Mokaddem 1826b170ec Added support of local paste view in dashboard for Mails event only 2016-10-27 11:27:26 +02:00
Mokaddem 5a4ea734a2 Fixed path issues in doc + added dependencies related to ssdeep 2016-10-14 14:26:33 +02:00
Mokaddem 4eafca2936 Added pystemon instruction section 2016-10-13 17:13:08 +02:00
Alexandre Dulaunoy eb14005fcf Fix: as JSON decode needs a string, the empty list is a string. 2016-08-29 12:08:12 +00:00
Alexandre Dulaunoy 3b6e3e156e Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-26 15:57:45 +00:00
Mokaddem 0c760d763b Added support of PID in ModuleInformation and index.html 2016-08-26 15:27:37 +02:00
Alexandre Dulaunoy 758f2c5334 Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-26 08:30:11 +00:00
Mokaddem 494980c48e Added comment in moduleInformation 2016-08-26 10:07:06 +02:00
Mokaddem 8b33b4e7ba Added not running module info + swapped order of module launch 2016-08-26 09:21:02 +02:00
Mokaddem 813cc7031a Changed lots of code into zrank 2016-08-25 17:48:12 +02:00
Mokaddem aaac798db9 updated individual launch script 2016-08-25 16:51:57 +02:00
Mokaddem 78c991539e Added number of same module running in ModuleInformation 2016-08-25 16:07:47 +02:00
Alexandre Dulaunoy 1ab08d4a05 Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-25 12:33:31 +00:00
Mokaddem f9bf00f022 Removed autokill feature if not specified 2016-08-25 11:09:31 +02:00
Mokaddem 2f136595fb Removed autokill feature if not specified 2016-08-25 11:05:18 +02:00
Mokaddem 9146feab88 Fixed bug in WebStats not deleting correct key + fixed typo in CurveSetManager preventing connecting to redis + modified display in moduleInfo 2016-08-25 10:43:04 +02:00
Mokaddem 3313064e51 removed useless import. 2016-08-24 17:32:50 +02:00
Mokaddem 64bf481480 Added moduleInfo + Renamed modules 2016-08-24 17:28:39 +02:00
Mokaddem 8fb552b7da ModuleInfo support autokill and auto restart after logging. 2016-08-24 16:52:01 +02:00
Mokaddem 3e88d0911d ModuleInfo now support logs. 2016-08-24 15:53:00 +02:00
Mokaddem ff6dcaf456 Fixed typo in creditcard + fixed bug in ModuleInformation (relative to date). 2016-08-24 15:35:23 +02:00
Mokaddem deeebec2f5 Added new module 'ModuleInformation' which gives real time information on running module 2016-08-24 15:21:41 +02:00
Mokaddem 07856f3119 Added script module-dataflow-graph + fixed bug in module.cfg preventing the credential module to receive data. 2016-08-24 11:32:48 +02:00
Alexandre Dulaunoy bde5f47ed0 Merge branch 'master' of github.com:CIRCL/AIL-framework 
Conflicts:
	bin/launch_scripts.sh
 2016-08-23 14:20:39 +00:00
Mokaddem 42b3966499 Added new module in individual launch scripts 2016-08-23 16:01:11 +02:00
Mokaddem 697812dbff Added config.sample 2016-08-23 09:59:39 +02:00
Mokaddem f9b27e4013 Fixed bug in curve preventing to track concerned_pastes. Now all tracked words are put in lower case. 2016-08-23 09:50:03 +02:00
Mokaddem 047331df3e Added save of paste_baseName in set for searches. This is done in ModuleStats to not surcharge the Global module. 2016-08-23 08:52:54 +02:00
Mokaddem 20fd191c20 Suppressed prints 2016-08-22 21:29:59 +02:00
Mokaddem 2fd2c8af39 Removed commented code 2016-08-22 20:59:56 +02:00
Mokaddem 003c72bd7c Fixed bug with redis where the inserted key was too long, causing a crash + fixed bug taking min instead of max in terms-top-set 2016-08-22 16:01:42 +02:00
Mokaddem 2466f355ab Fixed bug in manage-terms-top-set + added auto sorting in important-paste-by-module 2016-08-22 14:54:37 +02:00
Mokaddem d3c0fc5f94 Added terms trending module and web interface: 
- Curve module has been rewritten
- Added new module curve_manage_top_set
- Added all web-pages
 2016-08-22 13:35:49 +02:00
Mokaddem 0b6828fb16 Correctly handle and display add or remove a term in term-frequency + started sketch of terms-plot tool. 2016-08-19 16:53:46 +02:00
Mokaddem b78effe779 Added draft of terms_management + Added copyright in readme 2016-08-19 13:34:02 +02:00
Mokaddem 52b7124a6b - Modified redis connection (from levelDB to redis). 
- Added term frequency in curve.
- Modified ModuleStats and WebStats to use redis special command (incrby instead of get-set and zset) and Flask to perform the correct queries.
- Added panel color in sentiment trending
 2016-08-18 15:34:19 +02:00
Mokaddem 902b7e39f4 Probably fixed bug in creditcard (not sending to duplicate) 2016-08-17 14:40:13 +02:00
Mokaddem c10f28d9df Updated isolated launch scripts + Added switch displaying queue in index 2016-08-17 13:46:01 +02:00
Mokaddem dbc164f4e8 modified + into append 2016-08-17 11:26:07 +02:00
Mokaddem c12e67a35a Added tooltips + slight gauge modification 2016-08-17 10:43:43 +02:00
Mokaddem 894b9efda9 Added drop of really long line in sentiment-analysis module + Added description of sentiment module. Also, fixed bug in webpage sentiement-trending concerning avg and date range. 2016-08-17 09:46:25 +02:00
Mokaddem 1084e45f1b Added draft of filter in sentiment analysis (Discard syntaxical languages) + Added nice tooltip for sparkline. Trending displays avg in function of the number of elements processed and not for the complete week + fixed bug in gauge and canvasjs (was performing avg with only 1 graph instead of all 8). 2016-08-16 16:33:02 +02:00
Mokaddem cdd08d7df0 Added all providers for plot_tool in moduleStats + fixed date range bug 2016-08-16 11:07:36 +02:00
Mokaddem 1a5158b081 Added sentiment analyser module (draft) 2016-08-13 15:24:57 +02:00
Mokaddem c028ca80da Separated options from the plot funcion for sparkline. 2016-08-11 17:57:23 +02:00
Alexandre Dulaunoy 95a22abe9c Merge branch 'master' of github.com:CIRCL/AIL-framework 
Conflicts:
	bin/packages/modules.cfg
 2016-08-11 08:06:57 +00:00
Mokaddem cd74e6d1c5 Added module sentimentAnalyser 2016-08-11 09:40:42 +02:00
Mokaddem 64fe32b29b Fixed small bug in SQLInjection module 2016-08-10 14:23:36 +02:00
Mokaddem feafa8ffe2 Slight refacto + sql comments are now placed in injection suspision. 2016-08-10 09:06:37 +02:00
Mokaddem 943016d6ca Fixed one of the small display bug in DUplicate_ssdeep_v2 (was displaying [tlsh, tlsh] instead of simply tlsh) 2016-08-09 15:06:55 +02:00
Mokaddem a4595010c0 Added whole config in config.sample 2016-08-09 14:23:36 +02:00
Mokaddem a80f6ca0c2 Bug fix related with redis: 
Fixed typo key in redis for module creditcard and sqlinjection
Modified Curve redisLvlDb server
Modified Url.py so that it forwards name of protocol from saved protocolsfile
Added Cache control in Flask
Modified key-tab name into keys-tab
 2016-08-09 11:59:36 +02:00
Mokaddem 60512a30a2 Fixeed bug in Duplicate (The comparison value was not saved correctly in redis) + Modified to progression detection algo 2016-08-08 15:28:26 +02:00
Mokaddem aedb3198cd Fixed bug in CVE (always sending to browse_warning even if there was no match) 2016-08-08 11:54:27 +02:00
Mokaddem 14062a549c Fixed bug in credential (not sending correctly to ModuleStat) + Fixed date bug in ModuleStats 2016-08-08 11:37:18 +02:00
Mokaddem 4e56f76f62 Fixed bug CVE, 
CVE was not sending its data to BrowseWarningPaste.
 2016-08-08 10:05:21 +02:00
Mokaddem 1c40ad522f Merge branch 'master' into production 2016-08-08 09:26:58 +02:00
Mokaddem 78c611fead Added warning_paste module and created related webpages. 
Fixed a Faup bug in credential (multiple instanciation) and added correc populate_set_out in concerned modules (creditcard, credential, ...).
Linked browse_warning_paste module and Flask function with redis (created new sets).
 2016-08-08 09:17:44 +02:00
Mokaddem 3523d79a05 Started browse important paste feature. Still need to link with redis. + Fixed a bug in Flask (String to list parser) 2016-08-04 16:57:34 +02:00
Mokaddem 55ef917ef1 Added few comments 2016-08-04 12:10:56 +02:00
Mokaddem d9316771cd Draft: added new duplicate hash comparison - tlsh 2016-08-04 11:55:38 +02:00
Mokaddem 455911fc26 Better display printing 2016-08-02 16:21:52 +02:00
Mokaddem 3169dc27b0 Added SQLInjectionDetection module 2016-08-02 15:43:11 +02:00
Mokaddem a1b2329a9b Fixed bug in case of null elements arrising because of a new month 2016-08-01 10:32:24 +02:00
Mokaddem a9637ec7cf removed prints 2016-07-29 16:54:25 +02:00
Mokaddem feb1717dc6 fixed size bug, which in some cases display the total size instead of the average + few cleanup + commented out log scale in moduleStats.js and added large number division in case of div by 0 2016-07-29 16:52:50 +02:00
Mokaddem e8fae17cfd Fixed small race condition bug in ModuleStat.py and Added highlight of bar belonging to the same serie in moduleStats.js 2016-07-29 10:36:44 +02:00
Mokaddem 240bdd9cca Added number of paste by provider in module stats 2016-07-27 09:27:32 +02:00
Mokaddem baa042827d Removed useless comments + prevent trying to add an already present element in the set 2016-07-26 11:06:31 +02:00
Mokaddem cc1b49baaf Fixed variable bug in moduleStats and added draft of credential stats 2016-07-26 10:45:02 +02:00
Mokaddem 06be1f129a Added Draft of ModuleStats + Paste size average 2016-07-25 16:38:57 +02:00
Alexandre Dulaunoy d013546643 Merge branch 'master' of github.com:CIRCL/AIL-framework 
Conflicts:
	bin/packages/modules.cfg
 2016-07-25 07:56:46 +00:00
Alexandre Dulaunoy 722634adf4 Prod config 2016-07-25 07:49:14 +00:00
Raphaël Vinot 4aa484a91f Merge pull request #63 from mokaddem/uptodate-duplicate-module 
Uptodate duplicate module
 2016-07-25 09:44:49 +02:00
Mokaddem 34a00abdc2 Merge branch 'uptodate-duplicate-module' into production 2016-07-22 10:31:10 +02:00
Mokaddem c686f69ca6 Clean not compatible ssdeep hash encountered 2016-07-22 10:04:58 +02:00
Raphaël Vinot ea8703e608 Fix path to tldsfile 2016-07-22 09:32:13 +02:00
Mokaddem dad05073fd Fixed bug introduced in merge-conflict 2016-07-22 09:25:05 +02:00
Mokaddem 7c4f4a2f79 Merge branch 'statistics-v2' into uptodate-statistics-v2 2016-07-21 15:16:03 +02:00
Mokaddem b685f81752 Merge branch 'duplicate-module' into uptodate-duplicate-module 2016-07-21 14:59:52 +02:00
Mokaddem 0ab7560477 Added top_progression chart for tld, domain and scheme + Small modification in config file. 2016-07-21 13:44:22 +02:00
Raphaël Vinot ede46ec799 Fix default config. 2016-07-21 09:19:13 +02:00
Raphaël Vinot 781b441bb8 Fix path in config 2016-07-20 14:12:50 +02:00
Raphaël Vinot 620d69467f Add WebStats in launcher 2016-07-20 14:12:37 +02:00
Raphaël Vinot 62eef44ca8 Fix pep8 2016-07-20 14:12:18 +02:00
Raphaël Vinot 8a69110c8e Merge branch 'webpage-splitting' of https://github.com/mokaddem/AIL-framework into mokaddem-webpage-splitting 2016-07-19 16:35:31 +02:00
Mokaddem a6996c0b23 Added related functions and display of duplicated paste in search.py 2016-07-19 10:48:44 +02:00
Mokaddem 996c0e02de Duplicate module takes its messages from other modules and no more from Global. 2016-07-18 16:22:33 +02:00
Mokaddem 6805ed6488 Added default config 2016-07-18 15:52:53 +02:00
Mokaddem 4f6813350b Added two new version of duplicate module. 
One with hashes are saved in json on disk
The other with only leveldb
 2016-07-18 15:50:41 +02:00
Mokaddem 14e9850dd6 Added new module for Duplicate paste. Seems working but has some small bug (re-check same paste twice) 2016-07-15 16:58:48 +02:00
Mokaddem 2383db022f Added default configuration 2016-07-15 09:10:44 +02:00
Mokaddem 9a9e07f600 Added default configuration 2016-07-15 09:08:38 +02:00
Mokaddem 0332f23579 Added SimHash library 2016-07-15 08:56:16 +02:00
Mokaddem fba14bfb4b In index: Added number of processed pastes chart 2016-07-14 11:16:07 +02:00
Mokaddem ab61e32399 Commented out get_language because it adds too much overhead 2016-07-14 11:15:15 +02:00
Mokaddem 594d2def35 In index: Added number of processed pastes chart 2016-07-13 15:57:33 +02:00
Mokaddem 56b6659d8b Commented out get_language because it adds too much overhead 2016-07-13 08:59:48 +02:00
Mokaddem c51bdec8aa Merge branch 'mokaddem-testing' 2016-07-12 11:53:24 +02:00
Mokaddem 8a1247cf5d modified variable name str 2016-07-12 11:52:19 +02:00
Mokaddem 7e5ce0f17f Optimized create_plot and removed test commemts 2016-07-12 11:47:51 +02:00
Mokaddem 465244e1ce Added dynamic table sorting in search page. (Still need to add dependencies) 2016-07-07 16:38:00 +02:00
Mokaddem 7ff9b9a583 Added DomainTrending seems working. 
Started search features with related html pages, not finish yet.
 2016-07-05 16:53:03 +02:00
Mokaddem 5a9eca9291 Added few comments 2016-07-04 09:18:23 +02:00
Mokaddem 4b3101b7b6 Added template tld. Modified URL using Faup and refactored WebStats. 2016-07-01 16:59:08 +02:00
Mokaddem beeeb76de9 Added new modules and started WebTrending web interface 2016-06-30 14:38:28 +02:00
Mokaddem 3dc356dc5e Getting Started: Initial configuration working on the laptop 2016-06-30 14:36:47 +02:00
MaximeStor ab66cd255a Improve SourceCode, keywords and add description in /doc 2016-03-12 12:30:38 +01:00
MaximeStor 701d771aa5 Add first version of Source Code 2016-03-12 11:21:29 +01:00
Raphaël Vinot be86737ca7 Because 0MQ fails. 2016-03-11 16:16:53 +01:00
Raphaël Vinot f6e4ea2270 Fix logging, fix URL regex 2016-02-11 12:19:03 +01:00
Raphaël Vinot d160e4a2c8 Add Credential in the scripts to launch 2016-02-10 17:33:16 +01:00
Raphaël Vinot 90e1b25426 Split filepath and count in credential module 2016-02-10 17:31:52 +01:00
Raphaël Vinot 4895ee9fa2 Add new category (Credential) 2016-02-10 16:39:56 +01:00
Raphaël Vinot 1da8675750 Refactoring on Credential, Phone and Release 2016-02-10 16:39:06 +01:00
Alexandre Dulaunoy c68136b04b Merge branch 'pgp' of https://github.com/Rafiot/AIL-framework 
Conflicts:
	bin/packages/modules.cfg
 2016-02-08 10:13:44 +01:00
MaximeStor b7d2b64a86 Merge branch 'master' of https://github.com/CIRCL/AIL-framework into module 2016-02-06 11:28:48 +01:00
Alexandre Dulaunoy 192ee7f0ec Merge pull request #49 from Alainfou/master 
Phone module added (regex needs optimization)
 2016-02-06 11:20:46 +01:00
Alain ea52fd1068 Phone regex updated 
Might still need to be fixed / optimized, in case of maths or random numbers starting with a 0. Do not capture dates, hours, coordinates anymore. Captured formats are: e.g. +331234567890 ; 09 12 34 56 78 ; +4177/123.45.69 ; +352(0)6-23-23-23...
 2016-02-05 20:58:02 +01:00
Alain fabbfd8ae9 Update module.cfg (adding Keys and Phone section) 2016-02-05 14:00:41 -05:00
Alain 43b3556588 Starting Phone number recognition 2016-02-05 13:58:21 -05:00
MaximeStor 07513a5b37 Add modules Credential and Release 2016-02-05 16:15:09 +01:00
Raphaël Vinot 9171d5b118 Add module to find PGP encrypted blobs 2016-02-05 16:03:37 +01:00
Raphaël Vinot aef8ab0411 Listen locally for 0MQ 2016-02-04 15:32:50 +01:00
Raphaël Vinot 5ca13c42eb Launch redis and leveldb from local directory 2016-02-04 15:24:39 +01:00
Raphaël Vinot 12aca6b760 Add script to import from local directory, use local python from env 2016-02-04 15:22:51 +01:00
Raphaël Vinot 315cb48117 Add template file for writing a new module 2016-02-03 10:33:42 +01:00
Raphaël Vinot 0d6adc2063 Add initial Travis file 2016-01-19 11:43:34 +01:00
Alexandre Dulaunoy cdd0725e88 -v option added to list the path 2015-12-22 21:37:05 +00:00
Alexandre Dulaunoy e3971ac93a Onion fetching loop deactivated by default 2014-12-22 16:06:38 +00:00
Raphaël Vinot 08ceefc375 Re-add config option 2014-12-22 16:50:25 +01:00
Raphaël Vinot 50369c6706 Revert changes on the config file due to merging messup 2014-12-22 16:29:05 +01:00
Raphaël Vinot f717f9fe89 Merge branch 'master' of github.com:CIRCL/AIL-framework 2014-12-22 15:32:48 +01:00
Raphaël Vinot 9ee61db2cf Add hotfixes 2014-12-22 15:27:02 +01:00
Raphaël Vinot 8803c8447a Publish the fetched onions on a ZMQ feed. 2014-09-30 16:55:16 +02:00
Alexandre Dulaunoy 25757b0fff A simple feeder script feeding data from pystemon to AIL. 
The configuration matches the default Redis parameters used
in the pystemon configuration.

https://github.com/cvandeplas/pystemon/blob/master/pystemon.yaml#L16
 2014-09-19 14:03:05 +02:00
Raphaël Vinot 65b9a01644 Add config file for DomainClassifier, proper reporting 2014-09-17 17:22:56 +02:00
Alexandre Dulaunoy 27b134ec03 Add proper publisher for classified domains/hostnames 2014-09-10 09:27:47 +02:00
Raphaël Vinot f017680365 fix onions, cc and domain classifier modules 2014-09-08 16:51:43 +02:00
Alexandre Dulaunoy de6e21d5a7 DomainClassifier sample configuration added 2014-09-08 16:44:05 +02:00
Alexandre Dulaunoy 246621f663 First version of the DomainClassifier 2014-09-08 16:43:21 +02:00
Alexandre Dulaunoy 1397db9691 Global queue for DomainClassifier 2014-09-08 11:07:45 +02:00
Raphaël Vinot e983c839ad Categ now listen to the Global queue 2014-09-05 17:05:45 +02:00
Raphaël Vinot 46f27ada4e More cleanup 2014-09-05 10:42:01 +02:00
Raphaël Vinot fca00beed9 Add Domain Classifier module. 
Cleanup in the config files.
 2014-09-05 10:41:00 +02:00
Raphaël Vinot b7c9e489c9 Fix the exceptions 2014-09-04 11:46:07 +02:00
Raphaël Vinot 9e8611a42d stop killing the disk when creating the word curve 2014-09-02 18:20:28 +02:00
Raphaël Vinot 7542eaf739 Update starting script. 2014-09-02 15:21:36 +02:00
Raphaël Vinot 0c6b09f379 Fix the onion module, log the valid onions. 2014-09-01 16:18:06 +02:00
Raphaël Vinot f4b89669fc The onion module now fetches the URLs it finds. 2014-08-31 22:42:12 +02:00
Raphaël Vinot abfe13436b Big refactoring, make the queues more flexible 2014-08-29 19:37:56 +02:00
Raphaël Vinot 623e876f3b Cleanup. 
* Remove useless subscriber
* Fix typo in the config file
* Update Helper accordingly
 2014-08-26 17:36:57 +02:00
Alexandre Dulaunoy 3b499a2ec8 ZMQ Publisher removed 
ZMQ Publisher removed to allow concurrent use of the scripts.
In short term, we would replace all publishing part within AIL
into pub-sub Redis to avoid ZMQ limitation.
 2014-08-26 14:38:49 +02:00
Alexandre Dulaunoy f070ac2005 cymruwhois uses dotted decimal format 2014-08-25 10:05:36 +02:00
Raphaël Vinot 3886d1b834 Small fixes to make the refactoring production ready 
* the port for the logging is 6380
* use os.environ properly
* fix typos
 2014-08-22 17:35:40 +02:00
Raphaël Vinot 78125db4ea Use env variables everywhere 2014-08-22 14:52:02 +02:00
Raphaël Vinot 277d138a5d cleanup, add FIXME 2014-08-21 14:39:17 +02:00
Raphaël Vinot 63b29176c1 move Redis_Data_Merging to Paste 2014-08-21 12:22:07 +02:00
Raphaël Vinot 50cfac857e Update config 
Make all paths in the config file relative to the home directory.
 2014-08-20 16:00:56 +02:00
Raphaël Vinot a68f5b6a0e fix subscriber names, update default config 2014-08-20 15:54:21 +02:00
Raphaël Vinot 2485ba5df2 Merge remote-tracking branch 'origin/master' into testing 
Conflicts:
	bin/ZMQ_Sub_Urls.py
 2014-08-20 15:24:10 +02:00
Raphaël Vinot 99c8cc7941 completely remove ZMQ_PubSub.py 2014-08-20 15:14:57 +02:00
Alexandre Dulaunoy 1d64dc44c8 MIME type guessing - removed one duplicate call to libmagic 2014-08-20 10:22:33 +02:00
Raphaël Vinot 8d9ffbaa53 Do not create a ZMQ sub if it is not required. 2014-08-19 19:53:33 +02:00
Raphaël Vinot 45b0bf3983 Improve the cleanup. Still some to do. 2014-08-19 19:07:07 +02:00
Raphaël Vinot f1753d67c6 Cleanup the queues. 2014-08-19 16:05:37 +02:00
Alexandre Dulaunoy e8fcea6cd6 Remove undeclared variable 2014-08-18 16:17:36 +02:00
Alexandre Dulaunoy 7d8ee102a3 Assignment before use (if Enumerate fails) 2014-08-18 15:58:06 +02:00
Alexandre Dulaunoy 4304c6858e Configuration path fixed 2014-08-18 09:02:08 +02:00
Raphaël Vinot 078c8ea836 Big cleanup, pep8 2014-08-14 18:07:18 +02:00
Jules ab6765315e Merge pull request #13 from adulau/master 
Log where URLs are hosted - cc_critical option added
 2014-08-14 14:28:01 +02:00
Alexandre Dulaunoy 762def3a23 Log where URLs are hosted - cc_critical option added 
It logs where the hostname of the URL is hosted (ASN and geographic location).
A simple option cc_critical added to set the country code to log as critical.
 2014-08-14 14:22:11 +02:00
Raphaël Vinot 4a1f300a1a Cleanup (remove unused imports, more pep8 compatible) 2014-08-14 14:11:07 +02:00
Starow 04a8f1bdf2 maxi cleanup old code :'( 2014-08-14 11:48:46 +02:00
Starow 29b24b6466 printing set of domain for debugging 2014-08-13 16:35:27 +02:00
Raphaël Vinot ece3bc173e Cleanup of main Paste module 2014-08-13 11:56:22 +02:00
Raphaël Vinot 5b17d416c8 remove script installed by pubsublogger 2014-08-13 11:55:59 +02:00
Raphaël Vinot 935e51c961 Remove 3rd party code (pubsublogger), add it in the deps. 2014-08-13 10:19:43 +02:00
Starow 37033ca3a6 Minor logs modifications 2014-08-13 10:08:44 +02:00
Starow 6aa4d7cb7d Harmonising logs messages + Changing some dygraph options 2014-08-12 15:42:16 +02:00
Alexandre Dulaunoy 0b4a80b7ea -s option added to find similar documents 
By default, the index is not storing the vector of the document (Whoosh
document schema). It won't work if you don't change the schema of the
index for the content. It depends of your storage strategy.
 2014-08-12 13:42:26 +02:00
Alexandre Dulaunoy fd6e1a8436 -f option added: dump full document for each match 2014-08-12 13:26:56 +02:00
Alexandre Dulaunoy 0a6664ffba Indexer: Some index statistics added 
usage: indexer_lookup.py [-h] [-q Q] [-n] [-t] [-l]

Fulltext search for AIL

optional arguments:
  -h, --help  show this help message and exit
  -q Q        query to lookup (one or more)
  -n          return number of indexed documents
  -t          dump top 500 terms
  -l          dump all terms encountered in indexed documents
 2014-08-11 15:07:12 +02:00
Alexandre Dulaunoy f65a94d47b -l added -> dumping all terms indexed 2014-08-11 14:56:15 +02:00
Alexandre Dulaunoy f3d1ca052e Return the number of indexed documents 2014-08-11 14:50:35 +02:00
Alexandre Dulaunoy 611d2a466f Configuration that should not be there... 2014-08-11 14:24:27 +02:00
Alexandre Dulaunoy 2b8f2689bf Indexer queue and script added to "BBS-like" LAUNCH script 2014-08-11 14:06:52 +02:00
Alexandre Dulaunoy 9657c6bf80 Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2014-08-11 13:46:37 +02:00
Alexandre Dulaunoy b1053af3cd Indexer module: script to query the index 
Test script to query the index generated from the Indexer module.

python indexer_lookup.py -q Visa -q Mastercard
 2014-08-11 12:03:27 +02:00
Starow 079db6f80c Hardcoded path from ZMQ_Curve are now referring correctly in config.cfg.sample fix #6 2014-08-11 11:33:18 +02:00
Alexandre Dulaunoy 7bdd4a41a5 Indexer module added - initial version with Whoosh full-text indexer 
The indexer module indexes all the pastes using Whoosh. The module
can be extended to support additional full-text indexers in the future.
 2014-08-11 11:04:09 +02:00
Starow d1d4b2ebe0 Importing dns.exeption fix #4 fix #7 2014-08-11 09:27:50 +02:00
Starow 192074e569 Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2014-08-11 09:21:09 +02:00
Starow a5c1d59d29 Catching the exception dns.exception.Timeout fix #7 2014-08-11 09:18:55 +02:00
Starow 54091a2174 Catching the exception dns.exception.Timeout fix #4 2014-08-11 09:08:28 +02:00
Starow eb603e8762 Fixing a bug about caching paste inside Redis :) 2014-08-08 17:23:51 +02:00
Starow 7a1db94f9e Adding a letter (s) 2014-08-08 17:19:42 +02:00
Starow 043800287a adding a . 2014-08-08 17:18:03 +02:00
Starow bf682c4b44 Fixing last commit ... 2014-08-08 17:13:18 +02:00
Starow 503c23ca3b Fixing last commit 2014-08-08 17:08:41 +02:00
Starow c9e1eaf182 Improving cache code 2014-08-08 17:04:25 +02:00
Starow 44addf1afe Redis cache added fix #5 
The paste will be add in Redis during 5min and also saved on disk.
Now if a module want to get the paste for further processing, it will first try to get it in the cache
instead of getting it directly on the disk and wasting I/O.
 2014-08-08 16:48:02 +02:00
Starow 97f3a3df9e update pubsublogger with the last version 2014-08-07 14:49:34 +02:00
Starow c10003a630 Changing ZMQ Curve Module comment 2014-08-07 14:46:43 +02:00
Starow 1379ef705a Initial import of AIL framework - Analysis Information Leak framework 
AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past
ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen
sitive information
 2014-08-06 11:43:40 +02:00