CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
624 Commits
7 Branches
43 Tags
110 MiB
Commit Graph

299 Commits (318a557c560705ad5f9cf77b66fa885a6417ef14)

Author SHA1 Message Date
Alexandre Dulaunoy 2f4e5b9016 Merge branch 'master' of github.com:CIRCL/AIL-framework 2017-01-26 07:16:44 +00:00
Alexandre Dulaunoy cf903cc212 (partially fix #90 too) using a simple Alarm (SIGNAL) when exec-timeout 
Introducing a timer (in this case 60 seconds) to ensure that the
execution time of the analysis takes less than 60 seconds. This
is a simple and standard POSIX signal handler. If the timeout
is reached, the module will process the next one.

This approach fixes the specific issues we have currently
with some inputs where the sentiment analysis takes too much time. This
fix should be improved and be more generic:

 - Introducing statistics of content which timeouts.
 - Keeping a list/queue to further process those files using a different
   analysis approach. Maybe a set of "dirty" processes to handle the edge cases
   and to not impact the overall processing and analysis.
 - Make the timer configurable per module (at least for this one).
 2017-01-26 07:11:18 +00:00
Mokaddem c422db6e6d Merge remote-tracking branch 'upstream/master' into production 2017-01-17 16:38:48 +01:00
mokaddem b3a3616b0b Merge pull request #102 from ptitdoc/bugfix1 
Bug fixes in Workers and improvement of pystemon-feeder
 2017-01-17 16:35:32 +01:00
Mokaddem ea423fde0f Added requirement, better behaviors and confirmation box 2017-01-17 15:58:16 +01:00
Mokaddem 692f13d18b Improved module + Added support of click and keyboard navigation for killing and starting modules 2017-01-17 14:31:06 +01:00
Mokaddem 3b8149d840 Added action button and support of logs 2017-01-16 17:08:48 +01:00
Mokaddem 0ecfaf00f5 Added color in front of queue related to its state 2017-01-16 15:30:33 +01:00
Mokaddem af3241c008 Overwritted listbox update function to add formating 2017-01-16 14:41:02 +01:00
Mokaddem 30f303cb99 Added new ModuleInformation 2017-01-16 12:18:23 +01:00
Olivier MEDOC 16044d4d36 pystemon-feeder: add the option bind in configuration file instead of address 
This is essentially because multiple feeds can be used in the address
configuration variable.
 2017-01-13 14:58:18 +01:00
Alexandre Dulaunoy 3b101ea8f5 (partially) Fix #91 using a simple Alarm (SIGNAL) when exec-timeout 
Introducing a timer (in this case 5 seconds) to ensure that the
execution time of the tokenizer takes less than 5 seconds. This
is a simple and standard POSIX signal handler.

This approach fixes the specific issues we have currently
with some inputs where the tokenization takes too much time. This
fix should be improved and be more generic:

 - Introducing statistics of content which timeouts.
 - Keeping a list/queue to further process those files using a different
   tokenizer approach. Maybe a set of "dirty" processes to handle the edge cases
   and to not impact the overall processing and analysis.
 - Make the timer configurable per module (at least for this one).
 2017-01-12 07:32:55 +00:00
Olivier MEDOC 9f9c265cb0 SentimentAnalysis: use lexicon file path from the ail configuration file 2017-01-11 11:07:08 +01:00
ptitdoc c0fac82031 Add pystemon path in configuration sample 2017-01-10 18:33:46 +01:00
Olivier MEDOC 83db40104c import pystemon-feeder configuration from the central configuration file 2017-01-10 18:26:36 +01:00
Olivier MEDOC 3dc014dad9 fix invalid variable propagation 2017-01-10 18:26:24 +01:00
Olivier MEDOC 68ca20db8f fix to use the new pycountry API 2017-01-10 18:26:17 +01:00
Alexandre Dulaunoy 1950a2dc0e Merge pull request #99 from mokaddem/more-feed 
Usage of sha1 to filter paste content in Mixer.py
 2017-01-09 17:06:37 +01:00
Mokaddem 500608d0a1 Usage of sha1 to filter paste content in Mixer.py 2017-01-09 17:01:10 +01:00
Alexandre Dulaunoy f5adb65cb3 Merge pull request #98 from mokaddem/more-feed 
Config 2017
 2017-01-09 15:39:53 +01:00
Mokaddem 409c5b8b42 Added 2017 2017-01-09 15:36:50 +01:00
Alexandre Dulaunoy a96ba40d94 Merge pull request #96 from mokaddem/more-feed 
Handling of multiple feeders
 2017-01-09 15:29:28 +01:00
Mokaddem 0e39f516a9 Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed. 2017-01-09 14:12:26 +01:00
Mokaddem 3a298f0c7c Commented Mixer.py for further multiple feeds support. 2016-12-26 16:23:18 +01:00
Mokaddem 1c2169d3bc Improved interface for multiple feeds and refresh_script. 2016-12-26 16:16:44 +01:00
Mokaddem 97292e0899 Updated web interface to handle new mixer module. and fixed one dependency bug. 2016-12-23 15:44:46 +01:00
Mokaddem e70b9cd15c Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them. 2016-12-23 10:31:26 +01:00
Alexandre Dulaunoy ddfe3cf66a Merge pull request #95 from mokaddem/module_handling 
Module handling
 2016-12-22 22:18:31 +01:00
Alexandre Dulaunoy 3ce850e093 Merge pull request #94 from mokaddem/website-refacto 
Website refacto, support and fixes
 2016-12-22 22:17:58 +01:00
Mokaddem 9758579753 Merge branch 'module_handling' into production 2016-12-22 10:24:30 +01:00
Mokaddem 7438f16c63 Added new config 2016-12-22 10:06:35 +01:00
Mokaddem 73938cf41b Prepared moduleInfo to handle module click with curses (will come back later). 2016-12-22 08:45:53 +01:00
Mokaddem d37b243e1d Added legend + added more info if restarting disabled 2016-12-09 15:13:47 +01:00
Mokaddem d3ed1bbf40 Merge branch 'website-improvement' into production 2016-12-08 10:05:55 +01:00
Mokaddem 73d4f9e082 Webstats should correctly updates top_progression_zset (Not fully tested because not enough data. Will be tested latter) 2016-12-08 10:05:07 +01:00
Mokaddem 570324060e terms top_sets correctly supports blacklisted terms 2016-12-08 09:13:31 +01:00
Alexandre Dulaunoy 412b80c44d Merge pull request #88 from mokaddem/Issue-Feeding-AIL 
Pystemon tuto
 2016-11-26 12:45:28 +01:00
Mokaddem d831a79614 Do not try to restart not running queue if autokill is not set 2016-11-25 16:00:46 +01:00
Mokaddem 4bf45eea36 ModuleInformation sort by name as well as elapsed time 2016-11-25 14:44:27 +01:00
Mokaddem 866be4e884 Added row coloring in ModuleInformation 2016-11-25 14:12:09 +01:00
Mokaddem a3255d168c ModuleInfo correctly handle CurveManageTopSets, 
Changed queue coloring in dashboard,
ModuleInformation now have a history of executed command and better handle module killing.
 2016-11-25 11:54:16 +01:00
Mokaddem 224fbc8084 Better handle stuck modules. Differentiate between not running and no info and tries to restart stuck ones. 2016-11-24 16:58:32 +01:00
Mokaddem 5d269ea1ee Added date in Duplicate for better distinguish duplicate. Also, added a timeout for the dataTable in duplicate inside the modal. 2016-11-24 15:05:29 +01:00
Mokaddem ac254e0e63 Added support of browsing concerned paste in dashboard for all modules + Show in the page what was the concerned module 2016-10-27 11:50:24 +02:00
Mokaddem 1826b170ec Added support of local paste view in dashboard for Mails event only 2016-10-27 11:27:26 +02:00
Mokaddem 5a4ea734a2 Fixed path issues in doc + added dependencies related to ssdeep 2016-10-14 14:26:33 +02:00
Mokaddem 4eafca2936 Added pystemon instruction section 2016-10-13 17:13:08 +02:00
Alexandre Dulaunoy eb14005fcf Fix: as JSON decode needs a string, the empty list is a string. 2016-08-29 12:08:12 +00:00
Alexandre Dulaunoy 3b6e3e156e Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-26 15:57:45 +00:00
Mokaddem 0c760d763b Added support of PID in ModuleInformation and index.html 2016-08-26 15:27:37 +02:00
Alexandre Dulaunoy 758f2c5334 Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-26 08:30:11 +00:00
Mokaddem 494980c48e Added comment in moduleInformation 2016-08-26 10:07:06 +02:00
Mokaddem 8b33b4e7ba Added not running module info + swapped order of module launch 2016-08-26 09:21:02 +02:00
Mokaddem 813cc7031a Changed lots of code into zrank 2016-08-25 17:48:12 +02:00
Mokaddem aaac798db9 updated individual launch script 2016-08-25 16:51:57 +02:00
Mokaddem 78c991539e Added number of same module running in ModuleInformation 2016-08-25 16:07:47 +02:00
Alexandre Dulaunoy 1ab08d4a05 Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-25 12:33:31 +00:00
Mokaddem f9bf00f022 Removed autokill feature if not specified 2016-08-25 11:09:31 +02:00
Mokaddem 2f136595fb Removed autokill feature if not specified 2016-08-25 11:05:18 +02:00
Mokaddem 9146feab88 Fixed bug in WebStats not deleting correct key + fixed typo in CurveSetManager preventing connecting to redis + modified display in moduleInfo 2016-08-25 10:43:04 +02:00
Mokaddem 3313064e51 removed useless import. 2016-08-24 17:32:50 +02:00
Mokaddem 64bf481480 Added moduleInfo + Renamed modules 2016-08-24 17:28:39 +02:00
Mokaddem 8fb552b7da ModuleInfo support autokill and auto restart after logging. 2016-08-24 16:52:01 +02:00
Mokaddem 3e88d0911d ModuleInfo now support logs. 2016-08-24 15:53:00 +02:00
Mokaddem ff6dcaf456 Fixed typo in creditcard + fixed bug in ModuleInformation (relative to date). 2016-08-24 15:35:23 +02:00
Mokaddem deeebec2f5 Added new module 'ModuleInformation' which gives real time information on running module 2016-08-24 15:21:41 +02:00
Mokaddem 07856f3119 Added script module-dataflow-graph + fixed bug in module.cfg preventing the credential module to receive data. 2016-08-24 11:32:48 +02:00
Alexandre Dulaunoy bde5f47ed0 Merge branch 'master' of github.com:CIRCL/AIL-framework 
Conflicts:
	bin/launch_scripts.sh
 2016-08-23 14:20:39 +00:00
Mokaddem 42b3966499 Added new module in individual launch scripts 2016-08-23 16:01:11 +02:00
Mokaddem 697812dbff Added config.sample 2016-08-23 09:59:39 +02:00
Mokaddem f9b27e4013 Fixed bug in curve preventing to track concerned_pastes. Now all tracked words are put in lower case. 2016-08-23 09:50:03 +02:00
Mokaddem 047331df3e Added save of paste_baseName in set for searches. This is done in ModuleStats to not surcharge the Global module. 2016-08-23 08:52:54 +02:00
Mokaddem 20fd191c20 Suppressed prints 2016-08-22 21:29:59 +02:00
Mokaddem 2fd2c8af39 Removed commented code 2016-08-22 20:59:56 +02:00
Mokaddem 003c72bd7c Fixed bug with redis where the inserted key was too long, causing a crash + fixed bug taking min instead of max in terms-top-set 2016-08-22 16:01:42 +02:00
Mokaddem 2466f355ab Fixed bug in manage-terms-top-set + added auto sorting in important-paste-by-module 2016-08-22 14:54:37 +02:00
Mokaddem d3c0fc5f94 Added terms trending module and web interface: 
- Curve module has been rewritten
- Added new module curve_manage_top_set
- Added all web-pages
 2016-08-22 13:35:49 +02:00
Mokaddem 0b6828fb16 Correctly handle and display add or remove a term in term-frequency + started sketch of terms-plot tool. 2016-08-19 16:53:46 +02:00
Mokaddem b78effe779 Added draft of terms_management + Added copyright in readme 2016-08-19 13:34:02 +02:00
Mokaddem 52b7124a6b - Modified redis connection (from levelDB to redis). 
- Added term frequency in curve.
- Modified ModuleStats and WebStats to use redis special command (incrby instead of get-set and zset) and Flask to perform the correct queries.
- Added panel color in sentiment trending
 2016-08-18 15:34:19 +02:00
Mokaddem 902b7e39f4 Probably fixed bug in creditcard (not sending to duplicate) 2016-08-17 14:40:13 +02:00
Mokaddem c10f28d9df Updated isolated launch scripts + Added switch displaying queue in index 2016-08-17 13:46:01 +02:00
Mokaddem dbc164f4e8 modified + into append 2016-08-17 11:26:07 +02:00
Mokaddem c12e67a35a Added tooltips + slight gauge modification 2016-08-17 10:43:43 +02:00
Mokaddem 894b9efda9 Added drop of really long line in sentiment-analysis module + Added description of sentiment module. Also, fixed bug in webpage sentiement-trending concerning avg and date range. 2016-08-17 09:46:25 +02:00
Mokaddem 1084e45f1b Added draft of filter in sentiment analysis (Discard syntaxical languages) + Added nice tooltip for sparkline. Trending displays avg in function of the number of elements processed and not for the complete week + fixed bug in gauge and canvasjs (was performing avg with only 1 graph instead of all 8). 2016-08-16 16:33:02 +02:00
Mokaddem cdd08d7df0 Added all providers for plot_tool in moduleStats + fixed date range bug 2016-08-16 11:07:36 +02:00
Mokaddem 1a5158b081 Added sentiment analyser module (draft) 2016-08-13 15:24:57 +02:00
Mokaddem c028ca80da Separated options from the plot funcion for sparkline. 2016-08-11 17:57:23 +02:00
Alexandre Dulaunoy 95a22abe9c Merge branch 'master' of github.com:CIRCL/AIL-framework 
Conflicts:
	bin/packages/modules.cfg
 2016-08-11 08:06:57 +00:00
Mokaddem cd74e6d1c5 Added module sentimentAnalyser 2016-08-11 09:40:42 +02:00
Mokaddem 64fe32b29b Fixed small bug in SQLInjection module 2016-08-10 14:23:36 +02:00
Mokaddem feafa8ffe2 Slight refacto + sql comments are now placed in injection suspision. 2016-08-10 09:06:37 +02:00
Mokaddem 943016d6ca Fixed one of the small display bug in DUplicate_ssdeep_v2 (was displaying [tlsh, tlsh] instead of simply tlsh) 2016-08-09 15:06:55 +02:00
Mokaddem a4595010c0 Added whole config in config.sample 2016-08-09 14:23:36 +02:00
Mokaddem a80f6ca0c2 Bug fix related with redis: 
Fixed typo key in redis for module creditcard and sqlinjection
Modified Curve redisLvlDb server
Modified Url.py so that it forwards name of protocol from saved protocolsfile
Added Cache control in Flask
Modified key-tab name into keys-tab
 2016-08-09 11:59:36 +02:00
Mokaddem 60512a30a2 Fixeed bug in Duplicate (The comparison value was not saved correctly in redis) + Modified to progression detection algo 2016-08-08 15:28:26 +02:00
Mokaddem aedb3198cd Fixed bug in CVE (always sending to browse_warning even if there was no match) 2016-08-08 11:54:27 +02:00
Mokaddem 14062a549c Fixed bug in credential (not sending correctly to ModuleStat) + Fixed date bug in ModuleStats 2016-08-08 11:37:18 +02:00
Mokaddem 4e56f76f62 Fixed bug CVE, 
CVE was not sending its data to BrowseWarningPaste.
 2016-08-08 10:05:21 +02:00