CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
839 Commits
6 Branches
40 Tags
108 MiB
Commit Graph

391 Commits (80720f8ca5f9c4b9edb82933d603e6cd8a5e6469)

Author SHA1 Message Date
Sami Mokaddem 83e082e62a update: removed useless comments 2017-11-24 08:57:41 +01:00
Sami Mokaddem a0d07b1098 feature: try to send data to MISP if keys are provided, else do nothing 2017-11-23 13:47:02 +01:00
Sami Mokaddem 2a967c4d92 update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment 2017-11-23 07:13:44 +01:00
Sami Mokaddem 805171a8a5 Added a seemingly working version of ailleak to misp 2017-11-20 14:57:25 +01:00
Sami Mokaddem e0b188c069 Better support of python3 in Helper.py 2017-11-20 14:55:36 +01:00
Sami Mokaddem 1e76e10c01 Support of add_object to a MISP instance 2017-11-20 12:12:53 +01:00
Sami Mokaddem 6f0227a78a Starting support of python3 2017-11-20 12:12:06 +01:00
Sami Mokaddem ee7759be91 Updated alertHandler and ailleakObject 2017-11-16 11:18:13 +01:00
Sami Mokaddem 5c7c07fbea Merge branch 'launcher_update' into misp_obj 2017-11-16 10:09:05 +01:00
Sami Mokaddem 45a9897020 Added draft support of MISP ail-leak object 2017-11-16 09:52:37 +01:00
Sami Mokaddem 652b00f4b2 Renamed BrowseWarningPaste into alertHandler 2017-11-15 16:15:43 +01:00
Sami Mokaddem ec7d2d49b2 Added seconds args in import_dir 2017-11-15 16:03:42 +01:00
Sami Mokaddem 13f1c676d6 Added '_AIL' to the end of screen session name to better check an kills sessions. Also, gracefully close redis server + Check 1 time if redis-servers are running. 2017-09-26 16:17:06 +02:00
= a8dc9231c6 Typo2.. 2017-08-24 16:43:42 +02:00
= b2d4dd4a1f Typo 2017-08-24 16:43:05 +02:00
= 74ff401e1d Fixed bug: No CR between indexes in all_index.txt file 2017-08-24 16:41:30 +02:00
Alexandre Dulaunoy bb128a5e94 Merge pull request #138 from mokaddem/import_dir_update 
Improvement of import_dir
 2017-08-24 09:40:52 +02:00
= 429cc96d2a Improvement of import_dir.py. Added feed name parameter, send the correct path (without potential ../) and gzipEncode it if needed. 2017-08-23 15:10:26 +02:00
= e7c66d1aac Updated individual launch scripts 2017-08-23 12:28:12 +02:00
= d632335760 Automatically start redis_levelDB with the correct date 2017-08-23 10:16:56 +02:00
= 64ec44085d Dynamically choose correct port for LevelDB in script 2017-08-22 17:52:15 +02:00
= 66d60a1f01 Added LevelDB Curve for 2017 (3017, used for backward compatibility) 2017-08-21 13:54:44 +02:00
= 92adea38d0 Semi-automated directory creation with date as name in LAUNCH and installing_deps 2017-08-21 13:48:02 +02:00
= bc35747f00 Updated KillAll in LAUNCH 2017-08-21 13:19:26 +02:00
Mokaddem c3632c7d86 Added operation mode 3 for Mixer (do not filter) and added possibility to recurse through dir for import_dir 2017-08-03 00:34:53 +02:00
Mokaddem cbb2909156 Added comments 2017-07-20 10:24:48 +02:00
Mokaddem 05417f405c Finished basic link web interface + solved some bugs 2017-07-20 10:04:30 +02:00
Mokaddem ef741f85c0 Added similarity checks + started link webInterface 2017-07-19 11:52:06 +02:00
Mokaddem 882ab88b00 Draft module + web link 2017-07-18 16:57:15 +02:00
Mokaddem 9558aafe67 Prevent the loss of the 1 message in import_dir.py 2017-07-17 16:14:27 +02:00
Mokaddem 4d59871f00 Merged with previous changes for PR 2017-07-17 13:48:16 +02:00
Alexandre Dulaunoy 4986352099 Merge pull request #120 from PaulSec/master 
Added Python Phonenumbers module to parse numbers and retrieves stats…
 2017-06-08 07:05:52 +02:00
Alexandre Dulaunoy 446a97c47a Merge pull request #115 from mokaddem/regexInTerms 
New features for terms frequency
 2017-06-08 07:04:17 +02:00
Mokaddem 3a4dcd691d Improved description of modules inside the scripts 2017-05-09 11:13:16 +02:00
Paul A 747fff1df1 Added Python Phonenumbers module to parse numbers and retrieves stats from countrycode 2017-05-03 14:25:18 +02:00
Mokaddem 3ea0924120 Improved phone regex to handle end with 4 digits and presence of parenthesis. 2017-04-26 12:41:23 +02:00
Mokaddem b60e164380 Added support of onion in browseWarningPaste 2017-04-26 11:47:02 +02:00
Mokaddem 318a557c56 Added possibility to keep a backup of the old config + check if config is valid before starting scripts. 2017-04-25 09:32:05 +02:00
Mokaddem 72c1d903eb Merge branch 'newIndexer' into auto-conf 2017-04-25 08:14:31 +02:00
Mokaddem eccc8fc88b modified check_size timeout + utilisation of shutil instead of os.system() 2017-04-19 16:37:04 +02:00
Mokaddem da081bae42 Merge branch 'regexInTerms' into production 2017-04-18 16:59:40 +02:00
Mokaddem 2d907ae784 Merge branch 'newIndexer' into auto-conf 2017-04-18 16:20:00 +02:00
Mokaddem b0f1cf84b7 remplaced usage of global variable by a function param 2017-04-18 16:02:22 +02:00
Mokaddem 6e8509a0ba Removed wierd line 2017-04-18 15:57:07 +02:00
Mokaddem c8baabd882 Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...) 2017-04-18 15:28:21 +02:00
Mokaddem 2da4c572c7 New feature in Terms: regex and set of words support (draft) 2017-03-28 17:42:44 +02:00
Mokaddem 6e80eb0954 Typo 2017-03-28 09:55:59 +02:00
Mokaddem c79d4d65c8 Keep original section order while updating 2017-03-28 09:54:24 +02:00
Mokaddem e225090e07 Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample 2017-03-15 16:36:51 +01:00
Mokaddem be93af6f5a Merge branch 'terms-improvements' 2017-03-15 14:55:44 +01:00
Mokaddem 3261eca836 Fixed all_index newline bug + added print of what's happening 2017-03-15 14:29:49 +01:00
Mokaddem 5b7d047622 Changed os.system by os.mkdir + fixed bug during rotation 2017-03-15 14:05:13 +01:00
Mokaddem adc37ab50e Indexer max size now in config 2017-03-15 12:14:41 +01:00
Mokaddem 4005b72180 Added backward support + renamed index with timestamp 2017-03-15 11:51:35 +01:00
Mokaddem 248469d61e Indexer now create an index_dir when it became too large (search in index not updated yet) 2017-03-14 10:37:31 +01:00
Alexandre Dulaunoy 1540b719ed Merge pull request #108 from mokaddem/newModuleInformation 
New version of ModuleInformation
 2017-03-14 08:29:09 +01:00
Mokaddem 3c9cdac39a Merge branch 'terms-improvements' into production 2017-02-28 16:51:39 +01:00
Mokaddem dadc872f9f Merge branch 'newModuleInformation' into production 2017-02-28 16:51:08 +01:00
Mokaddem a2849c1bcd Added a catch for psutils if process get stopped by user 2017-02-28 16:13:01 +01:00
Mokaddem 4071bf2e91 fixed a bug in manage_term (usage of loop.index0) + fixed bug in top_set where graph was not displaying the correct per_paste value 2017-02-28 15:01:48 +01:00
Mokaddem a1c47d7a13 Merge branch 'production' into terms-improvements 2017-02-28 11:15:56 +01:00
Mokaddem fd4feb374f Merge branch 'newModuleInformation' into production 2017-02-28 11:12:06 +01:00
Mokaddem 75b4037519 Fixed bug - If we move the cursor from a line that has dissapear 2017-02-28 11:11:17 +01:00
Mokaddem d6e02591af Merge branch 'newModuleInformation' into production 2017-02-28 10:23:45 +01:00
Mokaddem 975c1a37d9 ModuleInformation create the all_modules files if missing 2017-02-28 10:23:22 +01:00
Mokaddem ccdf256383 Added possibility to show or not in dashboard text that is logged 2017-02-28 10:21:27 +01:00
Mokaddem 39104dea5e Merge branch 'bugfix_categ' into production 2017-02-28 09:56:25 +01:00
Mokaddem 26eb5fdff0 Fixed bug closing parenthesis in categ + changed behavior of sending to browseWarningPaste in Mail 2017-02-28 09:23:32 +01:00
Mokaddem 4a0dac1fae Merge branch 'production' into terms-improvements 2017-02-15 16:31:22 +01:00
Mokaddem a696ccf83b Merge branch 'newModuleInformation' into production 2017-02-15 16:30:44 +01:00
Mokaddem 85a6f6fa63 Fixed a bug (str+float) 2017-02-15 16:30:31 +01:00
Mokaddem 25adae0104 Added possibility to select in top_terms the data per_paste or not (need module testing) 2017-02-15 16:29:02 +01:00
Mokaddem e98e92b0d4 Merge branch 'fix-linkPaste-debug' into production 2017-02-15 13:43:03 +01:00
Mokaddem c552aced42 Merge branch 'newModuleInformation' into production 2017-02-15 13:42:46 +01:00
Mokaddem 1b0d9d9507 Fixed str and int concat + Added sorting by queue name in queue not running 2017-02-15 13:42:21 +01:00
Mokaddem 74e0bc7acb Merge branch 'newModuleInformation' into production 2017-02-15 11:22:50 +01:00
Mokaddem 7fc9ba18d7 Added support of ENTER for navigation 2017-02-15 11:01:11 +01:00
Mokaddem b84194542d Moved logging into a function 2017-02-15 10:57:41 +01:00
Mokaddem 0fce3f5d65 Renamed variables + added comments 2017-02-15 10:37:59 +01:00
Mokaddem 9ca3e79e70 Removed useless comments + commented 2017-02-15 10:19:03 +01:00
Mokaddem 9199ea3f0e Added comments 2017-02-15 10:04:51 +01:00
Mokaddem ff1df07b5d Added comments and moved portion of code for slightly better visibility 2017-02-15 09:43:18 +01:00
Mokaddem b272b1b07b Added a way to quit application with 'q' more or less properly 2017-02-15 09:35:03 +01:00
Mokaddem 0254c51164 Added end of paste marker + handling screen resize 2017-02-15 08:26:48 +01:00
Mokaddem 0b9609e4a9 Merge branch 'newModuleInformation' into production 2017-02-14 17:22:06 +01:00
Mokaddem 444f6071b7 Fixed screen bug by adding an input before starting asciimatics + improved path localisation in the logs from modules 2017-02-14 17:21:52 +01:00
Mokaddem b04bac4731 Merge branch 'newModuleInformation' into production 2017-02-14 16:08:42 +01:00
Mokaddem d520c45a15 Added support of display of the last paste processed by a module 2017-02-14 16:05:59 +01:00
Mokaddem d3b4409156 Added path for debug in SQLInjection 2017-02-14 11:23:34 +01:00
Mokaddem 135a1457e0 Merge branch 'fix-linkPaste-debug' into production 2017-02-14 11:12:22 +01:00
Mokaddem d4da3a316f Added path of the paste in the log of Categ.py 2017-02-14 10:59:47 +01:00
Mokaddem 303575b2a1 Merge remote-tracking branch 'upstream/master' into production 2017-02-14 09:31:18 +01:00
Alexandre Dulaunoy 2f4e5b9016 Merge branch 'master' of github.com:CIRCL/AIL-framework 2017-01-26 07:16:44 +00:00
Alexandre Dulaunoy cf903cc212 (partially fix #90 too) using a simple Alarm (SIGNAL) when exec-timeout 
Introducing a timer (in this case 60 seconds) to ensure that the
execution time of the analysis takes less than 60 seconds. This
is a simple and standard POSIX signal handler. If the timeout
is reached, the module will process the next one.

This approach fixes the specific issues we have currently
with some inputs where the sentiment analysis takes too much time. This
fix should be improved and be more generic:

 - Introducing statistics of content which timeouts.
 - Keeping a list/queue to further process those files using a different
   analysis approach. Maybe a set of "dirty" processes to handle the edge cases
   and to not impact the overall processing and analysis.
 - Make the timer configurable per module (at least for this one).
 2017-01-26 07:11:18 +00:00
Mokaddem c422db6e6d Merge remote-tracking branch 'upstream/master' into production 2017-01-17 16:38:48 +01:00
mokaddem b3a3616b0b Merge pull request #102 from ptitdoc/bugfix1 
Bug fixes in Workers and improvement of pystemon-feeder
 2017-01-17 16:35:32 +01:00
Mokaddem ea423fde0f Added requirement, better behaviors and confirmation box 2017-01-17 15:58:16 +01:00
Mokaddem 692f13d18b Improved module + Added support of click and keyboard navigation for killing and starting modules 2017-01-17 14:31:06 +01:00
Mokaddem 3b8149d840 Added action button and support of logs 2017-01-16 17:08:48 +01:00
Mokaddem 0ecfaf00f5 Added color in front of queue related to its state 2017-01-16 15:30:33 +01:00