CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
1,095 Commits
7 Branches
44 Tags
110 MiB
Commit Graph

385 Commits (979472df15b0a20b3e11f35a9f5c9d0b82e2c6d7)

Author SHA1 Message Date
Terrtia f66a528bc2
fix Duplicate, save list of duplicates on disk + prevent empty hash creation 2018-05-15 23:28:47 +02:00
Terrtia 225fe76c96
fix duplicate + get raw paste 2018-05-11 16:07:30 +02:00
Terrtia 142c1dc9b0
clean 2018-05-09 13:03:46 +02:00
Terrtia 3841c159a0
change leveldb to ardb 2018-05-07 14:50:40 +02:00
Terrtia c17b9402f7
clean 2018-05-04 14:25:47 +02:00
Terrtia 96a3df42bc
decode with redis connection 2018-05-04 13:53:29 +02:00
Terrtia a900d5f08d
add python 2 to 3 bash + fix Duplicate + clean 2018-05-03 16:21:33 +02:00
Terrtia c52caebe7c
improve Duplicate + tlsh + add time out handler + debug + clean 2018-05-02 17:07:10 +02:00
Terrtia c3428df5dc add apiKeys module 2018-04-26 14:42:39 +02:00
Terrtia 8738b7cf75 fix track term and avoid duplicate mail address 2018-04-24 16:44:37 +02:00
Terrtia f93fe9aeb2 add bitcoin module 2018-04-23 14:51:53 +02:00
Terrtia 41eb250278 fix python 3 compqtibility issues 2018-04-20 10:42:19 +02:00
Terrtia 8571a86344 frontend python 3.5 upgrade 2018-04-17 16:06:32 +02:00
Philipp Schmied 4835589ff4 Fixed typo 2018-03-01 08:50:27 +01:00
Philipp Schmied 29608ab29b Fixed e-mail notificatins for sets and regexes 2018-02-28 11:31:16 +01:00
Philipp Schmied b6fcf51309 Flask_terms.py: Fixed mapping of notifcation e-mails to terms with uppercase letters 2018-02-28 09:19:27 +01:00
Philipp Schmied 5b1f0b0212 Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms. 2018-02-27 15:12:02 +01:00
Alexandre Dulaunoy abdf41fe23
Merge pull request #164 from mokaddem/showDiff 
Show differences between duplicates
 2018-02-21 22:25:16 +01:00
Sami Mokaddem 9125930a77 fix: canvasjs changed its folder hierarchy 2017-12-14 11:47:11 +01:00
Sami Mokaddem 304213cfa4 update: Possibility to change the authorized max line length in the config for the diff of duplicates 2017-12-12 11:00:26 +01:00
Sami Mokaddem 3fe7ecf75a feature: diff in the web interface for duplicates pastes. 2017-12-12 10:48:21 +01:00
mokaddem 9cab76cf88
Merge pull request #162 from mokaddem/sentimentBugfix 
Fix: bug between front-end and back-end date representation
 2017-12-11 16:35:38 +01:00
Paul A 4a4a1d478b Quick refactor for the JavaScript 2017-10-22 18:48:54 +02:00
Paul A f6b6777095 Fixed some code for the JavaScript (coding-style) 2017-10-20 15:44:09 +02:00
Paul A 48fc8e3464 Removed duplicate function 2017-10-20 15:32:58 +02:00
Simon Tran a41c5d55c5 Migrated Dockerfile to Ubuntu 16.04 LTS 
Fixed AIL Web dependencies installation
Added Docker documentation
 2017-10-01 02:23:32 +02:00
Sami Mokaddem 2cfb8c173b Fixed from front-end date representation bug 2017-09-27 10:46:20 +02:00
= b9bd651033 Removed useless import 2017-08-23 14:06:59 +02:00
= 3b3f3aa89c Flask: Added support of LevelDB database depending of the year 2017-08-23 12:20:39 +02:00
= d632335760 Automatically start redis_levelDB with the correct date 2017-08-23 10:16:56 +02:00
= bd46c83fb6 BrowseImportantPaste: Fixed a bug where onClickListener was not set when the table was only 1 page long 2017-08-21 11:55:01 +02:00
= 409eb25a64 Deleted data-flow graph and header.html from the index + Added fixed typo where 10min was in fact 3min 2017-08-21 10:51:56 +02:00
Mokaddem fa52292029 Added comments, cleanup and added credentials search progress in Flask 2017-07-20 10:50:24 +02:00
Mokaddem cbb2909156 Added comments 2017-07-20 10:24:48 +02:00
Mokaddem 05417f405c Finished basic link web interface + solved some bugs 2017-07-20 10:04:30 +02:00
Mokaddem ef741f85c0 Added similarity checks + started link webInterface 2017-07-19 11:52:06 +02:00
Mokaddem 882ab88b00 Draft module + web link 2017-07-18 16:57:15 +02:00
Mokaddem 9b8570e40b Dirty draft webpage credential_tracking 2017-07-17 17:26:19 +02:00
Mokaddem f1a1e75c32 Restaured old example webpage 2017-07-17 16:18:21 +02:00
Mokaddem 9558aafe67 Prevent the loss of the 1 message in import_dir.py 2017-07-17 16:14:27 +02:00
Mokaddem b60e164380 Added support of onion in browseWarningPaste 2017-04-26 11:47:02 +02:00
Mokaddem 5e4d0170d4 Bug fixes, jinja2 iter0 and typo 2017-04-26 11:31:08 +02:00
Mokaddem e614da4748 Deleted print + Added curve fill for Mixer graph in Index 2017-04-25 15:52:41 +02:00
Mokaddem 71b2757ed2 New script create_new_web_module permits to rapidly create a new pre-configured webpage from a template. 2017-04-25 15:22:04 +02:00
Mokaddem 6afbf77af0 Modified skeleton template name into rawSkeleton + Added possibility to hide modules in the header. 2017-04-25 12:18:08 +02:00
Mokaddem f6e8582d9d Auto organisation of the navbar -> Web page are auto-inserted. Created skeleton webpage 2017-04-19 15:14:20 +02:00
Mokaddem 32906965d8 Draft of new organisation of the web interface. Usage of Flask's blueprint 2017-04-19 11:02:03 +02:00
Mokaddem da081bae42 Merge branch 'regexInTerms' into production 2017-04-18 16:59:40 +02:00
Mokaddem c8baabd882 Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...) 2017-04-18 15:28:21 +02:00
Mokaddem 2da4c572c7 New feature in Terms: regex and set of words support (draft) 2017-03-28 17:42:44 +02:00
Mokaddem 3261eca836 Fixed all_index newline bug + added print of what's happening 2017-03-15 14:29:49 +01:00
Mokaddem 5b7d047622 Changed os.system by os.mkdir + fixed bug during rotation 2017-03-15 14:05:13 +01:00
Mokaddem 4005b72180 Added backward support + renamed index with timestamp 2017-03-15 11:51:35 +01:00
Mokaddem 381e72ee99 Added display of number of elements inside the index + changed variables names 2017-03-15 10:07:46 +01:00
Mokaddem e4757f5ceb Added possibility to choose the index in search + Updated search page with jinja2 iter0 2017-03-15 09:39:48 +01:00
Mokaddem 41132fe0bc Added support of per_paste for term-management and plot-term 2017-02-28 15:54:39 +01:00
Mokaddem 4071bf2e91 fixed a bug in manage_term (usage of loop.index0) + fixed bug in top_set where graph was not displaying the correct per_paste value 2017-02-28 15:01:48 +01:00
Mokaddem 25adae0104 Added possibility to select in top_terms the data per_paste or not (need module testing) 2017-02-15 16:29:02 +01:00
Mokaddem d4da3a316f Added path of the paste in the log of Categ.py 2017-02-14 10:59:47 +01:00
Mokaddem 29241eb49f Added a line separator between the two graphs in dashboard. 2017-01-09 14:44:14 +01:00
Mokaddem 0e39f516a9 Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed. 2017-01-09 14:12:26 +01:00
Mokaddem 24c5621d29 BUG FIX in Flask_trendingmodule, Fixed member_set when get_top_relevant_data is None. 2016-12-26 16:33:10 +01:00
Mokaddem 1c2169d3bc Improved interface for multiple feeds and refresh_script. 2016-12-26 16:16:44 +01:00
Mokaddem a18c046dea Fixed bug multiple refresh instances and harmonized interface. 2016-12-23 16:15:05 +01:00
Mokaddem 03dddbc359 Reduced refresh rate of processed_pastes, synchro graphs and adjusted max on each graphs. 2016-12-23 16:01:30 +01:00
Mokaddem 97292e0899 Updated web interface to handle new mixer module. and fixed one dependency bug. 2016-12-23 15:44:46 +01:00
Mokaddem faa8f8302d Removed typo in the number of remaining pastes to display in browseImportantPaste 2016-12-22 09:28:55 +01:00
Mokaddem 9df12cdae8 Added sub-flask in a directory 2016-12-22 08:57:45 +01:00
Mokaddem d30f3ca6c7 Added missing depency datetime in dashboard 2016-12-09 13:53:57 +01:00
Mokaddem 0054be5bab Added missing json dependency in dashboard 2016-12-09 08:50:36 +01:00
Mokaddem 8daa72789e Splitted Flask_server into module related to website sections 2016-12-09 08:46:37 +01:00
Mokaddem d3ed1bbf40 Merge branch 'website-improvement' into production 2016-12-08 10:05:55 +01:00
Mokaddem 1abba4dcf9 Added support of re-plotting the plotted terms 2016-12-08 08:44:10 +01:00
Mokaddem 84dcd5586f Started plotting same data with different dates 2016-11-25 16:42:22 +01:00
Mokaddem a3255d168c ModuleInfo correctly handle CurveManageTopSets, 
Changed queue coloring in dashboard,
ModuleInformation now have a history of executed command and better handle module killing.
 2016-11-25 11:54:16 +01:00
Mokaddem 5d269ea1ee Added date in Duplicate for better distinguish duplicate. Also, added a timeout for the dataTable in duplicate inside the modal. 2016-11-24 15:05:29 +01:00
Mokaddem df8d978088 Added dynamic data loading in dataTable in search.html 2016-11-24 13:31:31 +01:00
Mokaddem 80ca49f018 Prevent browser to request reconnection for the eventStream + Added dynamic data loading in the dataTable of BrowseImportantPastes 2016-11-11 16:51:39 +01:00
Mokaddem 768fb6f2df removed server overhead for paste preview data 2016-11-10 16:41:07 +01:00
Mokaddem 699c819387 fixed bug in show concerned pastes previex en (terms managements) 2016-11-10 16:33:05 +01:00
Mokaddem 94e3034e7d Fixed bun in terms management, where action_button was not bind in other dataTable pages 2016-11-10 15:53:22 +01:00
Mokaddem 7763bfb4c7 Pastes dataTable now display only the latest clicked paste even if there were discarded queries 2016-11-10 15:39:45 +01:00
Mokaddem 2dd69fbb25 Fixed modal background stacking bug + button leading to saved paste bug fixed in search 2016-11-10 14:56:25 +01:00
Mokaddem 317005c2b4 tryed to fix html display problem in tooltip + fixed problem button showPaste in modal not pointing to the paste 2016-11-10 10:30:29 +01:00
Mokaddem 34221b083f Partially solved the browser freezing problem + tootlip is also binded at a new page generation of the DataTable 2016-11-10 09:28:55 +01:00
Mokaddem acdd1367a3 Added event source for getImportantPaste 2016-10-28 16:55:56 +02:00
Mokaddem c95000866d Dynamic table in search now load all the data + fixed bugs where tooltip where not shown on other than the first page displayed and 
tooltip interpret html (not supposed to...)
 2016-10-28 14:21:08 +02:00
Mokaddem f51808d914 Added support of html formatting in search result dynamic loading table 2016-10-28 09:48:22 +02:00
Mokaddem 4be62a4e2d Added draft of dynamic update table with search result 2016-10-28 09:28:42 +02:00
Mokaddem 340624fe1e Fixed bug in modal show paste (show more button not showing) 2016-10-27 16:18:49 +02:00
Mokaddem 7e7e679ab6 Added dataTable for duplicate in show saved paste 2016-10-27 15:53:45 +02:00
Mokaddem 59a7000527 Bootstrap support in show saved paste 2016-10-27 12:38:39 +02:00
Mokaddem ac254e0e63 Added support of browsing concerned paste in dashboard for all modules + Show in the page what was the concerned module 2016-10-27 11:50:24 +02:00
Mokaddem 1826b170ec Added support of local paste view in dashboard for Mails event only 2016-10-27 11:27:26 +02:00
Mokaddem 7c8d414948 modified provider name into no data when there is no data 2016-10-14 16:47:22 +02:00
Mokaddem 91678179fd Added msgs when queues or feed is not running/coming 2016-10-14 16:33:54 +02:00
Alexandre Dulaunoy 2781a2f0b7 Stuck not stucked 2016-09-21 08:35:20 +02:00
Jerome St-Hilaire ea13f87c2c Fixed dependency fetching for the web interface 2016-09-15 12:14:55 -07:00
Raphaël Vinot acfe9c1336 Update Flask_server.py 2016-08-26 18:06:48 +02:00
Raphaël Vinot 3290b51c91 Update update_thirdparty.sh 2016-08-26 18:05:18 +02:00
Mokaddem 79be8ab934 Fixed introduced bug relative to chart in index webpage. 2016-08-26 15:59:53 +02:00
Mokaddem 0c760d763b Added support of PID in ModuleInformation and index.html 2016-08-26 15:27:37 +02:00
Mokaddem 5c565068a6 Removed auto-reload in dashboard if window in undefined 2016-08-26 11:25:11 +02:00
Mokaddem 181c372131 Updated dependencies 2016-08-26 11:10:39 +02:00
Mokaddem 4cb3b672be Added comments 2016-08-26 10:28:35 +02:00
Mokaddem 86b655100e Deleted useless parsing function 2016-08-26 10:01:14 +02:00
Mokaddem 07d7559060 Added dataTable support in browse paste modal. 2016-08-26 08:30:09 +02:00
Mokaddem 813cc7031a Changed lots of code into zrank 2016-08-25 17:48:12 +02:00
Mokaddem 991176fdfd Suppressed metismenu import. 2016-08-25 16:40:13 +02:00
Mokaddem fb22816236 Added module number info in index webpage 2016-08-25 16:27:32 +02:00
Mokaddem 0b5a3e0cd2 Auto generate graph div + possibility to query all sentiments data. 2016-08-25 15:06:18 +02:00
Mokaddem 21f2c43572 Individualized sentiments-graph loading 2016-08-25 13:11:16 +02:00
Mokaddem f9bf00f022 Removed autokill feature if not specified 2016-08-25 11:09:31 +02:00
Mokaddem 611454ba82 Added color in web-index reprenting modules states. 2016-08-24 18:00:05 +02:00
Mokaddem e3daa84196 Added icon in header 2016-08-23 16:09:34 +02:00
Mokaddem e12244fb53 Added show/hide curve with legend in terms-top-set. 2016-08-23 15:39:13 +02:00
Mokaddem e56187eb7e Added position of each top term in the other sets + highligth ones not present. 2016-08-23 14:52:02 +02:00
Mokaddem a64d447b7b Auto hide button add_to_chart when no data plotted. 2016-08-23 12:09:41 +02:00
Mokaddem e59b7c4e47 Added tooltip in graph for terms-plot-tool 2016-08-23 12:03:23 +02:00
Mokaddem 0eba419651 Added multiple terms plot in terms-plot-tool. 2016-08-23 11:56:32 +02:00
Mokaddem 9f2d672ec8 harmonized tooltip and popover display 2016-08-23 11:20:48 +02:00
Mokaddem 8c956c22b1 Added autohide curves in terms-top-set. 2016-08-23 11:02:36 +02:00
Mokaddem 32dfd2b181 Moved legend place in term-plot-top 2016-08-23 10:16:33 +02:00
Mokaddem f9b27e4013 Fixed bug in curve preventing to track concerned_pastes. Now all tracked words are put in lower case. 2016-08-23 09:50:03 +02:00
Mokaddem 1766ced9e6 Fixed potential XSS problem with pastes display in 'search' and 'important_paste_by_module'. 2016-08-23 09:08:17 +02:00
Mokaddem 047331df3e Added save of paste_baseName in set for searches. This is done in ModuleStats to not surcharge the Global module. 2016-08-23 08:52:54 +02:00
Mokaddem 6be557890c Replaced negative hour by 24h based. 2016-08-23 07:19:51 +02:00
Mokaddem 85ab0e36ff In sentiment analysis. if today top provider set is empty, try to take yesterday data and then takes from all_providers. 2016-08-23 07:11:03 +02:00
Mokaddem 8359a6668f Suppressed useless comms 2016-08-22 21:36:19 +02:00
Mokaddem 2fd2c8af39 Removed commented code 2016-08-22 20:59:56 +02:00
Mokaddem 27ae577e00 Fixed stupid line remove in index.html... 2016-08-22 16:25:52 +02:00
Mokaddem a424069c64 Fixed bug in flask with zrange (was taking the bottom of the set instead of the top leading to get only the min instead of max) 2016-08-22 16:12:20 +02:00
Mokaddem 003c72bd7c Fixed bug with redis where the inserted key was too long, causing a crash + fixed bug taking min instead of max in terms-top-set 2016-08-22 16:01:42 +02:00
Mokaddem 2466f355ab Fixed bug in manage-terms-top-set + added auto sorting in important-paste-by-module 2016-08-22 14:54:37 +02:00
Mokaddem e144e8e1c6 Fixed bug in terms-management-interface (only display one paste) + removed get_language of all pastes due to the huge overhead. 2016-08-22 13:59:12 +02:00
Mokaddem d3c0fc5f94 Added terms trending module and web interface: 
- Curve module has been rewritten
- Added new module curve_manage_top_set
- Added all web-pages
 2016-08-22 13:35:49 +02:00
Mokaddem 841e87eb16 Fixed bug in sentiment trending, if there was no data for the avg -> there was one NaN leading to no display in the gauge. 2016-08-20 18:48:24 +02:00
Mokaddem ee8f26c7c4 In sentiment trending, if there are no providers in the today\'s set, get data from random providers 2016-08-20 18:14:19 +02:00
Mokaddem 2ed60cb9bf Fixed typo 2016-08-20 16:44:36 +02:00
Mokaddem 3f1cf705fc Deleted alert() 2016-08-20 16:41:49 +02:00
Mokaddem 1c96eac2e4 Enter simulate button click on input field for terms management and plot tool 2016-08-20 16:38:51 +02:00
Mokaddem 465d18fc73 Added terms-plot tool 2016-08-20 16:23:06 +02:00
Mokaddem fb3dea6bd2 Finnished terms-plot-top-set. Also added possibility to add terms to track or blacklist. 2016-08-20 15:59:22 +02:00
Mokaddem a5117d7cad Added top set webpage 2016-08-20 11:46:22 +02:00
Mokaddem 0b6828fb16 Correctly handle and display add or remove a term in term-frequency + started sketch of terms-plot tool. 2016-08-19 16:53:46 +02:00
Mokaddem 8d41b109e7 Added query of the data for the new tracked word 2016-08-19 14:33:01 +02:00
Mokaddem b78effe779 Added draft of terms_management + Added copyright in readme 2016-08-19 13:34:02 +02:00
Mokaddem 52b7124a6b - Modified redis connection (from levelDB to redis). 
- Added term frequency in curve.
- Modified ModuleStats and WebStats to use redis special command (incrby instead of get-set and zset) and Flask to perform the correct queries.
- Added panel color in sentiment trending
 2016-08-18 15:34:19 +02:00
Mokaddem 24072ece36 Added support of pasteName in search + added header in search.html + supressed mixed indent and trailing in Flask.py 2016-08-17 14:21:31 +02:00
Mokaddem c10f28d9df Updated isolated launch scripts + Added switch displaying queue in index 2016-08-17 13:46:01 +02:00