CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
1,295 Commits
6 Branches
43 Tags
110 MiB
Commit Graph

463 Commits (71a6860323a3e7a50f77e0ad878ee51534b8daf0)

Author SHA1 Message Date
Mokaddem 32dfd2b181 Moved legend place in term-plot-top 2016-08-23 10:16:33 +02:00
Mokaddem f9b27e4013 Fixed bug in curve preventing to track concerned_pastes. Now all tracked words are put in lower case. 2016-08-23 09:50:03 +02:00
Mokaddem 1766ced9e6 Fixed potential XSS problem with pastes display in 'search' and 'important_paste_by_module'. 2016-08-23 09:08:17 +02:00
Mokaddem 047331df3e Added save of paste_baseName in set for searches. This is done in ModuleStats to not surcharge the Global module. 2016-08-23 08:52:54 +02:00
Mokaddem 6be557890c Replaced negative hour by 24h based. 2016-08-23 07:19:51 +02:00
Mokaddem 85ab0e36ff In sentiment analysis. if today top provider set is empty, try to take yesterday data and then takes from all_providers. 2016-08-23 07:11:03 +02:00
Mokaddem 8359a6668f Suppressed useless comms 2016-08-22 21:36:19 +02:00
Mokaddem 2fd2c8af39 Removed commented code 2016-08-22 20:59:56 +02:00
Mokaddem 27ae577e00 Fixed stupid line remove in index.html... 2016-08-22 16:25:52 +02:00
Mokaddem a424069c64 Fixed bug in flask with zrange (was taking the bottom of the set instead of the top leading to get only the min instead of max) 2016-08-22 16:12:20 +02:00
Mokaddem 003c72bd7c Fixed bug with redis where the inserted key was too long, causing a crash + fixed bug taking min instead of max in terms-top-set 2016-08-22 16:01:42 +02:00
Mokaddem 2466f355ab Fixed bug in manage-terms-top-set + added auto sorting in important-paste-by-module 2016-08-22 14:54:37 +02:00
Mokaddem e144e8e1c6 Fixed bug in terms-management-interface (only display one paste) + removed get_language of all pastes due to the huge overhead. 2016-08-22 13:59:12 +02:00
Mokaddem d3c0fc5f94 Added terms trending module and web interface: 
- Curve module has been rewritten
- Added new module curve_manage_top_set
- Added all web-pages
 2016-08-22 13:35:49 +02:00
Mokaddem 841e87eb16 Fixed bug in sentiment trending, if there was no data for the avg -> there was one NaN leading to no display in the gauge. 2016-08-20 18:48:24 +02:00
Mokaddem ee8f26c7c4 In sentiment trending, if there are no providers in the today\'s set, get data from random providers 2016-08-20 18:14:19 +02:00
Mokaddem 2ed60cb9bf Fixed typo 2016-08-20 16:44:36 +02:00
Mokaddem 3f1cf705fc Deleted alert() 2016-08-20 16:41:49 +02:00
Mokaddem 1c96eac2e4 Enter simulate button click on input field for terms management and plot tool 2016-08-20 16:38:51 +02:00
Mokaddem 465d18fc73 Added terms-plot tool 2016-08-20 16:23:06 +02:00
Mokaddem fb3dea6bd2 Finnished terms-plot-top-set. Also added possibility to add terms to track or blacklist. 2016-08-20 15:59:22 +02:00
Mokaddem a5117d7cad Added top set webpage 2016-08-20 11:46:22 +02:00
Mokaddem 0b6828fb16 Correctly handle and display add or remove a term in term-frequency + started sketch of terms-plot tool. 2016-08-19 16:53:46 +02:00
Mokaddem 8d41b109e7 Added query of the data for the new tracked word 2016-08-19 14:33:01 +02:00
Mokaddem b78effe779 Added draft of terms_management + Added copyright in readme 2016-08-19 13:34:02 +02:00
Mokaddem 52b7124a6b - Modified redis connection (from levelDB to redis). 
- Added term frequency in curve.
- Modified ModuleStats and WebStats to use redis special command (incrby instead of get-set and zset) and Flask to perform the correct queries.
- Added panel color in sentiment trending
 2016-08-18 15:34:19 +02:00
Mokaddem 24072ece36 Added support of pasteName in search + added header in search.html + supressed mixed indent and trailing in Flask.py 2016-08-17 14:21:31 +02:00
Mokaddem c10f28d9df Updated isolated launch scripts + Added switch displaying queue in index 2016-08-17 13:46:01 +02:00
Mokaddem 23b65eddca Auto unchecked info checkbox in index 2016-08-17 11:25:05 +02:00
Mokaddem 4d09ab0f6b Modified icon sentiment in web-header. 2016-08-17 11:13:37 +02:00
Mokaddem 44f54b7b81 Added dependencies 2016-08-17 11:04:48 +02:00
Mokaddem f0b527167a Added script flexgauge 2016-08-17 10:50:30 +02:00
Mokaddem c12e67a35a Added tooltips + slight gauge modification 2016-08-17 10:43:43 +02:00
Mokaddem 894b9efda9 Added drop of really long line in sentiment-analysis module + Added description of sentiment module. Also, fixed bug in webpage sentiement-trending concerning avg and date range. 2016-08-17 09:46:25 +02:00
Mokaddem 1084e45f1b Added draft of filter in sentiment analysis (Discard syntaxical languages) + Added nice tooltip for sparkline. Trending displays avg in function of the number of elements processed and not for the complete week + fixed bug in gauge and canvasjs (was performing avg with only 1 graph instead of all 8). 2016-08-16 16:33:02 +02:00
Mokaddem ecd834ffb6 Added autoreload in sentiment-trending. 2016-08-16 11:27:49 +02:00
Mokaddem bb677b9061 Added avg text in sentiment-trending + fixed date range bug 2016-08-16 11:23:14 +02:00
Mokaddem cdd08d7df0 Added all providers for plot_tool in moduleStats + fixed date range bug 2016-08-16 11:07:36 +02:00
Mokaddem 8fa3c2b147 Added stacked choice or not for plot-tool (sentiments) 2016-08-16 10:49:05 +02:00
Mokaddem 4baac0b994 Tool-plot for sentiment working + Splitted script and webpage 2016-08-16 10:31:18 +02:00
Mokaddem 1e7547d16c Created draft of search for plot tool 2016-08-15 16:32:02 +02:00
Mokaddem 88ed8a239d Linked plot button with server 2016-08-15 15:54:53 +02:00
Mokaddem 82f70e61f4 Added provider checkbox 2016-08-15 15:16:31 +02:00
Mokaddem f90987d284 Added date on slider 2016-08-15 14:53:46 +02:00
Mokaddem d8c84ec674 Added link to data in analysis trending + fixed tooltip bug in sparkline 2016-08-15 12:57:39 +02:00
Mokaddem 6c7f660052 Added support of week and day average. 2016-08-13 22:17:11 +02:00
Mokaddem 1a5158b081 Added sentiment analyser module (draft) 2016-08-13 15:24:57 +02:00
Mokaddem 0431ecf434 Modified webpage display and added js. 2016-08-12 10:22:24 +02:00
Mokaddem 9c3b573af2 Splited js from html + Added some ID + small display modification. 2016-08-12 07:49:24 +02:00
Mokaddem 87a86b38d1 Added page Plot-tool 2016-08-11 21:08:43 +02:00
Mokaddem c028ca80da Separated options from the plot funcion for sparkline. 2016-08-11 17:57:23 +02:00
Mokaddem 98c4cacb98 Finished first panel in sentiment-today. 2016-08-11 17:22:50 +02:00
Mokaddem 8682fcc29c Added empty webpages for sentiments modules. 2016-08-11 11:34:22 +02:00
Mokaddem 5612a77af9 Separated webpage header code from all webpage 2016-08-11 10:55:51 +02:00
Mokaddem 0ba5637e06 Added bar chart hover binder in trending charts 2016-08-10 11:16:56 +02:00
Mokaddem 24318f6a86 Added the part 'other' in Trending chartscd 2016-08-10 10:42:06 +02:00
Mokaddem 1ca8cf69f6 Added trendingchart.js 2016-08-10 09:38:15 +02:00
Mokaddem b249676fe9 Removed console log 2016-08-09 16:54:53 +02:00
Mokaddem 507c83e06d In index webpage, reset value graph if no paste received 2016-08-09 16:48:57 +02:00
Mokaddem 943016d6ca Fixed one of the small display bug in DUplicate_ssdeep_v2 (was displaying [tlsh, tlsh] instead of simply tlsh) 2016-08-09 15:06:55 +02:00
Mokaddem 0d16a8beb7 Fixed bug where Listener was not placed if the row was not displayed (BrowseWarningPaste) 2016-08-09 12:30:40 +02:00
Mokaddem e2ba4060ee restaured loading gif when new tab is shown (BrowseWarningPaste) 2016-08-09 12:07:20 +02:00
Mokaddem a80f6ca0c2 Bug fix related with redis: 
Fixed typo key in redis for module creditcard and sqlinjection
Modified Curve redisLvlDb server
Modified Url.py so that it forwards name of protocol from saved protocolsfile
Added Cache control in Flask
Modified key-tab name into keys-tab
 2016-08-09 11:59:36 +02:00
Mokaddem 084737dd26 Fixed bug in flask not correctly displaying the number of paste/providers 2016-08-08 11:39:17 +02:00
Mokaddem 78c611fead Added warning_paste module and created related webpages. 
Fixed a Faup bug in credential (multiple instanciation) and added correc populate_set_out in concerned modules (creditcard, credential, ...).
Linked browse_warning_paste module and Flask function with redis (created new sets).
 2016-08-08 09:17:44 +02:00
Mokaddem 3523d79a05 Started browse important paste feature. Still need to link with redis. + Fixed a bug in Flask (String to list parser) 2016-08-04 16:57:34 +02:00
Mokaddem 55ef917ef1 Added few comments 2016-08-04 12:10:56 +02:00
Mokaddem d9316771cd Draft: added new duplicate hash comparison - tlsh 2016-08-04 11:55:38 +02:00
Mokaddem 50d2848a40 Tryed to fix the glob_tabvar bug 2016-08-02 16:39:57 +02:00
Mokaddem 9e82ec82b7 Fixed a bug in the hover of provider graph (happened when there was an 'other' in other graphs) 2016-08-02 16:17:25 +02:00
Mokaddem 9d37ebc860 Fixed a division by 0 in Flask + Harmonized webpage item positioning 2016-08-02 16:00:31 +02:00
Mokaddem f3cf2f853a Fixed a bug in the hover of providers graph. 
When we click on a part in providers graphs, we hover an item, then we click on a random part not belonging to providers graphs, and we re-hover one part of the providers graphs, there was an undefined variable (The random click overrided it).
 2016-08-01 11:08:53 +02:00
Mokaddem feb1717dc6 fixed size bug, which in some cases display the total size instead of the average + few cleanup + commented out log scale in moduleStats.js and added large number division in case of div by 0 2016-07-29 16:52:50 +02:00
Mokaddem a827e8ca3a Big refactoring and added lot of comments + fixed bug attaching multiple binder on barChart 2016-07-29 14:30:41 +02:00
Mokaddem e8fae17cfd Fixed small race condition bug in ModuleStat.py and Added highlight of bar belonging to the same serie in moduleStats.js 2016-07-29 10:36:44 +02:00
Mokaddem c2bf2ba630 Added tooltip in moduleStat above each bar chart + Added highlight of the selected bar. (Still not working for category 'Other') 2016-07-28 16:35:30 +02:00
Mokaddem 09fdaa3e39 Added tooltip for moduleStats bar chart 2016-07-28 12:06:54 +02:00
Mokaddem acec508f53 fixed multiple binding bug in moduleStats + Added multiple bar display for 'other pie part' + Added ModuleStats tab in all other pages 2016-07-28 09:52:07 +02:00
Mokaddem 5a02a7ffce Clean up and refactoring for back-end moduleStats 2016-07-27 11:55:57 +02:00
Mokaddem 27e0d5c578 Clean up and refactoring for front-end moduleStats 2016-07-27 11:26:56 +02:00
Mokaddem 240bdd9cca Added number of paste by provider in module stats 2016-07-27 09:27:32 +02:00
Mokaddem 63774cd160 In moduleStats, if there is data which have small percentage, display it in an 'other' part + Added stacked chart for them (still draft) 2016-07-26 16:56:52 +02:00
Mokaddem 5bab1a6f17 Added the click on a part in module stats 2016-07-26 12:12:26 +02:00
Mokaddem c07c12088d In moduleStat, display old data if there is no data today and notify the user. 2016-07-26 12:07:52 +02:00
Mokaddem baa042827d Removed useless comments + prevent trying to add an already present element in the set 2016-07-26 11:06:31 +02:00
Mokaddem cc1b49baaf Fixed variable bug in moduleStats and added draft of credential stats 2016-07-26 10:45:02 +02:00
Mokaddem 06be1f129a Added Draft of ModuleStats + Paste size average 2016-07-25 16:38:57 +02:00
Mokaddem b076765cf4 Added dependency flot.time 2016-07-21 16:35:49 +02:00
Mokaddem 7c4f4a2f79 Merge branch 'statistics-v2' into uptodate-statistics-v2 2016-07-21 15:16:03 +02:00
Mokaddem b685f81752 Merge branch 'duplicate-module' into uptodate-duplicate-module 2016-07-21 14:59:52 +02:00
Mokaddem 9a34a587cc bug global_tabvar seems solved: 
Caused by a race condition, Just switched two functions execution
 2016-07-21 14:53:34 +02:00
Mokaddem 6f4bfeb4ef restored deleted part of a failed merge-conflict 2016-07-21 14:45:41 +02:00
Mokaddem 37557da8a2 Merge remote-tracking branch 'upstream/master' 2016-07-21 14:04:18 +02:00
Mokaddem ef76d7aba3 re-ordered tab in trending charts 2016-07-21 13:48:44 +02:00
Mokaddem 0ab7560477 Added top_progression chart for tld, domain and scheme + Small modification in config file. 2016-07-21 13:44:22 +02:00
Raphaël Vinot d2557aae9a Update web thirdparty modules 2016-07-20 14:11:39 +02:00
Raphaël Vinot a0a4d0dff3 Cleanup and remove useless deps 2016-07-19 18:07:29 +02:00
mokaddem 7e5cf7cc55 Added dependency for flot chart 2016-07-19 16:51:29 +02:00
mokaddem 4bc84a2580 Added dependency for flot chart 2016-07-19 16:49:57 +02:00
Raphaël Vinot 8a69110c8e Merge branch 'webpage-splitting' of https://github.com/mokaddem/AIL-framework into mokaddem-webpage-splitting 2016-07-19 16:35:31 +02:00
Mokaddem a6996c0b23 Added related functions and display of duplicated paste in search.py 2016-07-19 10:48:44 +02:00
Mokaddem 4ecb9a4b44 Fixed a flexibility issue 2016-07-15 09:47:00 +02:00
Mokaddem 321064bf0d Fixed a bug in processed_pastes graph 2016-07-14 11:17:08 +02:00
Mokaddem fba14bfb4b In index: Added number of processed pastes chart 2016-07-14 11:16:07 +02:00
Mokaddem 60552bca4d Fixed a bug in processed_pastes graph 2016-07-14 10:31:47 +02:00
Mokaddem 594d2def35 In index: Added number of processed pastes chart 2016-07-13 15:57:33 +02:00
Mokaddem 921da8aeb0 Display the correct active tab in navbar. 2016-07-12 14:44:59 +02:00
Mokaddem 4e4dfb2782 Searching now works in trending graphs. 2016-07-12 14:38:22 +02:00
Mokaddem 6305f1eba7 Deleted redundant code and separeted all trendings chart from the main page. 2016-07-12 14:33:54 +02:00
Mokaddem 7e5ce0f17f Optimized create_plot and removed test commemts 2016-07-12 11:47:51 +02:00
Mokaddem 4ac448e71e restored working search>html + Added max curves display to all graphs. 2016-07-12 10:02:01 +02:00
Mokaddem 8cd36982b8 Added a dynamic limit on the number of elements to display. Not fully tested. (recovered old doc, searching not working) 2016-07-11 16:57:45 +02:00
Mokaddem f6c7917149 Graphs now displayed the highest value. Plotting and filtering is done in its own script. 2016-07-11 15:05:04 +02:00
Mokaddem 880359265c Renamed tldstrending into trending and added graphics into their respective pannels. Separated graph generation from html source code for wordtrending and protocoltrending 2016-07-08 15:08:37 +02:00
Mokaddem 0174d46788 Added loading GIF when searching and fixed small display bug in modal 2016-07-08 12:09:33 +02:00
Mokaddem f475f18f06 Added comments and renamed variables and separeted chunks of codes into function 2016-07-08 10:19:24 +02:00
Mokaddem f846255d84 fixed small sorting problem (date) 2016-07-08 09:16:43 +02:00
Mokaddem 6d760b8dfc Added dataTable resources in update-third-party.sh 2016-07-08 09:04:19 +02:00
Mokaddem e206d27f63 Added query search display 2016-07-07 16:43:18 +02:00
Mokaddem 465244e1ce Added dynamic table sorting in search page. (Still need to add dependencies) 2016-07-07 16:38:00 +02:00
Mokaddem f1979f6dcb Search preview modal more dynamic and responsive 2016-07-07 15:34:08 +02:00
Mokaddem 74b337f1a1 search preview modal do not duplicate anymore but still needs improvements 2016-07-07 09:31:04 +02:00
Mokaddem f29620cf96 Added load more content in search modal preview. Works fine but may duplicate content (need to fix). 2016-07-06 16:54:27 +02:00
Mokaddem 3345124219 Added pannel in search modal + small fix related to config 2016-07-06 14:48:27 +02:00
Mokaddem 9209d48e30 searching now working with preview, details and redirection to the paste display page. 2016-07-06 14:30:32 +02:00
Mokaddem 7ff9b9a583 Added DomainTrending seems working. 
Started search features with related html pages, not finish yet.
 2016-07-05 16:53:03 +02:00
Mokaddem 8c1eeea6e6 created new graph pannel for further url stats and dissociate graph script from html 2016-07-04 11:43:38 +02:00
Mokaddem 5a9eca9291 Added few comments 2016-07-04 09:18:23 +02:00
Mokaddem 4b3101b7b6 Added template tld. Modified URL using Faup and refactored WebStats. 2016-07-01 16:59:08 +02:00
Mokaddem beeeb76de9 Added new modules and started WebTrending web interface 2016-06-30 14:38:28 +02:00
Mokaddem 3dc356dc5e Getting Started: Initial configuration working on the laptop 2016-06-30 14:36:47 +02:00
Alexandre Dulaunoy d89ae365f9 web: basic search functionality added 2014-12-24 14:42:20 +00:00
Alexandre Dulaunoy 9b7e329888 Font awesome link fixed 2014-12-22 20:51:05 +00:00
Alexandre Dulaunoy c2d4e8b437 logo added and layout changed 2014-12-22 17:08:02 +00:00
Alexandre Dulaunoy dc57454d82 Update start-bootstrap urls 2014-12-22 15:15:56 +00:00
Alexandre Dulaunoy f15caa70f5 ultag removed 2014-09-17 17:24:08 +02:00
Alexandre Dulaunoy b36cf76889 Removed ultag 2014-09-17 17:24:08 +02:00
Raphaël Vinot 7542eaf739 Update starting script. 2014-09-02 15:21:36 +02:00
Raphaël Vinot abfe13436b Big refactoring, make the queues more flexible 2014-08-29 19:37:56 +02:00
Raphaël Vinot d7a7da4814 Make sure the webserver fails properly if there is no config file. 
Create the queue list in a more pythonesque fashion.
 2014-08-26 17:33:28 +02:00
Alexandre Dulaunoy 1567e452f5 Cleanup: removed unused parameter 2014-08-26 09:25:28 +02:00
Raphaël Vinot 078c8ea836 Big cleanup, pep8 2014-08-14 18:07:18 +02:00
Starow 7a3c216787 Adding a js library for the tag cloud (to install with update_thirdparty.sh 2014-08-14 10:04:39 +02:00
Starow 6cce9f7dac Javascript modif (Tagclouds) ... 2014-08-13 17:32:24 +02:00
Starow 0d25b960af logs display modifications 2014-08-13 10:59:47 +02:00
Starow 1df2ffd094 Splitting the graph in two one for words, one for paste monitoring 2014-08-12 16:44:35 +02:00
Starow 731141c723 Padding log hours + adding some super nice bootstrap icon :) 2014-08-12 16:11:11 +02:00
Starow 5a4218035a minor javascript update 2014-08-12 15:48:53 +02:00
Starow 6aa4d7cb7d Harmonising logs messages + Changing some dygraph options 2014-08-12 15:42:16 +02:00
Starow c3948867ad Adding an option to have better view of the curves 2014-08-12 14:51:03 +02:00
Starow 8d6c2f4288 Improving javascript and Fixing minor curves drawing bug 
Everything now is fully "dynamic"
 2014-08-12 14:00:00 +02:00
Starow ef6bece82c Major javascript update: Queues curves are now being set dynamically. 
Now the javacript part is a little bit less disgusting than before.
 2014-08-12 11:41:56 +02:00
Starow 60cfc9789f Adding logging proc time inside the table 2014-08-11 16:29:38 +02:00
Starow e96be56b82 Checkbox default checked at true. 2014-08-11 15:30:44 +02:00
Starow f02cb10cac Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2014-08-11 15:08:16 +02:00
Starow 25cebf314c Index.html log table modification 2014-08-11 15:07:55 +02:00
Starow 7cf05f959d Javascript modification dashboard log table 2014-08-11 15:06:38 +02:00
Starow 80fe77da26 Making bigger the graph 2014-08-11 13:46:02 +02:00
Starow a7679787b7 Removing "Dashboard" Title to win some place for displaying infos.. 2014-08-11 13:41:58 +02:00
Starow edd4ea9624 Queue name tab moved to left side of the dashboard 2014-08-11 11:15:44 +02:00
Starow 2306783882 Updating dashboard navigation menu list 2014-08-11 10:48:03 +02:00
Starow af827a3402 Removing the hardcoded IP adress 2014-08-08 13:49:38 +02:00
Starow 1379ef705a Initial import of AIL framework - Analysis Information Leak framework 
AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past
ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen
sitive information
 2014-08-06 11:43:40 +02:00