CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
472 Commits
6 Branches
40 Tags
107 MiB
Commit Graph

472 Commits (4eafca29366194be5110064a525d4164dff4caa8)

Author SHA1 Message Date
Raphaël Vinot 9e8611a42d stop killing the disk when creating the word curve 2014-09-02 18:20:28 +02:00
Raphaël Vinot 7542eaf739 Update starting script. 2014-09-02 15:21:36 +02:00
Raphaël Vinot 0c6b09f379 Fix the onion module, log the valid onions. 2014-09-01 16:18:06 +02:00
Raphaël Vinot f4b89669fc The onion module now fetches the URLs it finds. 2014-08-31 22:42:12 +02:00
Raphaël Vinot abfe13436b Big refactoring, make the queues more flexible 2014-08-29 19:37:56 +02:00
Raphaël Vinot 623e876f3b Cleanup. 
* Remove useless subscriber
* Fix typo in the config file
* Update Helper accordingly
 2014-08-26 17:36:57 +02:00
Raphaël Vinot d7a7da4814 Make sure the webserver fails properly if there is no config file. 
Create the queue list in a more pythonesque fashion.
 2014-08-26 17:33:28 +02:00
Alexandre Dulaunoy 910e0be9be Merge pull request #23 from adulau/master 
ZMQ Publisher removed
 2014-08-26 16:12:44 +02:00
Alexandre Dulaunoy 3b499a2ec8 ZMQ Publisher removed 
ZMQ Publisher removed to allow concurrent use of the scripts.
In short term, we would replace all publishing part within AIL
into pub-sub Redis to avoid ZMQ limitation.
 2014-08-26 14:38:49 +02:00
Alexandre Dulaunoy 264da48768 Merge pull request #22 from adulau/master 
Cleanup: removed unused parameter
 2014-08-26 09:29:21 +02:00
Alexandre Dulaunoy 1567e452f5 Cleanup: removed unused parameter 2014-08-26 09:25:28 +02:00
Starow 55999d3058 Readme updated 2014-08-25 15:02:53 +02:00
Starow 80bf7a4762 Add missing dependencies 2014-08-25 14:44:40 +02:00
Raphaël Vinot b5bf4583e3 automatically copy the config file if there is no. 2014-08-25 14:12:06 +02:00
Jules 93006243c9 Merge pull request #21 from adulau/master 
cymruwhois uses dotted decimal format
 2014-08-25 10:09:00 +02:00
Alexandre Dulaunoy f070ac2005 cymruwhois uses dotted decimal format 2014-08-25 10:05:36 +02:00
Jules 87a57e1f05 Merge pull request #20 from CIRCL/testing 
Big refactoring, make the workflow much easyer to follow.
 2014-08-22 18:36:15 +02:00
Raphaël Vinot 7934c7c8f3 Update deps and install script to work out of the box. 2014-08-22 17:39:58 +02:00
Raphaël Vinot de207e4bb3 Disable the sockets and the pid files from the redis config files. 2014-08-22 17:39:13 +02:00
Raphaël Vinot 3886d1b834 Small fixes to make the refactoring production ready 
* the port for the logging is 6380
* use os.environ properly
* fix typos
 2014-08-22 17:35:40 +02:00
Raphaël Vinot 4f2575723a Update gitignore to have a clean dev directory 2014-08-22 17:34:13 +02:00
Raphaël Vinot 78125db4ea Use env variables everywhere 2014-08-22 14:52:02 +02:00
Raphaël Vinot 277d138a5d cleanup, add FIXME 2014-08-21 14:39:17 +02:00
Raphaël Vinot 63b29176c1 move Redis_Data_Merging to Paste 2014-08-21 12:22:07 +02:00
Raphaël Vinot 50cfac857e Update config 
Make all paths in the config file relative to the home directory.
 2014-08-20 16:00:56 +02:00
Raphaël Vinot a68f5b6a0e fix subscriber names, update default config 2014-08-20 15:54:21 +02:00
Alexandre Dulaunoy 17a9185404 Merge pull request #19 from adulau/master 
Redis explanation added
 2014-08-20 15:35:16 +02:00
Alexandre Dulaunoy 5096230f05 Markdown fixed 2014-08-20 15:33:26 +02:00
Alexandre Dulaunoy a030f939d9 Overview of Redis database added 2014-08-20 15:31:10 +02:00
Raphaël Vinot 2485ba5df2 Merge remote-tracking branch 'origin/master' into testing 
Conflicts:
	bin/ZMQ_Sub_Urls.py
 2014-08-20 15:24:10 +02:00
Raphaël Vinot cf814468e7 Merge branch 'rewrite_zmq' into testing 
Conflicts:
	bin/indexer_lookup.py
 2014-08-20 15:18:02 +02:00
Raphaël Vinot 99c8cc7941 completely remove ZMQ_PubSub.py 2014-08-20 15:14:57 +02:00
Alexandre Dulaunoy a84ffc9ef7 Merge pull request #18 from adulau/master 
Fix one assignment before use and remove duplicate call to libmagic
 2014-08-20 10:25:57 +02:00
Alexandre Dulaunoy 1d64dc44c8 MIME type guessing - removed one duplicate call to libmagic 2014-08-20 10:22:33 +02:00
Raphaël Vinot 8d9ffbaa53 Do not create a ZMQ sub if it is not required. 2014-08-19 19:53:33 +02:00
Raphaël Vinot 45b0bf3983 Improve the cleanup. Still some to do. 2014-08-19 19:07:07 +02:00
Raphaël Vinot f1753d67c6 Cleanup the queues. 2014-08-19 16:05:37 +02:00
Alexandre Dulaunoy e8fcea6cd6 Remove undeclared variable 2014-08-18 16:17:36 +02:00
Alexandre Dulaunoy 7d8ee102a3 Assignment before use (if Enumerate fails) 2014-08-18 15:58:06 +02:00
Raphaël Vinot ebef13fa5b Merge branch 'master' of github.com:CIRCL/AIL-framework 2014-08-18 13:40:22 +02:00
Raphaël Vinot 3f066ddaec fix install script 2014-08-18 13:40:07 +02:00
Alexandre Dulaunoy e01ac78538 Merge pull request #15 from adulau/master 
Configuration path fixed
 2014-08-18 13:23:21 +02:00
Alexandre Dulaunoy 4304c6858e Configuration path fixed 2014-08-18 09:02:08 +02:00
Raphaël Vinot 078c8ea836 Big cleanup, pep8 2014-08-14 18:07:18 +02:00
Jules 715adb4546 Merge pull request #14 from adulau/master 
Requirements updated for CC/ASN lookup in URL module
 2014-08-14 15:07:40 +02:00
Alexandre Dulaunoy 0d23a3952c Requirements updated for CC/ASN lookup in URL module 2014-08-14 14:56:33 +02:00
Jules ab6765315e Merge pull request #13 from adulau/master 
Log where URLs are hosted - cc_critical option added
 2014-08-14 14:28:01 +02:00
Alexandre Dulaunoy 762def3a23 Log where URLs are hosted - cc_critical option added 
It logs where the hostname of the URL is hosted (ASN and geographic location).
A simple option cc_critical added to set the country code to log as critical.
 2014-08-14 14:22:11 +02:00
Raphaël Vinot 4a1f300a1a Cleanup (remove unused imports, more pep8 compatible) 2014-08-14 14:11:07 +02:00
Starow 04a8f1bdf2 maxi cleanup old code :'( 2014-08-14 11:48:46 +02:00