CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
AIL-framework/bin/Shutdown.py

69 lines
2.1 KiB
Python
Raw Normal View History

decode with redis connection
2018-05-04 13:53:29 +02:00
#!/usr/bin/env python3
Initial import of AIL framework - Analysis Information Leak framework AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen sitive information
2014-08-06 11:43:40 +02:00
# -*-coding:UTF-8 -*
"""
The ZMQ_Feed_Q Module
=====================
This module is consuming the Redis-list created by the ZMQ_Feed_Q Module,
And save the paste on disk to allow others modules to work on them.
..todo:: Be able to choose to delete or not the saved paste after processing.
..todo:: Store the empty paste (unprocessed) somewhere in Redis.
..note:: Module ZMQ_Something_Q and ZMQ_Something are closely bound, always put
the same Subscriber name in both of them.
Requirements
------------
*Need running Redis instances.
*Need the ZMQ_Feed_Q Module running to be able to work properly.
"""
Cleanup (remove unused imports, more pep8 compatible)
2014-08-14 14:11:07 +02:00
import redis
python 3 backend upgrade
2018-04-16 14:50:04 +02:00
import configparser
Use env variables everywhere
2014-08-22 14:52:02 +02:00
import os
Initial import of AIL framework - Analysis Information Leak framework AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen sitive information
2014-08-06 11:43:40 +02:00
Small fixes to make the refactoring production ready * the port for the logging is 6380 * use os.environ properly * fix typos
2014-08-22 17:35:40 +02:00
configfile = os.path.join(os.environ['AIL_BIN'], './packages/config.cfg')
Initial import of AIL framework - Analysis Information Leak framework AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen sitive information
2014-08-06 11:43:40 +02:00
Cleanup (remove unused imports, more pep8 compatible)
2014-08-14 14:11:07 +02:00
Initial import of AIL framework - Analysis Information Leak framework AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen sitive information
2014-08-06 11:43:40 +02:00
def main():
"""Main Function"""
# CONFIG #
python 3 backend upgrade
2018-04-16 14:50:04 +02:00
cfg = configparser.ConfigParser()
Initial import of AIL framework - Analysis Information Leak framework AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen sitive information
2014-08-06 11:43:40 +02:00
cfg.read(configfile)
Cleanup (remove unused imports, more pep8 compatible)
2014-08-14 14:11:07 +02:00
# REDIS
r_serv = redis.StrictRedis(host=cfg.get("Redis_Queues", "host"),
port=cfg.getint("Redis_Queues", "port"),
decode with redis connection
2018-05-04 13:53:29 +02:00
db=cfg.getint("Redis_Queues", "db"),
decode_responses=True)
Initial import of AIL framework - Analysis Information Leak framework AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen sitive information
2014-08-06 11:43:40 +02:00
Do not create a ZMQ sub if it is not required.
2014-08-19 19:53:33 +02:00
# FIXME: automatic based on the queue name.
Cleanup (remove unused imports, more pep8 compatible)
2014-08-14 14:11:07 +02:00
# ### SCRIPTS ####
Initial import of AIL framework - Analysis Information Leak framework AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen sitive information
2014-08-06 11:43:40 +02:00
r_serv.sadd("SHUTDOWN_FLAGS", "Feed")
r_serv.sadd("SHUTDOWN_FLAGS", "Categ")
r_serv.sadd("SHUTDOWN_FLAGS", "Lines")
r_serv.sadd("SHUTDOWN_FLAGS", "Tokenize")
r_serv.sadd("SHUTDOWN_FLAGS", "Attributes")
r_serv.sadd("SHUTDOWN_FLAGS", "Creditcards")
r_serv.sadd("SHUTDOWN_FLAGS", "Duplicate")
r_serv.sadd("SHUTDOWN_FLAGS", "Mails")
r_serv.sadd("SHUTDOWN_FLAGS", "Onion")
r_serv.sadd("SHUTDOWN_FLAGS", "Urls")
r_serv.sadd("SHUTDOWN_FLAGS", "Feed_Q")
r_serv.sadd("SHUTDOWN_FLAGS", "Categ_Q")
r_serv.sadd("SHUTDOWN_FLAGS", "Lines_Q")
r_serv.sadd("SHUTDOWN_FLAGS", "Tokenize_Q")
r_serv.sadd("SHUTDOWN_FLAGS", "Attributes_Q")
r_serv.sadd("SHUTDOWN_FLAGS", "Creditcards_Q")
r_serv.sadd("SHUTDOWN_FLAGS", "Duplicate_Q")
r_serv.sadd("SHUTDOWN_FLAGS", "Mails_Q")
r_serv.sadd("SHUTDOWN_FLAGS", "Onion_Q")
r_serv.sadd("SHUTDOWN_FLAGS", "Urls_Q")
if __name__ == "__main__":
main()