5.9 KiB

Raw Blame History

Feeding, adding new features and contributing

How to feed the AIL framework

For the moment, there are three different ways to feed AIL with data:

Be a collaborator of CIRCL and ask to access our feed. It will be sent to the static IP you are using for AIL.
You can setup pystemon and use the custom feeder provided by AIL (see below).
You can feed your own data using the ./bin/import_dir.py script.

Feeding AIL with pystemon

AIL is an analysis tool, not a collector! However, if you want to collect some pastes and feed them to AIL, the procedure is described below. Nevertheless, moderate your queries!

Feed data to AIL:

Clone the pystemon's git repository
Install its python dependencies inside your virtual environment
Launch pystemon ./pystemon
Edit your configuration file bin/packages/config.cfg and modify the pystemonpath path accordingly
Launch pystemon-feeder ./pystemon-feeder.py

How to create a new module

If you want to add a new processing or analysis module in AIL, follow these simple steps:

Add your module name in ./bin/packages/modules.cfg and subscribe to at least one module at minimum (Usually, Redis_Global).
Use ./bin/template.py as a sample module and create a new file in bin/ with the module name used in the modules.cfg configuration.

How to create a new webpage

If you want to add a new webpage for a module in AIL, follow these simple steps:

Launch ./var/www/create_new_web_module.py and enter the name to use for your webpage (Usually, your newly created python module).
A template and flask skeleton has been created for your new webpage in ./var/www/modules/
Edit the created html files under the template folder as well as the Flask_* python script so that they fit your needs.
You can change the order of your module in the top navigation header in the file ./var/www/templates/header_base.html
You can ignore module, and so, not display them in the top navigation header by adding the module name in the file ./var/www/templates/ignored_modules.txt

How to contribute a module

Feel free to fork the code, play with it, make some patches or add additional analysis modules.

To contribute your module, feel free to pull your contribution.

Additional information

Manage modules: ModulesInformationV2.py

You can do a lots of things easily with the ./bin/ModulesInformationV2 script:

Monitor the health of other modules
Monitor the ressources comsumption of other modules
Start one or more modules
Kill running modules
Restart automatically stuck modules
Show the paste currently processed by a module

You can navigate into the interface by using arrow keys. In order to perform an action on a selected module, you can either press or to show the dialog box.

To change list, you can press the key.

Also, you can quickly stop or start modules by clicking on the or ~~symbol respectively. These are located in the Action column.~~

Finally, you can quit this program by pressing either or

Terms frequency usage

In AIL, you can track terms, set of terms and even regexes without creating a dedicated module. To do so, go to the tab Terms Frequency in the web interface.

You can track a term by simply putting it in the box.

You can track a set of terms by simply putting terms in an array surrounded by the '' character. You can also set a custom threshold regarding the number of terms that must match to trigger the detection. For example, if you want to track the terms term1 and term2 at the same time, you can use the following rule: \[term1, term2, [100]]\

You can track regexes as easily as tracking a term. You just have to put your regex in the box surrounded by the '/' character. For example, if you want to track the regex matching all email address having the domain domain.net, you can use the following aggressive rule: /*.domain.net/.

Crawler

In AIL, you can crawl hidden services.

two types of configutation [explaination for what]: 1) use local Splash dockers (use the same host for Splash servers and AIL) 2) use remote Splash servers

(Splash host) Launch crawler_hidden_services_install.sh to install all requirement (type y if a localhost splah server is used)

(Splash host) Setup your tor proxy[is already installed]:

Add the following line in /etc/tor/torrc: SOCKSPolicy accept 172.17.0.0/16 (for a linux docker, the localhost IP is 172.17.0.1; Should be adapted for other platform)

Restart the tor proxy: sudo service tor restart

(Splash host) Launch all Splash servers with: sudo ./bin/torcrawler/launch_splash_crawler.sh [-f <config absolute_path>] [-p <port_start>] [-n <number_of_splash>] all the Splash dockers are launched inside the Docker_Splash screen. You can use sudo screen -r Docker_Splash to connect to the screen session and check all Splash servers status.

(AIL host) Edit the /bin/packages/config.cfg file:

In the crawler section, set activate_crawler to True

Change the IP address of Splash servers if needed (remote only)

Set splash_onion_port according to your Splash servers port numbers who are using the tor proxy. those ports numbers should be described as a single port (ex: 8050) or a port range (ex: 8050-8052 for 8050,8051,8052 ports).

(AIL host) launch all AIL crawler scripts using: ./bin/LAUNCH.sh -c

5.9 KiB Raw Blame History