mirror of https://github.com/CIRCL/AIL-framework
144 lines
4.7 KiB
Markdown
144 lines
4.7 KiB
Markdown
|
|
Feeding, adding new features and contributing
|
|
=============================================
|
|
|
|
How to feed the AIL framework
|
|
-----------------------------
|
|
|
|
For the moment, there are three different ways to feed AIL with data:
|
|
|
|
1. Be a collaborator of CIRCL and ask to access our feed. It will be sent to the static IP you are using for AIL.
|
|
|
|
2. You can setup [pystemon](https://github.com/cvandeplas/pystemon) and use the custom feeder provided by AIL (see below).
|
|
|
|
3. You can feed your own data using the [./bin/file_dir_importer.py](./bin/import_dir.py) script.
|
|
|
|
### Feeding AIL with pystemon
|
|
|
|
AIL is an analysis tool, not a collector!
|
|
However, if you want to collect some pastes and feed them to AIL, the procedure is described below. Nevertheless, moderate your queries!
|
|
|
|
Feed data to AIL:
|
|
|
|
1. Clone the [pystemon's git repository](https://github.com/cvandeplas/pystemon):
|
|
``` git clone https://github.com/cvandeplas/pystemon.git ```
|
|
|
|
2. Edit configuration file for pystemon ```pystemon/pystemon.yaml```:
|
|
* Configuration of storage section (adapt to your needs):
|
|
```
|
|
storage:
|
|
archive:
|
|
storage-classname: FileStorage
|
|
save: yes
|
|
save-all: yes
|
|
dir: "alerts"
|
|
dir-all: "archive"
|
|
compress: yes
|
|
|
|
redis:
|
|
storage-classname: RedisStorage
|
|
save: yes
|
|
save-all: yes
|
|
server: "localhost"
|
|
port: 6379
|
|
database: 10
|
|
lookup: no
|
|
```
|
|
* Change configuration for paste-sites according to your needs (don't forget to throttle download time and/or update time).
|
|
3. Install python dependencies inside the virtual environment:
|
|
```
|
|
cd ail-framework/
|
|
. ./AILENV/bin/activate
|
|
cd pystemon/ #cd to pystemon folder
|
|
pip3 install -U -r requirements.txt
|
|
```
|
|
4. Edit configuration file ```ail-framework/configs/core.cfg```:
|
|
* Modify the "pystemonpath" path accordingly
|
|
|
|
5. Launch ail-framework, pystemon and pystemon-feeder.py (still inside virtual environment):
|
|
* Option 1 (recommended):
|
|
```
|
|
./ail-framework/bin/LAUNCH.py -l #starts ail-framework
|
|
./ail-framework/bin/LAUNCH.py -f #starts pystemon and the pystemon-feeder.py
|
|
```
|
|
* Option 2 (you may need two terminal windows):
|
|
```
|
|
./ail-framework/bin/LAUNCH.py -l #starts ail-framework
|
|
./pystemon/pystemon.py
|
|
./ail-framework/bin/feeder/pystemon-feeder.py
|
|
```
|
|
|
|
How to create a new module
|
|
--------------------------
|
|
|
|
If you want to add a new processing or analysis module in AIL, follow these simple steps:
|
|
|
|
1. Add your module name in [./bin/packages/modules.cfg](./bin/packages/modules.cfg) and subscribe to at least one module at minimum (Usually, Redis_Global).
|
|
|
|
2. Use [./bin/template.py](./bin/template.py) as a sample module and create a new file in bin/ with the module name used in the modules.cfg configuration.
|
|
|
|
|
|
How to contribute a module
|
|
--------------------------
|
|
|
|
Feel free to fork the code, play with it, make some patches or add additional analysis modules.
|
|
|
|
To contribute your module, feel free to pull your contribution.
|
|
|
|
|
|
Additional information
|
|
======================
|
|
|
|
Crawler
|
|
---------------------
|
|
|
|
In AIL, you can crawl websites and Tor hidden services. Don't forget to review the proxy configuration of your Tor client and especially if you enabled the SOCKS5 proxy
|
|
|
|
[//]: # (and binding on the appropriate IP address reachable via the dockers where Splash runs.)
|
|
|
|
### Installation
|
|
|
|
|
|
[Install Lacus](https://github.com/ail-project/lacus)
|
|
|
|
### Configuration
|
|
|
|
1. Lacus URL:
|
|
In the webinterface, go to ``Crawlers>Settings`` and click on the Edit button
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2. Launch AIL Crawlers:
|
|
Choose the number of crawlers you want to launch
|
|
|
|
|
|
|
|
|
|
|
|
Kvrocks Migration
|
|
---------------------
|
|
**Important Note:
|
|
We are currently working on a [migration script](https://github.com/ail-project/ail-framework/blob/master/bin/DB_KVROCKS_MIGRATION.py) to facilitate the migration to Kvrocks.
|
|
Once this script is ready, AIL version 5.0 will be released.**
|
|
|
|
Please note that the current version of this migration script only supports migrating the database on the same server.
|
|
(If you plan to migrate to another server, we will provide additional instructions in this section once the migration script is completed)
|
|
|
|
To migrate your database to Kvrocks:
|
|
1. Launch ARDB and Kvrocks
|
|
2. Pull from remote
|
|
```
|
|
git checkout master
|
|
git pull
|
|
```
|
|
3. Launch the migration script:
|
|
```
|
|
git checkout master
|
|
git pull
|
|
cd bin/
|
|
./DB_KVROCKS_MIGRATION.py
|
|
```
|