Analysis Information Leak framework
 
 
 
 
 
Go to file
Terrtia 8b1c10b38c
chg: [Onion] add onion splash crawler
2018-08-09 17:42:21 +02:00
ansible Update README.md 2018-07-23 19:10:05 +02:00
bin chg: [Onion] add onion splash crawler 2018-08-09 17:42:21 +02:00
configs fix: [LAUNCH] use envar, add options (launch all, flask, killall) 2018-07-24 10:41:10 +02:00
doc new: Added pass the SALT workshop slides 2018-07-04 10:18:18 +02:00
docsphinx/source Initial import of AIL framework - Analysis Information Leak framework 2014-08-06 11:43:40 +02:00
etc/splash/proxy-profiles chg: [Onion] add onion splash crawler 2018-08-09 17:42:21 +02:00
files add apiKeys module 2018-04-26 14:42:39 +02:00
logs Travis, print logs 2016-01-19 12:01:45 +01:00
samples/2018/01/01 create first test 2018-04-11 10:14:33 +02:00
tests clean 2018-05-04 14:25:47 +02:00
var/www fix: [UI-Showpaste] show raw paste, force text/plain mimetype 2018-08-03 09:15:34 +02:00
.dockerignore Added Dockerfile to automate the build of a Docker image based on Ubuntu 2016-08-27 10:29:20 +02:00
.gitignore fix hive connection + submitted dir creation 2018-06-19 11:31:30 +02:00
.travis.yml fix travis tests 2 2018-04-26 17:06:19 +02:00
Dockerfile Fixes encoding issue with MISP Galaxies JSON files 2018-06-22 16:58:13 -04:00
HOWTO.md Added ore info on new webpage creation 2017-07-17 13:26:10 +02:00
LICENSE Initial import of AIL framework - Analysis Information Leak framework 2014-08-06 11:43:40 +02:00
OVERVIEW.md fix: LevelDB not more used 2018-06-20 10:48:13 +02:00
README.md README.md: Add note regarding Direct I/O 2018-07-18 13:44:48 +02:00
docker_start.sh chg: [doc] Marked unmaintained efforts as such. 2018-07-23 18:27:23 +02:00
installing_deps.sh [fix] switch from git:// to https:// while cloning tlsh.git 2018-07-06 11:15:02 +01:00
installing_deps_archlinux.sh chg: [doc] Marked unmaintained efforts as such. 2018-07-23 18:27:23 +02:00
pip3_packages_requirement.txt add libinjection bindings to pip requirements 2018-07-04 14:28:34 +02:00
python3_upgrade.sh python 3 upgrade mv old redis db 2018-05-09 13:46:36 +02:00
reset_AIL.sh Added AIL reset script 2017-08-23 15:05:51 +02:00

README.md

Build Status

AIL

Logo

AIL framework - Framework for Analysis of Information Leaks

AIL is a modular framework to analyse potential information leaks from unstructured data sources like pastes from Pastebin or similar services or unstructured data streams. AIL framework is flexible and can be extended to support other functionalities to mine or process sensitive information (e.g. data leak prevention).

Dashboard

Features

  • Modular architecture to handle streams of unstructured or structured information
  • Default support for external ZMQ feeds, such as provided by CIRCL or other providers
  • Multiple feed support
  • Each module can process and reprocess the information already processed by AIL
  • Detecting and extracting URLs including their geographical location (e.g. IP address location)
  • Extracting and validating potential leak of credit cards numbers, credentials, ...
  • Extracting and validating email addresses leaked including DNS MX validation
  • Module for extracting Tor .onion addresses (to be further processed for analysis)
  • Keep tracks of duplicates (and diffing between each duplicate found)
  • Extracting and validating potential hostnames (e.g. to feed Passive DNS systems)
  • A full-text indexer module to index unstructured information
  • Statistics on modules and web
  • Real-time modules manager in terminal
  • Global sentiment analysis for each providers based on nltk vader module
  • Terms, Set of terms and Regex tracking and occurrence
  • Many more modules for extracting phone numbers, credentials and others
  • Alerting to MISP to share found leaks within a threat intelligence platform using MISP standard
  • Detect and decode Base64 and store files
  • Detect Amazon AWS and Google API keys
  • Detect Bitcoin address and Bitcoin private keys
  • Detect private keys and certificate
  • Tagging system with MISP Galaxy and MISP Taxonomies tags
  • UI paste submission
  • Create events on MISP and cases on The Hive
  • Automatic paste export at detection on MISP (events) and The Hive (alerts) on selected tags

Installation

Type these command lines for a fully automated installation and start AIL framework:

git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
./installing_deps.sh
cd var/www/
./update_thirdparty.sh
cd ~/AIL-framework/
. ./AILENV/bin/activate
cd bin/
./LAUNCH.sh

The default installing_deps.sh is for Debian and Ubuntu based distributions. For Arch linux based distributions, you can replace it with installing_deps_archlinux.sh.

There is also a Travis file used for automating the installation that can be used to build and install AIL on other systems.

Installation Notes

In order to use AIL combined with ZFS or unprivileged LXC it's necessary to disable Direct I/O in $AIL_HOME/configs/6382.conf by changing the value of the directive use_direct_io_for_flush_and_compaction to false.

Python 3 Upgrade

To upgrade from an existing AIL installation, you have to launch python3_upgrade.sh, this script will delete and create a new virtual environment. The script will upgrade the packages but won't keep your previous data (neverthless the data is copied into a directory called old). If you install from scratch, you don't require to launch the python3_upgrade.sh.

Docker Quick Start (Ubuntu 16.04 LTS)

  1. Install Docker
sudo su
apt-get install -y curl
curl https://get.docker.com | /bin/bash
  1. Type these commands to build the Docker image:
git clone https://github.com/CIRCL/ail-framework
cd AIL-framework
docker build -t ail-framework .
  1. To start AIL on port 7000, type the following command below:
docker run -p 7000:7000 ail-framework
  1. To debug the running container, type the following command and note the container name or identifier:
docker ps

After getting the name or identifier type the following commands:

docker exec -it CONTAINER_NAME_OR_IDENTIFIER bash
cd /opt/ail

Install using Ansible

Please check the Ansible readme.

Starting AIL web interface

To start the web interface, you first need to fetch the required JavaScript/CSS files:

cd $AILENV
cd var/www/
bash update_thirdparty.sh

and then you can start the web interface python script:

cd $AILENV
cd var/www/
./Flask_server.py

Eventually you can browse the status of the AIL framework website at the following URL:

http://localhost:7000/

HOWTO

HOWTO are available in HOWTO.md

Privacy and GDPR

AIL information leaks analysis and the GDPR in the context of collection, analysis and sharing information leaks document provides an overview how to use AIL in a lawfulness context especially in the scope of General Data Protection Regulation.

Screenshots

Trending-Web Trending-Modules

Browsing

Browse-Pastes

Tagging system

Tags

MISP and The Hive, automatic events and alerts creation

paste_submit

Paste submission

paste_submit

Sentiment analysis

Sentiment

Terms manager and occurence

Term-Manager

Top terms

Term-Top Term-Plot

AIL framework screencast

Command line module manager

Module-Manager

License

    Copyright (C) 2014 Jules Debra
    Copyright (C) 2014-2018 CIRCL - Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d'Intérêt Economique)
    Copyright (c) 2014-2018 Raphaël Vinot
    Copyright (c) 2014-2018 Alexandre Dulaunoy
    Copyright (c) 2016-2018 Sami Mokaddem
    Copyright (c) 2018 Thirion Aurélien

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU Affero General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU Affero General Public License for more details.

    You should have received a copy of the GNU Affero General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.