Analysis Information Leak framework
 
 
 
 
 
Go to file
Sami Mokaddem ec7d2d49b2 Added seconds args in import_dir 2017-11-15 16:03:42 +01:00
bin Added seconds args in import_dir 2017-11-15 16:03:42 +01:00
configs Added config redis 6382 2016-08-23 16:27:50 +02:00
doc Merge branch 'bugfix' 2017-08-21 14:08:14 +02:00
docsphinx/source Initial import of AIL framework - Analysis Information Leak framework 2014-08-06 11:43:40 +02:00
files Bug fix related with redis: 2016-08-09 11:59:36 +02:00
logs Travis, print logs 2016-01-19 12:01:45 +01:00
var/www Removed useless import 2017-08-23 14:06:59 +02:00
.dockerignore Added Dockerfile to automate the build of a Docker image based on Ubuntu 2016-08-27 10:29:20 +02:00
.gitignore Deleted data-flow graph and header.html from the index + Added fixed typo where 10min was in fact 3min 2017-08-21 10:51:56 +02:00
.travis.yml Update travis 2016-07-25 11:38:41 +02:00
Dockerfile Added Dockerfile to automate the build of a Docker image based on Ubuntu 2016-08-27 10:29:20 +02:00
HOWTO.md Added ore info on new webpage creation 2017-07-17 13:26:10 +02:00
LICENSE Initial import of AIL framework - Analysis Information Leak framework 2014-08-06 11:43:40 +02:00
OVERVIEW.md Improved overview 2017-05-03 14:42:37 +02:00
README.md Added link towards HOWTO in README.md 2017-07-17 13:36:41 +02:00
docker_start.sh Updated docker_start dataset year + Mixer 2017-03-01 08:34:35 +01:00
installing_deps.sh Removed 301x Dir in install_deps 2017-08-23 14:00:04 +02:00
installing_deps_archlinux.sh Updated archlinux instaler 2017-02-28 10:50:00 +01:00
pip_packages_requirement.txt Restored adns dependency 2017-08-21 13:23:29 +02:00
reset_AIL.sh Added AIL reset script 2017-08-23 15:05:51 +02:00

README.md

Build Status

AIL

Logo

AIL framework - Framework for Analysis of Information Leaks

AIL is a modular framework to analyse potential information leaks from unstructured data sources like pastes from Pastebin or similar services or unstructured data streams. AIL framework is flexible and can be extended to support other functionalities to mine sensitive information.

Dashboard

Features

  • Modular architecture to handle streams of unstructured or structured information
  • Default support for external ZMQ feeds, such as provided by CIRCL or other providers
  • Multiple feed support
  • Each module can process and reprocess the information already processed by AIL
  • Detecting and extracting URLs including their geographical location (e.g. IP address location)
  • Extracting and validating potential leak of credit cards numbers, credentials, ...
  • Extracting and validating email addresses leaked including DNS MX validation
  • Module for extracting Tor .onion addresses (to be further processed for analysis)
  • Keep tracks of duplicates
  • Extracting and validating potential hostnames (e.g. to feed Passive DNS systems)
  • A full-text indexer module to index unstructured information
  • Statistics on modules and web
  • Realtime modules manager in terminal
  • Global sentiment analysis for each providers based on nltk vader module
  • Terms, Set of terms and Regex tracking and occurrence
  • Many more modules for extracting phone numbers, credentials and others

Installation

Type these command lines for a fully automated installation and start AIL framework

git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
./installing_deps.sh
cd var/www/
./update_thirdparty.sh
cd ~/AIL-framework/
. ./AILENV/bin/activate
cd bin/
./LAUNCH.sh

The default installing_deps.sh is for Debian and Ubuntu based distributions. For Arch linux based distributions, you can replace it with installing_deps_archlinux.sh.

There is also a Travis file used for automating the installation that can be used to build and install AIL on other systems.

Starting AIL web interface

To start the web interface, you first need to fetch the required Javascript/CSS files:

cd $AILENV
cd var/www/
bash update_thirdparty.sh

and then you can start the web interface python script:

cd $AILENV
cd var/www/
Flask_server.py

Eventually you can browse the status of the AIL framework website at the following URL:

    ``http://localhost:7000/``

HOWTO

HOWTO are available in HOWTO.md

Screenshots

Trending-Web Trending-Modules

Browsing

Browse-Pastes

Sentiment analysis

Sentiment

Terms manager and occurence

Term-Manager

Top terms

Term-Top Term-Plot

AIL framework screencast

Command line module manager

Module-Manager

License

    Copyright (C) 2014 Jules Debra
    Copyright (C) 2014-2016 CIRCL - Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d'Intérêt Economique)
    Copyright (c) 2014-2016 Raphaël Vinot
    Copyright (c) 2014-2016 Alexandre Dulaunoy
    Copyright (c) 2016 Sami Mokaddem

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU Affero General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU Affero General Public License for more details.

    You should have received a copy of the GNU Affero General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.