Analysis Information Leak framework
 
 
 
 
 
Go to file
Terrtia 046984dec1
chg: [update Readme logo + links]
2020-04-20 15:59:55 +02:00
bin crawler_time -> crawler_delta 2020-04-17 20:06:40 +05:30
configs merge master => import_export_v1 2020-02-19 09:35:00 +01:00
doc chg: [doc] add cookiejar screenshot 2020-04-01 10:27:33 +02:00
docsphinx/source Initial import of AIL framework - Analysis Information Leak framework 2014-08-06 11:43:40 +02:00
etc/splash/proxy-profiles chg: [Onion] add onion splash crawler 2018-08-09 17:42:21 +02:00
files chg: [Crawler] change BDD, save i2p links 2018-08-21 15:54:53 +02:00
logs Travis, print logs 2016-01-19 12:01:45 +01:00
other_installers fix: [ansible docker] move files, fix 365 2019-07-22 14:16:47 +02:00
samples/2018/01/01 create first test 2018-04-11 10:14:33 +02:00
tests chg: [test api] increase import timeout 2020-01-21 14:44:51 +01:00
update chg: [Updater] relauch updater on change (git pull) 2020-02-26 10:19:36 +01:00
var/www chg: [add new logo] 2020-04-20 12:03:09 +02:00
.dockerignore Added Dockerfile to automate the build of a Docker image based on Ubuntu 2016-08-27 10:29:20 +02:00
.gitignore chg: [Update v2.4] cp config 2019-11-05 16:44:10 +01:00
.travis.yml fix: [travis] LAUNCH.sh needs -l to launch... 2019-05-01 22:03:32 +09:00
HOWTO.md chg: [core] mv bin/packages/config.cfg configs/core.cfg + use ConfigLoader 2019-11-05 15:18:03 +01:00
LICENSE Initial import of AIL framework - Analysis Information Leak framework 2014-08-06 11:43:40 +02:00
OVERVIEW.md chg [telegram + correlation] new module: telegram (username + login code + join_chat) + add simple_correlation backend 2020-04-10 10:49:21 +02:00
README.md chg: [update Readme logo + links] 2020-04-20 15:59:55 +02:00
crawler_hidden_services_install.sh chg: [Crawler] add docs 2018-09-27 11:14:29 +02:00
crawler_requirements.txt chg: [Crawler] add launcher and install 2018-09-24 16:23:14 +02:00
install_virtualenv.sh chg: [AIL logo UI] 2020-04-20 11:56:59 +02:00
installing_deps.sh chg: [AIL logo UI] 2020-04-20 11:56:59 +02:00
python3_upgrade.sh python 3 upgrade mv old redis db 2018-05-09 13:46:36 +02:00
requirements.txt Fix pybgpranking dependency in requirements 2019-10-29 00:38:33 +01:00
reset_AIL.sh chg: [reset_AIL] add echo message 2019-07-30 16:21:39 +02:00

README.md

AIL

Logo

Latest Release
Travis
Gitter
Contributors
License

Logo

AIL framework - Framework for Analysis of Information Leaks

AIL is a modular framework to analyse potential information leaks from unstructured data sources like pastes from Pastebin or similar services or unstructured data streams. AIL framework is flexible and can be extended to support other functionalities to mine or process sensitive information (e.g. data leak prevention).

Dashboard

Finding webshells with AIL

Features

  • Modular architecture to handle streams of unstructured or structured information
  • Default support for external ZMQ feeds, such as provided by CIRCL or other providers
  • Multiple feed support
  • Each module can process and reprocess the information already processed by AIL
  • Detecting and extracting URLs including their geographical location (e.g. IP address location)
  • Extracting and validating potential leaks of credit card numbers, credentials, ...
  • Extracting and validating leaked email addresses, including DNS MX validation
  • Module for extracting Tor .onion addresses (to be further processed for analysis)
  • Keep tracks of duplicates (and diffing between each duplicate found)
  • Extracting and validating potential hostnames (e.g. to feed Passive DNS systems)
  • A full-text indexer module to index unstructured information
  • Statistics on modules and web
  • Real-time modules manager in terminal
  • Global sentiment analysis for each providers based on nltk vader module
  • Terms, Set of terms and Regex tracking and occurrence
  • Many more modules for extracting phone numbers, credentials and others
  • Alerting to MISP to share found leaks within a threat intelligence platform using MISP standard
  • Detect and decode encoded file (Base64, hex encoded or your own decoding scheme) and store files
  • Detect Amazon AWS and Google API keys
  • Detect Bitcoin address and Bitcoin private keys
  • Detect private keys, certificate, keys (including SSH, OpenVPN)
  • Detect IBAN bank accounts
  • Tagging system with MISP Galaxy and MISP Taxonomies tags
  • UI paste submission
  • Create events on MISP and cases on The Hive
  • Automatic paste export at detection on MISP (events) and The Hive (alerts) on selected tags
  • Extracted and decoded files can be searched by date range, type of file (mime-type) and encoding discovered
  • Graph relationships between decoded file (hashes), similar PGP UIDs and addresses of cryptocurrencies
  • Tor hidden services crawler to crawl and parse output
  • Tor onion availability is monitored to detect up and down of hidden services
  • Browser hidden services are screenshot and integrated in the analysed output including a blurring screenshot interface (to avoid "burning the eyes" of the security analysis with specific content)
  • Tor hidden services is part of the standard framework, all the AIL modules are available to the crawled hidden services
  • Generic web crawler to trigger crawling on demand or at regular interval URL or Tor hidden services

Installation

Type these command lines for a fully automated installation and start AIL framework:

git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
./installing_deps.sh

cd ~/AIL-framework/
cd bin/
./LAUNCH.sh -l

The default installing_deps.sh is for Debian and Ubuntu based distributions.

There is also a Travis file used for automating the installation that can be used to build and install AIL on other systems.

Requirement:

  • Python 3.6+

Installation Notes

In order to use AIL combined with ZFS or unprivileged LXC it's necessary to disable Direct I/O in $AIL_HOME/configs/6382.conf by changing the value of the directive use_direct_io_for_flush_and_compaction to false.

Tor installation instructions can be found in the HOWTO

Starting AIL

cd bin/
./LAUNCH.sh -l

Eventually you can browse the status of the AIL framework website at the following URL:

https://localhost:7000/

The default credentials for the web interface are located in DEFAULT_PASSWORD. This file is removed when you change your password.

Training

CIRCL organises training on how to use or extend the AIL framework. AIL training materials are available at https://www.circl.lu/services/ail-training-materials/.

API

The API documentation is available in doc/README.md

HOWTO

HOWTO are available in HOWTO.md

Privacy and GDPR

AIL information leaks analysis and the GDPR in the context of collection, analysis and sharing information leaks document provides an overview how to use AIL in a lawfulness context especially in the scope of General Data Protection Regulation.

Research using AIL

If you write academic paper, relying or using AIL, it can be cited with the following BibTeX:

@inproceedings{mokaddem2018ail,
  title={AIL-The design and implementation of an Analysis Information Leak framework},
  author={Mokaddem, Sami and Wagener, G{\'e}rard and Dulaunoy, Alexandre},
  booktitle={2018 IEEE International Conference on Big Data (Big Data)},
  pages={5049--5057},
  year={2018},
  organization={IEEE}
}

Screenshots

Tor hidden service crawler

Tor hidden service

Trending-Modules

Extracted encoded files from pastes

Extracted files from pastes Relationships between extracted files from encoded file in unstructured data

Browsing

Browse-Pastes

Tagging system

Tags

MISP and The Hive, automatic events and alerts creation

paste_submit

Paste submission

paste_submit

Sentiment analysis

Sentiment

Terms tracker

Term-tracker

AIL framework screencast

Command line module manager

Module-Manager

License

    Copyright (C) 2014 Jules Debra
    Copyright (C) 2014-2019 CIRCL - Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d'Intérêt Economique)
    Copyright (c) 2014-2019 Raphaël Vinot
    Copyright (c) 2014-2019 Alexandre Dulaunoy
    Copyright (c) 2016-2019 Sami Mokaddem
    Copyright (c) 2018-2019 Thirion Aurélien

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU Affero General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU Affero General Public License for more details.

    You should have received a copy of the GNU Affero General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.