diff --git a/HOWTO.md b/HOWTO.md index b7ebd1bf..695da97e 100644 --- a/HOWTO.md +++ b/HOWTO.md @@ -1,73 +1,18 @@ # Feeding, adding new features and contributing +## [Documentation AIL Importers](./doc/README.md#ail-importers) + +[Documentation AIL Importers](./doc/README.md#ail-importers) + ## How to feed the AIL framework -Currently, there are three different ways to feed data into AIL: - -1. Be a collaborator of CIRCL and ask to access our feed. It will be sent to the static IP you are using for AIL. - -2. You can setup [pystemon](https://github.com/cvandeplas/pystemon) and use the custom feeder provided by AIL (see below). - -3. You can feed your own data using the [./tool/file_dir_importer.py](./tool/file_dir_importer.py) script. - -### Feeding AIL with pystemon - AIL is an analysis tool, not a collector! However, if you want to collect some pastes and feed them to AIL, the procedure is described below. Nevertheless, moderate your queries! -Feed data to AIL: +1. [AIL Importers](./doc/README.md#ail-importers) -1. Clone the [pystemon's git repository](https://github.com/cvandeplas/pystemon): - ``` - git clone https://github.com/cvandeplas/pystemon.git - ``` - -2. Edit configuration file for pystemon ```pystemon/pystemon.yaml```: - - Configure the storage section according to your needs: - ``` - storage: - archive: - storage-classname: FileStorage - save: yes - save-all: yes - dir: "alerts" - dir-all: "archive" - compress: yes - - redis: - storage-classname: RedisStorage - save: yes - save-all: yes - server: "localhost" - port: 6379 - database: 10 - lookup: no - ``` - - Adjust the configuration for paste-sites based on your requirements (remember to throttle download and update times). - -3. Install python dependencies inside the virtual environment: - ```shell - cd ail-framework/ - . ./AILENV/bin/activate - cd pystemon/ - pip install -U -r requirements.txt - ``` -4. Edit the configuration file ```ail-framework/configs/core.cfg```: - - Modify the "pystemonpath" path accordingly. - -5. Launch ail-framework, pystemon and PystemonImporter.py (all within the virtual environment): - - Option 1 (recommended): - ``` - ./ail-framework/bin/LAUNCH.py -l #starts ail-framework - ./ail-framework/bin/LAUNCH.py -f #starts pystemon and the PystemonImporter.py - ``` - - Option 2 (may require two terminal windows): - ``` - ./ail-framework/bin/LAUNCH.py -l #starts ail-framework - ./pystemon/pystemon.py - ./ail-framework/bin/importer/PystemonImporter.py - ``` +2. ZMQ: Be a collaborator of CIRCL and ask to access our feed. It will be sent to the static IP you are using for AIL. ## How to create a new module diff --git a/README.md b/README.md index 178ab9a2..7b5818fc 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,25 @@ AIL is a modular framework to analyse potential information leaks from unstructu ![Finding webshells with AIL](./doc/screenshots/webshells.gif?raw=true "Finding webshells with AIL") +## AIL V5.0 Version: + +AIL v5.0 introduces significant improvements and new features: + +- **Codebase Rewrite**: The codebase has undergone a substantial rewrite, +resulting in enhanced performance and speed improvements. +- **Database Upgrade**: The database has been migrated from ARDB to Kvrocks. +- **New Correlation Engine**: AIL v5.0 introduces a new powerful correlation engine with two new correlation types: CVE and Title. +- **Enhanced Logging**: The logging system has been improved to provide better troubleshooting capabilities. +- **Tagging Support**: [AIL objects](./doc/README.md#ail-objects) now support tagging, +allowing users to categorize and label extracted information for easier analysis and organization. +- **Trackers**: Improved objects filtering, PGP and decoded tracking added. +- **UI Content Visualization**: The user interface has been upgraded to visualize extracted and tracked information. +- **New Crawler Lacus**: improve crawling capabilities. +- **Modular Importers and Exporters**: New importers (ZMQ, AIL Feeders) and exporters (MISP, Mail, TheHive) modular design. +Allow easy creation and customization by extending an abstract class. +- **Module Queues**: improved the queuing mechanism between detection modules. +- **New Object CVE and Title**: Extract an correlate CVE IDs and web page titles. + ## Features - Modular architecture to handle streams of unstructured or structured information diff --git a/doc/README.md b/doc/README.md index 366f4146..d6b707e8 100644 --- a/doc/README.md +++ b/doc/README.md @@ -1,1295 +1,223 @@ -# API DOCUMENTATION -## General -### Automation key +# AIL objects + +AIL is using different types of objects to classify, correlate and describe extracted information: + +- **Cryptocurrency**: Represents extracted cryptocurrency addresses. + - bitcoin + - bitcoin-cash + - dash + - ethereum + - litecoin + - monero + - zcash +- **Cve**: Represents extracted CVE (Common Vulnerabilities and Exposures) IDs. +- **Decoded**: Represents information that has been decoded from an encoded format, such as base64. +- **Domain**: Represents crawled domains and includes metadata related to them. +- **Item**: Represents a piece of text that has been processed by AIL. It can include various types of extracted information. +- **Pgp**: Represents PGP key/block metadata. + - key: PGP key IDs + - mail: email addresses associated with PGP keys + - name: names associated with PGP keys. +- **Screenshot**: Represents screenshots captured from crawled domains. +- **Title**: Represents the HTML title extracted from web pages. +- **Username**: + - telegram: telegram username handles + - twitter: twitter username handles + - jabber: Jabber (XMPP) username handles + + +# AIL Importers + +AIL Importers play a crucial role in the AIL ecosystem, +enabling the import of various types of data into the framework. + +These importers are located in the `/bin/importer` directory. +The modular design of importers allows for easy expansion and customization, +ensuring that AIL can adapt to new types of data. + +Available Importers: +- [AIL Feeders](#ail-feeders): Extract and feed JSON data from external sources via The API. +- ZMQ +- [pystemon](https://github.com/cvandeplas/pystemon) +- File: Import files and directories. +(Manually Feed File/Dir: [./tool/file_dir_importer.py](./tool/file_dir_importer.py)). + +[//]: # (### ZMQ Importer:) + +### pystemon: + +1. Clone the [pystemon's git repository](https://github.com/cvandeplas/pystemon): + ``` + git clone https://github.com/cvandeplas/pystemon.git + ``` + +2. Edit configuration file for pystemon ```pystemon/pystemon.yaml```: + - Configure the storage section according to your needs: + ``` + storage: + archive: + storage-classname: FileStorage + save: yes + save-all: yes + dir: "alerts" + dir-all: "archive" + compress: yes + + redis: + storage-classname: RedisStorage + save: yes + save-all: yes + server: "localhost" + port: 6379 + database: 10 + lookup: no + ``` + - Adjust the configuration for paste-sites based on your requirements (remember to throttle download and update times). + +3. Install python dependencies inside the virtual environment: + ```shell + cd ail-framework/ + . ./AILENV/bin/activate + cd pystemon/ + pip install -U -r requirements.txt + ``` +4. Edit the configuration file ```ail-framework/configs/core.cfg```: + - Modify the "pystemonpath" path accordingly. + +5. Launch ail-framework, pystemon and PystemonImporter.py (all within the virtual environment): + - Option 1 (recommended): + ``` + ./ail-framework/bin/LAUNCH.py -l #starts ail-framework + ./ail-framework/bin/LAUNCH.py -f #starts pystemon and the PystemonImporter.py + ``` + - Option 2 (may require two terminal windows): + ``` + ./ail-framework/bin/LAUNCH.py -l #starts ail-framework + ./pystemon/pystemon.py + ./ail-framework/bin/importer/PystemonImporter.py + ``` + +### File Importer `importer/FileImporter.py`: + +Manually import File and Directory with the [./tool/file_dir_importer.py](./tool/file_dir_importer.py) script: + +- Import Files: + ```shell + . ./AILENV/bin/activate + cd tools/ + ./file_dir_importer.py -f MY_FILE_PAT + ``` + +- Import Dirs: + ```shell + . ./AILENV/bin/activate + cd tools/ + ./file_dir_importer.py -d MY_DIR_PATH + ``` + +### Create a New Importer: + +```python +from importer.abstract_importer import AbstractImporter +from modules.abstract_module import AbstractModule + +class MyNewImporter(AbstractImporter): + + def __init__(self): + super().__init__() + # super().__init__(queue=True) # if it's an one-time run importer + self.logger.info(f'Importer {self.name} initialized') + + def importer(self, my_var): + # Process my_var and get content to import + content = GET_MY_CONTENT_TO_IMPORT + # if content is not gzipped and/or not b64 encoded, + # set gzipped and/or b64 to False + message = self.create_message(item_id, content, b64=False, gzipped=False) + return message + # if it's an one-time run, otherwise create an AIL Module + # self.add_message_to_queue(message) + +class MyNewModuleImporter(AbstractModule): + def __init__(self): + super().__init__() + # init module ... + self.importer = MyNewImporter() + + def get_message(self): + return self.importer.importer() + + def compute(self, message): + self.add_message_to_queue(message) + +if __name__ == '__main__': + module = MyNewModuleImporter() + module.run() + + # if it's an one-time run: + # importer = MyImporter() + # importer.importer(my_var) +``` + +## AIL Feeders + +AIL Feeders are a special type of Importer within AIL, specifically designed +to *extract* and *feed* data from external sources into the framework. + +- **Extract Data**: AIL Feeders extract data from external sources, such as APK files, +certificate transparency logs, GitHub archives, repositories, ActivityPub sources, +leaked files, Atom/RSS feeds, JSON logs, Discord, and Telegram, ... +- **Run Independently**: Feeders can run on separate systems or infrastructure, +providing flexibility and scalability. They operate independently from the core AIL framework. +- **Internal Logic**: Each feeder can implement its own custom logic and processing +to extract and transform data and metadata from the source into JSON. +- **Push to AIL API**: The generated JSON is then pushed to the AIL API +for ingestion and further analysis within the AIL framework. + +[//]: # (- Customize medata parsing) + + +### AIL Feeders List: +- [ail-feeder-apk](https://github.com/ail-project/ail-feeder-apk): Pushes annotated APK to an AIL instance for yara detection. +- [ail-feeder-ct](https://github.com/ail-project/ail-feeder-ct): AIL feeder for certificate transparency. +- [ail-feeder-github-gharchive](https://github.com/ail-project/ail-feeder-gharchive): extract informations +from GHArchive, collect and feed AIL +- [ail-feeder-github-repo](https://github.com/ail-project/ail-feeder-github-repo): Pushes github repositories to AIL. +- [ail-feeder-activity-pub](https://github.com/ail-project/ail-feeder-activity-pub) ActivityPub feeder. +- [ail-feeder-leak](https://github.com/ail-project/ail-feeder-leak): Automates the process of feeding files to AIL, using data chunking to handle large files. +- [ail-feeder-atom-rss](https://github.com/ail-project/ail-feeder-atom-rss) Atom and RSS feeder for AIL. +- [ail-feeder-jsonlogs](https://github.com/ail-project/ail-feeder-jsonlogs) Aggregate JSON log lines and pushes them to AIL. +- [ail-feeder-discord](https://github.com/ail-project/ail-feeder-discord) Discord Feeder. +- [ail-feeder-telegram](https://github.com/ail-project/ail-feeder-telegram) Telegram Channels and User Feeder. + +#### Example: Feeding AIL with Conti leaks + +```python +from pyail import PyAIL +pyail = PyAIL(URL, API_KEY, ssl=verifycert) + +#. . . imports +#. . . setup code + +for content in sys.stdin: + elm = json.loads(content) + tmp = elm['body'] + meta = {} + meta['jabber:to'] = elm['to'] + meta['jabber:from'] = elm['from'] + meta['jabber:ts]' = elm['ts'] + pyail.feed_json_item(tmp , meta, feeder_name, feeder_uuid) +``` + +# AIL SYNC + +The synchronisation mechanism allow the sync from one AIL instance to another AIL using a standard WebSocket +using [AIL JSON protocol](https://github.com/ail-project/ail-exchange-format/blob/main/ail-stream.md). +The synchronisation allows to filter and sync specific collected items including crawled items or +specific tagged items matching defined rules. +This feature can be very useful to limit the scope of analysis in specific fields or resource intensive activity. +This sync can be also used to share filtered streams with other partners. -The authentication of the automation is performed via a secure key available in the AIL UI interface. Make sure you keep that key secret. It gives access to the entire database! The API key is available in the ``Server Management`` menu under ``My Profile``. - -The authorization is performed by using the following header: - -~~~~ -Authorization: YOUR_API_KEY -~~~~ -### Accept and Content-Type headers - -When submitting data in a POST, PUT or DELETE operation you need to specify in what content-type you encoded the payload. This is done by setting the below Content-Type headers: - -~~~~ -Content-Type: application/json -~~~~ - -Example: - -~~~~ -curl --header "Authorization: YOUR_API_KEY" --header "Content-Type: application/json" https://AIL_URL/ -~~~~ - -## Item management - -### Get item: `api/v1/get/item/default` - -#### Description -Get item default info. - -**Method** : `POST` - -#### Parameters -- `id` - - item id - - *str - relative item path* - - mandatory - -#### JSON response -- `content` - - item content - - *str* -- `id` - - item id - - *str* -- `date` - - item date - - *str - YYMMDD* -- `tags` - - item tags list - - *list* - -#### Example -``` -curl https://127.0.0.1:7000/api/v1/get/item/default --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz" - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` - -```json - { - "content": "item content test", - "date": "20190726", - "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", - "tags": - [ - "misp-galaxy:backdoor=\"Rosenbridge\"", - "infoleak:automatic-detection=\"pgp-message\"", - "infoleak:automatic-detection=\"encrypted-private-key\"", - "infoleak:submission=\"manual\"", - "misp-galaxy:backdoor=\"SLUB\"" - ] - } -``` - -#### Expected Fail Response - -**HTTP Status Code** : `400` -```json - {"status": "error", "reason": "Mandatory parameter(s) not provided"} -``` -**HTTP Status Code** : `404` -```json - {"status": "error", "reason": "Item not found"} -``` - - - - -### Get item content: `api/v1/get/item/content` - -#### Description -Get a specific item content. - -**Method** : `POST` - -#### Parameters -- `id` - - item id - - *str - relative item path* - - mandatory - -#### JSON response -- `content` - - item content - - *str* -- `id` - - item id - - *str* - -#### Example -``` -curl https://127.0.0.1:7000/api/v1/get/item/content --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz" - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` - -```json - { - "content": "item content test", - "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz" - } -``` - -#### Expected Fail Response - -**HTTP Status Code** : `400` -```json - {"status": "error", "reason": "Mandatory parameter(s) not provided"} -``` -**HTTP Status Code** : `404` -```json - {"status": "error", "reason": "Item not found"} -``` - - - -### Get item content: `api/v1/get/item/tag` - -#### Description -Get all tags from an item. - -**Method** : `POST` - -#### Parameters -- `id` - - item id - - *str - relative item path* - - mandatory - -#### JSON response -- `content` - - item content - - *str* -- `tags` - - item tags list - - *list* - -#### Example -``` -curl https://127.0.0.1:7000/api/v1/get/item/tag --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz" - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` - -```json - { - "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", - "tags": - [ - "misp-galaxy:backdoor=\"Rosenbridge\"", - "infoleak:automatic-detection=\"pgp-message\"", - "infoleak:automatic-detection=\"encrypted-private-key\"", - "infoleak:submission=\"manual\"", - "misp-galaxy:backdoor=\"SLUB\"" - ] - } -``` - -#### Expected Fail Response - -**HTTP Status Code** : `400` -```json - {"status": "error", "reason": "Mandatory parameter(s) not provided"} -``` -**HTTP Status Code** : `404` -```json - {"status": "error", "reason": "Item not found"} -``` - - - -### Advanced Get item: `api/v1/get/item` - -#### Description -Get item. Filter requested field. - -**Method** : `POST` - -#### Parameters -- `id` - - item id - - *str - relative item path* - - mandatory -- `date` - - get item date - - *boolean* - - default: `true` -- `tags` - - get item tags - - *boolean* - - default: `true` -- `content` - - get item content - - *boolean* - - default: `false` -- `size` - - get item size - - *boolean* - - default: `false` -- `lines` - - get item lines info - - *boolean* - - default: `false` -- `cryptocurrency` - - `bitcoin` - - get item bitcoin adress - - *boolean* - - default: `false` -- `pgp` - - `key` - - get item pgp key - - *boolean* - - default: `false` - - `mail` - - get item pgp mail - - *boolean* - - default: `false` - - `name` - - get item pgp name - - *boolean* - - default: `false` - - -#### JSON response -- `content` - - item content - - *str* -- `id` - - item id - - *str* -- `date` - - item date - - *str - YYMMDD* -- `tags` - - item tags list - - *list* -- `size` - - item size (Kb) - - *int* -- `lines` - - item lines info - - *{}* - - `max_length` - - line max length line - - *int* - - `nb` - - nb lines item - - *int* -- `cryptocurrency` - - `bitcoin` - - item bitcoin adress - - *list* -- `pgp` - - `key` - - item pgp keys - - *list* - - `mail` - - item pgp mails - - *list* - - `name` - - item pgp name - - *list* - - -#### Example -``` -curl https://127.0.0.1:7000/api/v1/get/item --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json -{ - "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", - "content": true, - "lines_info": true, - "tags": true, - "size": true -} -``` - -#### Expected Success Response -**HTTP Status Code** : `200` -```json - { - "content": "dsvcdsvcdsc vvvv", - "cryptocurrency": { - "bitcoin": [ - "132M1aGTGodHkQNh1augLeMjEXH51wgoCc" - ] - }, - "date": "20190726", - "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", - "lines": { - "max_length": 19, - "nb": 1 - }, - "pgp": { - "key": [ - "0x5180D21F4C20F975" - ], - "mail": [ - "mail@test.test" - ], - "name": [ - "user_test" - ] - }, - "size": 0.03, - "tags": [ - "misp-galaxy:stealer=\"Vidar\"", - "infoleak:submission=\"manual\"" - ] - } -``` - -#### Expected Fail Response -**HTTP Status Code** : `400` -```json - {"status": "error", "reason": "Mandatory parameter(s) not provided"} -``` -**HTTP Status Code** : `404` -```json - {"status": "error", "reason": "Item not found"} -``` - - - - - -### Add item tags: `api/v1/add/item/tag` - -#### Description -Add tags to an item. - -**Method** : `POST` - -#### Parameters -- `id` - - item id - - *str - relative item path* - - mandatory -- `tags` - - list of tags - - *list* - - default: `[]` -- `galaxy` - - list of galaxy - - *list* - - default: `[]` - -#### JSON response -- `id` - - item id - - *str - relative item path* -- `tags` - - list of item tags added - - *list* - -#### Example -``` -curl https://127.0.0.1:7000/api/v1/import/item --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", - "tags": [ - "infoleak:analyst-detection=\"private-key\"", - "infoleak:analyst-detection=\"api-key\"" - ], - "galaxy": [ - "misp-galaxy:stealer=\"Vidar\"" - ] - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` - -```json - { - "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", - "tags": [ - "infoleak:analyst-detection=\"private-key\"", - "infoleak:analyst-detection=\"api-key\"", - "misp-galaxy:stealer=\"Vidar\"" - ] - } -``` - -#### Expected Fail Response -**HTTP Status Code** : `400` - -```json - {"status": "error", "reason": "Item id not found"} - {"status": "error", "reason": "Tags or Galaxy not specified"} - {"status": "error", "reason": "Tags or Galaxy not enabled"} -``` - - - - -### Delete item tags: `api/v1/delete/item/tag` - -#### Description -Delete tags from an item. - -**Method** : `DELETE` - -#### Parameters -- `id` - - item id - - *str - relative item path* - - mandatory -- `tags` - - list of tags - - *list* - - default: `[]` - -#### JSON response -- `id` - - item id - - *str - relative item path* -- `tags` - - list of item tags deleted - - *list* - -#### Example -``` -curl https://127.0.0.1:7000/api/v1/delete/item/tag --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X DELETE -``` - -#### input.json Example -```json - { - "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", - "tags": [ - "infoleak:analyst-detection=\"private-key\"", - "infoleak:analyst-detection=\"api-key\"", - "misp-galaxy:stealer=\"Vidar\"" - ] - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` - -```json - { - "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", - "tags": [ - "infoleak:analyst-detection=\"private-key\"", - "infoleak:analyst-detection=\"api-key\"", - "misp-galaxy:stealer=\"Vidar\"" - ] - } -``` - -#### Expected Fail Response -**HTTP Status Code** : `400` - -```json - {"status": "error", "reason": "Item id not found"} - {"status": "error", "reason": "No Tag(s) specified"} -``` - - - - - - -## Tag management - - -### Get all AIL tags: `api/v1/get/tag/all` - -#### Description -Get all tags used in AIL. - -**Method** : `GET` - -#### JSON response -- `tags` - - list of tag - - *list* -#### Example -``` -curl https://127.0.0.1:7000/api/v1/get/tag/all --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" -``` - -#### Expected Success Response -**HTTP Status Code** : `200` -```json - { - "tags": [ - "misp-galaxy:backdoor=\"Rosenbridge\"", - "infoleak:automatic-detection=\"pgp-private-key\"", - "infoleak:automatic-detection=\"pgp-signature\"", - "infoleak:automatic-detection=\"base64\"", - "infoleak:automatic-detection=\"encrypted-private-key\"", - "infoleak:submission=\"crawler\"", - "infoleak:automatic-detection=\"binary\"", - "infoleak:automatic-detection=\"pgp-public-key-block\"", - "infoleak:automatic-detection=\"hexadecimal\"", - "infoleak:analyst-detection=\"private-key\"", - "infoleak:submission=\"manual\"", - "infoleak:automatic-detection=\"private-ssh-key\"", - "infoleak:automatic-detection=\"iban\"", - "infoleak:automatic-detection=\"pgp-message\"", - "infoleak:automatic-detection=\"certificate\"", - "infoleak:automatic-detection=\"credential\"", - "infoleak:automatic-detection=\"cve\"", - "infoleak:automatic-detection=\"google-api-key\"", - "infoleak:automatic-detection=\"phone-number\"", - "infoleak:automatic-detection=\"rsa-private-key\"", - "misp-galaxy:backdoor=\"SLUB\"", - "infoleak:automatic-detection=\"credit-card\"", - "misp-galaxy:stealer=\"Vidar\"", - "infoleak:automatic-detection=\"private-key\"", - "infoleak:automatic-detection=\"api-key\"", - "infoleak:automatic-detection=\"mail\"" - ] - } -``` - - - - -### Get tag metadata: `api/v1/get/tag/metadata` - -#### Description -Get tag metadata. - -**Method** : `POST` - -#### Parameters -- `tag` - - tag name - - *str* - - mandatory - -#### JSON response -- `tag` - - tag name - - *str* -- `first_seen` - - date: first seen - - *str - YYYYMMDD* -- `last_seen` - - date: last seen - - *str - YYYYMMDD* -#### Example -``` -curl https://127.0.0.1:7000/api/v1/get/tag/metadata --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "tag": "infoleak:submission=\"manual\"" - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` -```json - { - "first_seen": "20190605", - "last_seen": "20190726", - "tag": "infoleak:submission=\"manual\"" - } -``` - -#### Expected Fail Response -**HTTP Status Code** : `404` -```json - {"status": "error", "reason": "Tag not found"} -``` - - - - -## Cryptocurrency - - - -### Get bitcoin metadata: `api/v1/get/cryptocurrency/bitcoin/metadata` - -#### Description -Get all metdata from a bitcoin address. - -**Method** : `POST` - -#### Parameters -- `bitcoin` - - bitcoin address - - *str* - - mandatory - -#### JSON response -- `bitcoin` - - bitcoin address - - *str* -- `first_seen` - - date: first seen - - *str - YYYYMMDD* -- `last_seen` - - date: last seen - - *str - YYYYMMDD* -#### Example -``` -curl https://127.0.0.1:7000/api/v1/get/cryptocurrency/bitcoin/metadata --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "bitcoin": "3DZfm5TQaJKcJm9PsuaWmSz9XmHMLxVv3y" - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` -```json - { - "bitcoin": "3DZfm5TQaJKcJm9PsuaWmSz9XmHMLxVv3y", - "first_seen": "20190605", - "last_seen": "20190726" - } -``` - -#### Expected Fail Response -**HTTP Status Code** : `404` -```json - {"status": "error", "reason": "Item not found"} -``` - - - -### Get bitcoin metadata: `api/v1/get/cryptocurrency/bitcoin/item` - -#### Description -Get all items related to a bitcoin address. - -**Method** : `POST` - -#### Parameters -- `bitcoin` - - bitcoin address - - *str* - - mandatory - -#### JSON response -- `bitcoin` - - bitcoin address - - *str* -- `items` - - list of item id - - *list* -#### Example -``` -curl https://127.0.0.1:7000/api/v1/get/cryptocurrency/bitcoin/item --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "bitcoin": "3DZfm5TQaJKcJm9PsuaWmSz9XmHMLxVv3y" - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` -```json - { - "bitcoin": "3DZfm5TQaJKcJm9PsuaWmSz9XmHMLxVv3y", - "items": [ - "archive/2019/08/26/test_bitcoin001", - "archive/2019/08/26/test_bitcoin002", - "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz" - ] - } -``` - -#### Expected Fail Response -**HTTP Status Code** : `404` -```json - {"status": "error", "reason": "Item not found"} -``` - - - - - - - -## Tracker - - - -### Add term tracker: `api/v1/add/tracker` - -#### Description -Create a new tracker (word, set, regex). - -You need to use a regex if you want to use one of the following special characters [<>~!?@#$%^&*|()_-+={}\":;,.\'\n\r\t]/\\ - - -**Method** : `POST` - -#### Parameters -- `term` - - term to add - - *str - word(s)* - - mandatory -- `nb_words` - - number of words in set - - *int* - - default: `1` -- `type` - - term type - - *str* - - mandatory: `word`, `set`, `regex` -- `tags` - - list of tags - - *list* - - default: `[]` -- `mails` - - list of mails to notify - - *list* - - default: `[]` -- `level` - - tracker visibility - - *int - 0: user only, 1: all users* - - default: `1` -- `description` - - tracker description - - *str* - -#### JSON response -- `uuid` - - import uuid - - *uuid4* - -#### Example -``` -curl https://127.0.0.1:7000/api/v1/add/tracker --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "term": "test test2 test3", - "type": "set", - "nb_words": 2, - "tags": [ - "mytags", - "othertags" - ], - "mails": [ - "mail@mail.test", - "othermail@mail.test" - ], - "level": 1 - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` - -```json - { - "uuid": "6a16b06e-38e5-41e1-904d-3960610647e8" - } -``` - -#### Expected Fail Response -**HTTP Status Code** : 400 - -```json - {"status": "error", "reason": "Term not provided"} - {"status": "error", "reason": "Term type not provided"} - {"status": "error", "reason": "special character not allowed", "message": "Please use a regex or remove all special characters"} - {"status": "error", "reason": "Incorrect type"} -``` -**HTTP Status Code** : 409 - -```json - {"status": "error", "reason": "Term already tracked"} -``` - - - -### Delete term tracker: `api/v1/delete/tracker` - -#### Description -Delete a tracker - -**Method** : `DELETE` - -#### Parameters -- `uuid` - - tracked term uuid - - *uuid4* - - mandatory - -#### JSON response -- `uuid` - - deleted uuid - - *uuid4* - -#### Example -``` -curl https://127.0.0.1:7000/api/v1/delete/tracker --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "uuid": "6a16b06e-38e5-41e1-904d-3960610647e8" - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` - -```json - { - "uuid": "6a16b06e-38e5-41e1-904d-3960610647e8" - } -``` - -#### Expected Fail Response -**HTTP Status Code** : `400` - -```json - {"status": "error", "reason": "Invalid uuid"} - -``` - -**HTTP Status Code** : `404` - -```json - ({"status": "error", "reason": "Unknown uuid"} - -``` - - -### Delete term tracker: `api/v1/get/tracker/item` - -#### Description -Get tracked items by date-range - -**Method** : `POST` - -#### Parameters -- `uuid` - - tracked term uuid - - *uuid4* - - mandatory -- `date_from` - - date from - - *str - YYMMDD* - - default: last tracked items date -- `date_to` - - date to - - *str - YYMMDD* - - default: `None` - -#### JSON response -- `uuid` - - term uuid - - *uuid4* -- `date_from` - - date from - - *str - YYMMDD* -- `date_to` - - date to - - *str - YYMMDD* -- `items` - - list of item id - - *list* - -#### Example -``` -curl https://127.0.0.1:7000/api/v1/get/tracker/item --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "uuid": "6a16b06e-38e5-41e1-904d-3960610647e8", - "date_from": "20190823", - "date_to": "20190829", - "items": [ - { - "id": "submitted/2019/08/25/4f929998-3921-4be3-b448-be3bf1722d6b.gz", - "date": 20190825, - "tags": [ - "infoleak:automatic-detection=\"credential\"", - "mytags", - "othertags", - ] - } - ] - } -``` - -**HTTP Status Code** : `400` - -```json - {"status": "error", "reason": "Invalid uuid"} - -``` - -**HTTP Status Code** : `404` - -```json - ({"status": "error", "reason": "Unknown uuid"} - -``` - - - -## Domain - - -### Get min domain metadata: `api/v1/get/crawled/domain/list` - -#### Description -Get crawled domain by date-range and status (default status = *UP*) - -**Method** : `POST` - -#### Parameters -- `domain_type` - - domain type: *onion* or *regular* - - *str* - - default: *regular* -- `date_from` - - date from - - *str - YYYYMMDD* - - mandatory -- `date_to` - - date to - - *str - YYYYMMDD* - - mandatory - -#### JSON response -- `domain_type` - - domain type: *onion* or *regular* - - *str* -- `date_from` - - date from - - *str - YYYYMMDD* -- `date_to` - - date to - - *str - YYYYMMDD* -- `domains` - - list of domains - - *list - list of domains* - -#### Example -``` -curl https://127.0.0.1:7000/api/v1/get/crawled/domain/list --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "date_from": "20191001", - "date_to": "20191222", - "domain_type": "onion" - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` - -```json - { - "date_from": "20191001", - "date_to": "20191222", - "domain_status": "UP", - "domain_type": "onion", - "domains": [ - "2222222222222222.onion" - ] - } -``` - - - - -### Get min domain metadata: `api/v1/get/domain/status/minimal` - -#### Description -Get min domain metadata - -**Method** : `POST` - -#### Parameters -- `domain` - - domain name - - *str* - - mandatory - -#### JSON response -- `domain` - - domain - - *str* -- `first_seen` - - domain first up time - - *epoch* -- `last_seen` - - domain last up time - - *epoch* - -#### Example -``` -curl https://127.0.0.1:7000/api/v1/get/domain/status/minimal --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "domain": "2222222222222222.onion", - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` - -```json - { - "domain": "2222222222222222.onion", - "first_seen": 1571314000, - "last_seen": 1571314000 - } -``` - -**HTTP Status Code** : `404` - -```json - ({"status": "error", "reason": "Domain not found"} - -``` - - - -## Import management - - - -### Import item (currently: text only): `api/v1/import/item` - -#### Description -Allows users to import new items. asynchronous function. - -**Method** : `POST` - -#### Parameters -- `type` - - import type - - *str* - - default: `text` -- `text` - - text to import - - *str* - - mandatory if type = text -- `default_tags` - - add default import tag - - *boolean* - - default: True -- `tags` - - list of tags - - *list* - - default: `[]` -- `galaxy` - - list of galaxy - - *list* - - default: `[]` - -#### JSON response -- `uuid` - - import uuid - - *uuid4* - -#### Example -``` -curl https://127.0.0.1:7000/api/v1/import/item --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "type": "text", - "tags": [ - "infoleak:analyst-detection=\"private-key\"" - ], - "text": "text to import" - } -``` - -#### Expected Success Response -**HTTP Status Code** : `200` - -```json - { - "uuid": "0c3d7b34-936e-4f01-9cdf-2070184b6016" - } -``` - -#### Expected Fail Response -**HTTP Status Code** : `400` - -```json - {"status": "error", "reason": "Malformed JSON"} - {"status": "error", "reason": "No text supplied"} - {"status": "error", "reason": "Tags or Galaxy not enabled"} - {"status": "error", "reason": "Size exceeds default"} -``` - - - - - -### GET Import item info: `api/v1/get/import/item/` - -#### Description - -Get import status and all items imported by uuid - -**Method** : `POST` - -#### Parameters - -- `uuid` - - import uuid - - *uuid4* - - mandatory - -#### JSON response - -- `status` - - import status - - *str* - - values: `in queue`, `in progress`, `imported` -- `items` - - list of imported items id - - *list* - - The full list of imported items is not complete until `status` = `"imported"` - -#### Example - -``` -curl -k https://127.0.0.1:7000/api/v1/get/import/item --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST -``` - -#### input.json Example -```json - { - "uuid": "0c3d7b34-936e-4f01-9cdf-2070184b6016" - } -``` - -#### Expected Success Response - -**HTTP Status Code** : `200` - -```json - { - "items": [ - "submitted/2019/07/26/b20a69f1-99ad-4cb3-b212-7ce24b763b50.gz" - ], - "status": "imported" - } -``` - -#### Expected Fail Response - -**HTTP Status Code** : `400` - -```json - {"status": "error", "reason": "Invalid uuid"} - {"status": "error", "reason": "Unknown uuid"} -``` - - - - -# FUTURE endpoints - -
-Endpoints - -### Submit a domain to crawl TODO -##### ``api/add/crawler/task`` POST - -### Create a term/set/regex/yara tracker -##### ``api/add/tracker/`` POST - -### Get tracker -##### ``api/get/tracker`` POST - ------ - - - -### Get domain tags -##### ``api/get/domain/tags/`` POST - -### Get domain history -##### ``api/get/domain/history/`` POST - ------ - -### Get decoded item metadata -### Check if a decoded item exists (via sha1) -##### ``api/get/decoded/metadata/`` POST - ------ - - ------ -##### ``api/get/cryptocurrency`` POST - -### Check if a cryptocurrency address (bitcoin, ..) exists -##### ``api/get/cryptocurrency/`` POST - -### Get cryptocurrency address metadata -##### ``api/get/cryptocurrency/metadata/`` POST - ------ - -### Object correlation (1 depth) -##### ``api/get/correlation/`` POST - -### Create MISP event from object -##### ``api/export/misp`` POST - -
- ------ diff --git a/doc/ail_modules_queues.png b/doc/ail_modules_queues.png new file mode 100644 index 00000000..66a082dd Binary files /dev/null and b/doc/ail_modules_queues.png differ diff --git a/doc/api.md b/doc/api.md new file mode 100644 index 00000000..366f4146 --- /dev/null +++ b/doc/api.md @@ -0,0 +1,1295 @@ +# API DOCUMENTATION + +## General + +### Automation key + +The authentication of the automation is performed via a secure key available in the AIL UI interface. Make sure you keep that key secret. It gives access to the entire database! The API key is available in the ``Server Management`` menu under ``My Profile``. + +The authorization is performed by using the following header: + +~~~~ +Authorization: YOUR_API_KEY +~~~~ +### Accept and Content-Type headers + +When submitting data in a POST, PUT or DELETE operation you need to specify in what content-type you encoded the payload. This is done by setting the below Content-Type headers: + +~~~~ +Content-Type: application/json +~~~~ + +Example: + +~~~~ +curl --header "Authorization: YOUR_API_KEY" --header "Content-Type: application/json" https://AIL_URL/ +~~~~ + +## Item management + +### Get item: `api/v1/get/item/default` + +#### Description +Get item default info. + +**Method** : `POST` + +#### Parameters +- `id` + - item id + - *str - relative item path* + - mandatory + +#### JSON response +- `content` + - item content + - *str* +- `id` + - item id + - *str* +- `date` + - item date + - *str - YYMMDD* +- `tags` + - item tags list + - *list* + +#### Example +``` +curl https://127.0.0.1:7000/api/v1/get/item/default --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz" + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` + +```json + { + "content": "item content test", + "date": "20190726", + "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", + "tags": + [ + "misp-galaxy:backdoor=\"Rosenbridge\"", + "infoleak:automatic-detection=\"pgp-message\"", + "infoleak:automatic-detection=\"encrypted-private-key\"", + "infoleak:submission=\"manual\"", + "misp-galaxy:backdoor=\"SLUB\"" + ] + } +``` + +#### Expected Fail Response + +**HTTP Status Code** : `400` +```json + {"status": "error", "reason": "Mandatory parameter(s) not provided"} +``` +**HTTP Status Code** : `404` +```json + {"status": "error", "reason": "Item not found"} +``` + + + + +### Get item content: `api/v1/get/item/content` + +#### Description +Get a specific item content. + +**Method** : `POST` + +#### Parameters +- `id` + - item id + - *str - relative item path* + - mandatory + +#### JSON response +- `content` + - item content + - *str* +- `id` + - item id + - *str* + +#### Example +``` +curl https://127.0.0.1:7000/api/v1/get/item/content --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz" + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` + +```json + { + "content": "item content test", + "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz" + } +``` + +#### Expected Fail Response + +**HTTP Status Code** : `400` +```json + {"status": "error", "reason": "Mandatory parameter(s) not provided"} +``` +**HTTP Status Code** : `404` +```json + {"status": "error", "reason": "Item not found"} +``` + + + +### Get item content: `api/v1/get/item/tag` + +#### Description +Get all tags from an item. + +**Method** : `POST` + +#### Parameters +- `id` + - item id + - *str - relative item path* + - mandatory + +#### JSON response +- `content` + - item content + - *str* +- `tags` + - item tags list + - *list* + +#### Example +``` +curl https://127.0.0.1:7000/api/v1/get/item/tag --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz" + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` + +```json + { + "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", + "tags": + [ + "misp-galaxy:backdoor=\"Rosenbridge\"", + "infoleak:automatic-detection=\"pgp-message\"", + "infoleak:automatic-detection=\"encrypted-private-key\"", + "infoleak:submission=\"manual\"", + "misp-galaxy:backdoor=\"SLUB\"" + ] + } +``` + +#### Expected Fail Response + +**HTTP Status Code** : `400` +```json + {"status": "error", "reason": "Mandatory parameter(s) not provided"} +``` +**HTTP Status Code** : `404` +```json + {"status": "error", "reason": "Item not found"} +``` + + + +### Advanced Get item: `api/v1/get/item` + +#### Description +Get item. Filter requested field. + +**Method** : `POST` + +#### Parameters +- `id` + - item id + - *str - relative item path* + - mandatory +- `date` + - get item date + - *boolean* + - default: `true` +- `tags` + - get item tags + - *boolean* + - default: `true` +- `content` + - get item content + - *boolean* + - default: `false` +- `size` + - get item size + - *boolean* + - default: `false` +- `lines` + - get item lines info + - *boolean* + - default: `false` +- `cryptocurrency` + - `bitcoin` + - get item bitcoin adress + - *boolean* + - default: `false` +- `pgp` + - `key` + - get item pgp key + - *boolean* + - default: `false` + - `mail` + - get item pgp mail + - *boolean* + - default: `false` + - `name` + - get item pgp name + - *boolean* + - default: `false` + + +#### JSON response +- `content` + - item content + - *str* +- `id` + - item id + - *str* +- `date` + - item date + - *str - YYMMDD* +- `tags` + - item tags list + - *list* +- `size` + - item size (Kb) + - *int* +- `lines` + - item lines info + - *{}* + - `max_length` + - line max length line + - *int* + - `nb` + - nb lines item + - *int* +- `cryptocurrency` + - `bitcoin` + - item bitcoin adress + - *list* +- `pgp` + - `key` + - item pgp keys + - *list* + - `mail` + - item pgp mails + - *list* + - `name` + - item pgp name + - *list* + + +#### Example +``` +curl https://127.0.0.1:7000/api/v1/get/item --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json +{ + "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", + "content": true, + "lines_info": true, + "tags": true, + "size": true +} +``` + +#### Expected Success Response +**HTTP Status Code** : `200` +```json + { + "content": "dsvcdsvcdsc vvvv", + "cryptocurrency": { + "bitcoin": [ + "132M1aGTGodHkQNh1augLeMjEXH51wgoCc" + ] + }, + "date": "20190726", + "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", + "lines": { + "max_length": 19, + "nb": 1 + }, + "pgp": { + "key": [ + "0x5180D21F4C20F975" + ], + "mail": [ + "mail@test.test" + ], + "name": [ + "user_test" + ] + }, + "size": 0.03, + "tags": [ + "misp-galaxy:stealer=\"Vidar\"", + "infoleak:submission=\"manual\"" + ] + } +``` + +#### Expected Fail Response +**HTTP Status Code** : `400` +```json + {"status": "error", "reason": "Mandatory parameter(s) not provided"} +``` +**HTTP Status Code** : `404` +```json + {"status": "error", "reason": "Item not found"} +``` + + + + + +### Add item tags: `api/v1/add/item/tag` + +#### Description +Add tags to an item. + +**Method** : `POST` + +#### Parameters +- `id` + - item id + - *str - relative item path* + - mandatory +- `tags` + - list of tags + - *list* + - default: `[]` +- `galaxy` + - list of galaxy + - *list* + - default: `[]` + +#### JSON response +- `id` + - item id + - *str - relative item path* +- `tags` + - list of item tags added + - *list* + +#### Example +``` +curl https://127.0.0.1:7000/api/v1/import/item --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", + "tags": [ + "infoleak:analyst-detection=\"private-key\"", + "infoleak:analyst-detection=\"api-key\"" + ], + "galaxy": [ + "misp-galaxy:stealer=\"Vidar\"" + ] + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` + +```json + { + "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", + "tags": [ + "infoleak:analyst-detection=\"private-key\"", + "infoleak:analyst-detection=\"api-key\"", + "misp-galaxy:stealer=\"Vidar\"" + ] + } +``` + +#### Expected Fail Response +**HTTP Status Code** : `400` + +```json + {"status": "error", "reason": "Item id not found"} + {"status": "error", "reason": "Tags or Galaxy not specified"} + {"status": "error", "reason": "Tags or Galaxy not enabled"} +``` + + + + +### Delete item tags: `api/v1/delete/item/tag` + +#### Description +Delete tags from an item. + +**Method** : `DELETE` + +#### Parameters +- `id` + - item id + - *str - relative item path* + - mandatory +- `tags` + - list of tags + - *list* + - default: `[]` + +#### JSON response +- `id` + - item id + - *str - relative item path* +- `tags` + - list of item tags deleted + - *list* + +#### Example +``` +curl https://127.0.0.1:7000/api/v1/delete/item/tag --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X DELETE +``` + +#### input.json Example +```json + { + "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", + "tags": [ + "infoleak:analyst-detection=\"private-key\"", + "infoleak:analyst-detection=\"api-key\"", + "misp-galaxy:stealer=\"Vidar\"" + ] + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` + +```json + { + "id": "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz", + "tags": [ + "infoleak:analyst-detection=\"private-key\"", + "infoleak:analyst-detection=\"api-key\"", + "misp-galaxy:stealer=\"Vidar\"" + ] + } +``` + +#### Expected Fail Response +**HTTP Status Code** : `400` + +```json + {"status": "error", "reason": "Item id not found"} + {"status": "error", "reason": "No Tag(s) specified"} +``` + + + + + + +## Tag management + + +### Get all AIL tags: `api/v1/get/tag/all` + +#### Description +Get all tags used in AIL. + +**Method** : `GET` + +#### JSON response +- `tags` + - list of tag + - *list* +#### Example +``` +curl https://127.0.0.1:7000/api/v1/get/tag/all --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" +``` + +#### Expected Success Response +**HTTP Status Code** : `200` +```json + { + "tags": [ + "misp-galaxy:backdoor=\"Rosenbridge\"", + "infoleak:automatic-detection=\"pgp-private-key\"", + "infoleak:automatic-detection=\"pgp-signature\"", + "infoleak:automatic-detection=\"base64\"", + "infoleak:automatic-detection=\"encrypted-private-key\"", + "infoleak:submission=\"crawler\"", + "infoleak:automatic-detection=\"binary\"", + "infoleak:automatic-detection=\"pgp-public-key-block\"", + "infoleak:automatic-detection=\"hexadecimal\"", + "infoleak:analyst-detection=\"private-key\"", + "infoleak:submission=\"manual\"", + "infoleak:automatic-detection=\"private-ssh-key\"", + "infoleak:automatic-detection=\"iban\"", + "infoleak:automatic-detection=\"pgp-message\"", + "infoleak:automatic-detection=\"certificate\"", + "infoleak:automatic-detection=\"credential\"", + "infoleak:automatic-detection=\"cve\"", + "infoleak:automatic-detection=\"google-api-key\"", + "infoleak:automatic-detection=\"phone-number\"", + "infoleak:automatic-detection=\"rsa-private-key\"", + "misp-galaxy:backdoor=\"SLUB\"", + "infoleak:automatic-detection=\"credit-card\"", + "misp-galaxy:stealer=\"Vidar\"", + "infoleak:automatic-detection=\"private-key\"", + "infoleak:automatic-detection=\"api-key\"", + "infoleak:automatic-detection=\"mail\"" + ] + } +``` + + + + +### Get tag metadata: `api/v1/get/tag/metadata` + +#### Description +Get tag metadata. + +**Method** : `POST` + +#### Parameters +- `tag` + - tag name + - *str* + - mandatory + +#### JSON response +- `tag` + - tag name + - *str* +- `first_seen` + - date: first seen + - *str - YYYYMMDD* +- `last_seen` + - date: last seen + - *str - YYYYMMDD* +#### Example +``` +curl https://127.0.0.1:7000/api/v1/get/tag/metadata --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "tag": "infoleak:submission=\"manual\"" + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` +```json + { + "first_seen": "20190605", + "last_seen": "20190726", + "tag": "infoleak:submission=\"manual\"" + } +``` + +#### Expected Fail Response +**HTTP Status Code** : `404` +```json + {"status": "error", "reason": "Tag not found"} +``` + + + + +## Cryptocurrency + + + +### Get bitcoin metadata: `api/v1/get/cryptocurrency/bitcoin/metadata` + +#### Description +Get all metdata from a bitcoin address. + +**Method** : `POST` + +#### Parameters +- `bitcoin` + - bitcoin address + - *str* + - mandatory + +#### JSON response +- `bitcoin` + - bitcoin address + - *str* +- `first_seen` + - date: first seen + - *str - YYYYMMDD* +- `last_seen` + - date: last seen + - *str - YYYYMMDD* +#### Example +``` +curl https://127.0.0.1:7000/api/v1/get/cryptocurrency/bitcoin/metadata --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "bitcoin": "3DZfm5TQaJKcJm9PsuaWmSz9XmHMLxVv3y" + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` +```json + { + "bitcoin": "3DZfm5TQaJKcJm9PsuaWmSz9XmHMLxVv3y", + "first_seen": "20190605", + "last_seen": "20190726" + } +``` + +#### Expected Fail Response +**HTTP Status Code** : `404` +```json + {"status": "error", "reason": "Item not found"} +``` + + + +### Get bitcoin metadata: `api/v1/get/cryptocurrency/bitcoin/item` + +#### Description +Get all items related to a bitcoin address. + +**Method** : `POST` + +#### Parameters +- `bitcoin` + - bitcoin address + - *str* + - mandatory + +#### JSON response +- `bitcoin` + - bitcoin address + - *str* +- `items` + - list of item id + - *list* +#### Example +``` +curl https://127.0.0.1:7000/api/v1/get/cryptocurrency/bitcoin/item --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "bitcoin": "3DZfm5TQaJKcJm9PsuaWmSz9XmHMLxVv3y" + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` +```json + { + "bitcoin": "3DZfm5TQaJKcJm9PsuaWmSz9XmHMLxVv3y", + "items": [ + "archive/2019/08/26/test_bitcoin001", + "archive/2019/08/26/test_bitcoin002", + "submitted/2019/07/26/3efb8a79-08e9-4776-94ab-615eb370b6d4.gz" + ] + } +``` + +#### Expected Fail Response +**HTTP Status Code** : `404` +```json + {"status": "error", "reason": "Item not found"} +``` + + + + + + + +## Tracker + + + +### Add term tracker: `api/v1/add/tracker` + +#### Description +Create a new tracker (word, set, regex). + +You need to use a regex if you want to use one of the following special characters [<>~!?@#$%^&*|()_-+={}\":;,.\'\n\r\t]/\\ + + +**Method** : `POST` + +#### Parameters +- `term` + - term to add + - *str - word(s)* + - mandatory +- `nb_words` + - number of words in set + - *int* + - default: `1` +- `type` + - term type + - *str* + - mandatory: `word`, `set`, `regex` +- `tags` + - list of tags + - *list* + - default: `[]` +- `mails` + - list of mails to notify + - *list* + - default: `[]` +- `level` + - tracker visibility + - *int - 0: user only, 1: all users* + - default: `1` +- `description` + - tracker description + - *str* + +#### JSON response +- `uuid` + - import uuid + - *uuid4* + +#### Example +``` +curl https://127.0.0.1:7000/api/v1/add/tracker --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "term": "test test2 test3", + "type": "set", + "nb_words": 2, + "tags": [ + "mytags", + "othertags" + ], + "mails": [ + "mail@mail.test", + "othermail@mail.test" + ], + "level": 1 + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` + +```json + { + "uuid": "6a16b06e-38e5-41e1-904d-3960610647e8" + } +``` + +#### Expected Fail Response +**HTTP Status Code** : 400 + +```json + {"status": "error", "reason": "Term not provided"} + {"status": "error", "reason": "Term type not provided"} + {"status": "error", "reason": "special character not allowed", "message": "Please use a regex or remove all special characters"} + {"status": "error", "reason": "Incorrect type"} +``` +**HTTP Status Code** : 409 + +```json + {"status": "error", "reason": "Term already tracked"} +``` + + + +### Delete term tracker: `api/v1/delete/tracker` + +#### Description +Delete a tracker + +**Method** : `DELETE` + +#### Parameters +- `uuid` + - tracked term uuid + - *uuid4* + - mandatory + +#### JSON response +- `uuid` + - deleted uuid + - *uuid4* + +#### Example +``` +curl https://127.0.0.1:7000/api/v1/delete/tracker --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "uuid": "6a16b06e-38e5-41e1-904d-3960610647e8" + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` + +```json + { + "uuid": "6a16b06e-38e5-41e1-904d-3960610647e8" + } +``` + +#### Expected Fail Response +**HTTP Status Code** : `400` + +```json + {"status": "error", "reason": "Invalid uuid"} + +``` + +**HTTP Status Code** : `404` + +```json + ({"status": "error", "reason": "Unknown uuid"} + +``` + + +### Delete term tracker: `api/v1/get/tracker/item` + +#### Description +Get tracked items by date-range + +**Method** : `POST` + +#### Parameters +- `uuid` + - tracked term uuid + - *uuid4* + - mandatory +- `date_from` + - date from + - *str - YYMMDD* + - default: last tracked items date +- `date_to` + - date to + - *str - YYMMDD* + - default: `None` + +#### JSON response +- `uuid` + - term uuid + - *uuid4* +- `date_from` + - date from + - *str - YYMMDD* +- `date_to` + - date to + - *str - YYMMDD* +- `items` + - list of item id + - *list* + +#### Example +``` +curl https://127.0.0.1:7000/api/v1/get/tracker/item --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "uuid": "6a16b06e-38e5-41e1-904d-3960610647e8", + "date_from": "20190823", + "date_to": "20190829", + "items": [ + { + "id": "submitted/2019/08/25/4f929998-3921-4be3-b448-be3bf1722d6b.gz", + "date": 20190825, + "tags": [ + "infoleak:automatic-detection=\"credential\"", + "mytags", + "othertags", + ] + } + ] + } +``` + +**HTTP Status Code** : `400` + +```json + {"status": "error", "reason": "Invalid uuid"} + +``` + +**HTTP Status Code** : `404` + +```json + ({"status": "error", "reason": "Unknown uuid"} + +``` + + + +## Domain + + +### Get min domain metadata: `api/v1/get/crawled/domain/list` + +#### Description +Get crawled domain by date-range and status (default status = *UP*) + +**Method** : `POST` + +#### Parameters +- `domain_type` + - domain type: *onion* or *regular* + - *str* + - default: *regular* +- `date_from` + - date from + - *str - YYYYMMDD* + - mandatory +- `date_to` + - date to + - *str - YYYYMMDD* + - mandatory + +#### JSON response +- `domain_type` + - domain type: *onion* or *regular* + - *str* +- `date_from` + - date from + - *str - YYYYMMDD* +- `date_to` + - date to + - *str - YYYYMMDD* +- `domains` + - list of domains + - *list - list of domains* + +#### Example +``` +curl https://127.0.0.1:7000/api/v1/get/crawled/domain/list --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "date_from": "20191001", + "date_to": "20191222", + "domain_type": "onion" + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` + +```json + { + "date_from": "20191001", + "date_to": "20191222", + "domain_status": "UP", + "domain_type": "onion", + "domains": [ + "2222222222222222.onion" + ] + } +``` + + + + +### Get min domain metadata: `api/v1/get/domain/status/minimal` + +#### Description +Get min domain metadata + +**Method** : `POST` + +#### Parameters +- `domain` + - domain name + - *str* + - mandatory + +#### JSON response +- `domain` + - domain + - *str* +- `first_seen` + - domain first up time + - *epoch* +- `last_seen` + - domain last up time + - *epoch* + +#### Example +``` +curl https://127.0.0.1:7000/api/v1/get/domain/status/minimal --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "domain": "2222222222222222.onion", + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` + +```json + { + "domain": "2222222222222222.onion", + "first_seen": 1571314000, + "last_seen": 1571314000 + } +``` + +**HTTP Status Code** : `404` + +```json + ({"status": "error", "reason": "Domain not found"} + +``` + + + +## Import management + + + +### Import item (currently: text only): `api/v1/import/item` + +#### Description +Allows users to import new items. asynchronous function. + +**Method** : `POST` + +#### Parameters +- `type` + - import type + - *str* + - default: `text` +- `text` + - text to import + - *str* + - mandatory if type = text +- `default_tags` + - add default import tag + - *boolean* + - default: True +- `tags` + - list of tags + - *list* + - default: `[]` +- `galaxy` + - list of galaxy + - *list* + - default: `[]` + +#### JSON response +- `uuid` + - import uuid + - *uuid4* + +#### Example +``` +curl https://127.0.0.1:7000/api/v1/import/item --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "type": "text", + "tags": [ + "infoleak:analyst-detection=\"private-key\"" + ], + "text": "text to import" + } +``` + +#### Expected Success Response +**HTTP Status Code** : `200` + +```json + { + "uuid": "0c3d7b34-936e-4f01-9cdf-2070184b6016" + } +``` + +#### Expected Fail Response +**HTTP Status Code** : `400` + +```json + {"status": "error", "reason": "Malformed JSON"} + {"status": "error", "reason": "No text supplied"} + {"status": "error", "reason": "Tags or Galaxy not enabled"} + {"status": "error", "reason": "Size exceeds default"} +``` + + + + + +### GET Import item info: `api/v1/get/import/item/` + +#### Description + +Get import status and all items imported by uuid + +**Method** : `POST` + +#### Parameters + +- `uuid` + - import uuid + - *uuid4* + - mandatory + +#### JSON response + +- `status` + - import status + - *str* + - values: `in queue`, `in progress`, `imported` +- `items` + - list of imported items id + - *list* + - The full list of imported items is not complete until `status` = `"imported"` + +#### Example + +``` +curl -k https://127.0.0.1:7000/api/v1/get/import/item --header "Authorization: iHc1_ChZxj1aXmiFiF1mkxxQkzawwriEaZpPqyTQj " -H "Content-Type: application/json" --data @input.json -X POST +``` + +#### input.json Example +```json + { + "uuid": "0c3d7b34-936e-4f01-9cdf-2070184b6016" + } +``` + +#### Expected Success Response + +**HTTP Status Code** : `200` + +```json + { + "items": [ + "submitted/2019/07/26/b20a69f1-99ad-4cb3-b212-7ce24b763b50.gz" + ], + "status": "imported" + } +``` + +#### Expected Fail Response + +**HTTP Status Code** : `400` + +```json + {"status": "error", "reason": "Invalid uuid"} + {"status": "error", "reason": "Unknown uuid"} +``` + + + + +# FUTURE endpoints + +
+Endpoints + +### Submit a domain to crawl TODO +##### ``api/add/crawler/task`` POST + +### Create a term/set/regex/yara tracker +##### ``api/add/tracker/`` POST + +### Get tracker +##### ``api/get/tracker`` POST + +----- + + + +### Get domain tags +##### ``api/get/domain/tags/`` POST + +### Get domain history +##### ``api/get/domain/history/`` POST + +----- + +### Get decoded item metadata +### Check if a decoded item exists (via sha1) +##### ``api/get/decoded/metadata/`` POST + +----- + + +----- +##### ``api/get/cryptocurrency`` POST + +### Check if a cryptocurrency address (bitcoin, ..) exists +##### ``api/get/cryptocurrency/`` POST + +### Get cryptocurrency address metadata +##### ``api/get/cryptocurrency/metadata/`` POST + +----- + +### Object correlation (1 depth) +##### ``api/get/correlation/`` POST + +### Create MISP event from object +##### ``api/export/misp`` POST + +
+ +----- +