CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
1,192 Commits
7 Branches
44 Tags
110 MiB
Commit Graph

1192 Commits (2589fc2161a78004475b97c83f8f5db2bf844f84)

Author SHA1 Message Date
Terrtia 2589fc2161
chg: [merge] merge master into update branche 2019-04-10 15:43:15 +02:00
Terrtia 870fa1981c
fix: [update_thirdparty] detect and use virtualenv 2019-04-02 09:50:52 +02:00
Thirion Aurélien c87f0ffcb1
Merge pull request #336 from 5thColumn/docker 
Add indexdir volume to docker-compose
 2019-03-14 09:35:19 +01:00
Jason Hedden 9279f56048 Add indexdir volume to docker-compose 
Make the indexdir data persistent and shared with Flask for searching pastes.
 2019-03-13 17:05:54 -05:00
Alexandre Dulaunoy 85d2ea578c
Merge pull request #335 from 5thColumn/docker 
Add docker-compose support
 2019-03-13 21:38:12 +01:00
Jason Hedden 23fb91d326 Add docker-compose support 
This commit separates the AIL Framework services into individual Docker
containers that can be managed with docker-compose. It's intended to ease the
installation, development and troubleshooting procedures for all-in-one
deployments.

No changes to architecture or base code are required in this commit.

Changes to existing files:
.gitignore
  * Ignore pystemon archive directory

Dockerfile:
  * Add AIL shell environment variables
  * Install pystemon and crawler pip requirements into AIL virtual environment

Docker-compose notes:
  * All containers are participating in a service network `network_mode:
  service:flask`.  This allows the containers to share the same IP namespace
  to accommodate hard-coded localhost entries.
  * By default persistent data is saved to the following directories in the
  local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS,
  pystemon/archives

This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment
would look like:
```
git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg
	(optionally enable activate_crawler in config.cfg)
docker-compose build
docker-compose up -d
```
 2019-03-11 19:46:34 -05:00
Alexandre Dulaunoy d54c4e2d86
Merge pull request #334 from 5thColumn/pybgpranking 
Add pybgpranking to install deps
 2019-03-11 07:46:46 +01:00
Jason Hedden c825ea6b2b Add pybgpranking to install deps 
The python module DomainClassifier requires pybgpranking which is not
available in the community PyPI repository. This commit installs pybgpranking
from git using the same version required by DomainClassifier https://github.com/adulau/DomainClassifier

Fixes the following installation error when using installing_deps.sh:
```
Could not find a version that satisfies the requirement pybgpranking (from DomainClassifier->-r pip3_packages_requirement.txt (line 50)) (from versions: )
No matching distribution found for pybgpranking (from DomainClassifier->-r pip3_packages_requirement.txt (line 50))
```
 2019-03-10 22:40:00 -05:00
Thirion Aurélien 77e4078f3d
Merge pull request #333 from 5thColumn/fix_modulesinformationv2 
Fix current_selected amount and action vars
 2019-02-25 08:53:03 +01:00
Jason Hedden 9eae25bdf5 Fix current_selected amount and action vars 
This commit fixes issue #268

```
  File "./bin/ModulesInformationV2.py", line 319, in _setValue
    action = current_selected_action + " " + str(current_selected_amount) + "x"
```

`current_selected_action` was defined twice. The second occurance should be `current_selected_amount`.

This also fixes the generic exception from failing when `current_selected_value` is not defined.
 2019-02-22 12:39:27 -06:00
Terrtia 3d36ddbc1e
fix: [CVE] fix stat regex 2019-02-21 10:51:05 +01:00
Terrtia 1114aa2ffd
chg: [CVE] add stat script 2019-02-21 09:58:10 +01:00
Alexandre Dulaunoy 0dc27c37e0
chg: [doc] badge release updated 2019-02-17 09:31:59 +01:00
Terrtia 709d5487b8
fix: [Onion] filter too many subdomain 2019-02-13 16:53:38 +01:00
Terrtia b87707e8bc
fix: [Crawler] typo 2019-02-12 15:54:42 +01:00
Terrtia 37276e52a3
fix: [Crawler] typo 2019-02-12 15:53:40 +01:00
Terrtia 7cb03fc769
fix: [Crawler] typo 2019-02-12 15:51:19 +01:00
Terrtia 7a4989ce10
fix: [Global Crawler] max filename size 2019-02-12 15:45:58 +01:00
Terrtia 423c7b1455
fix: [UI] fix flot js install 2019-02-04 14:56:54 +01:00
Thirion Aurélien 44c513dcbb
chg: [Crawler] add onion to blacklist 2019-01-31 16:56:45 +01:00
Alexandre Dulaunoy be8f73f2dd
chg: [doc] how to cite AIL in academic publications 2019-01-31 13:24:41 +01:00
Thirion Aurélien a26b85ad5e
Merge pull request #319 from xme/master 
fix: [Onion] Timeout error in Onion.py
 2019-01-31 11:06:07 +01:00
Xavier Mertens 43d08d0d3e Cleanup 2019-01-30 22:36:10 +01:00
Xavier Mertens 1be91b88bf Fix: reset alarm timeout 2019-01-30 22:30:07 +01:00
Xavier Mertens 0372a78704 Merge remote-tracking branch 'upstream/master' 2019-01-30 22:27:44 +01:00
Xavier Mertens b099e2ae4a Update LAUNCH.sh 2019-01-30 22:27:33 +01:00
Terrtia 92d192238b
fix: [Crawler] change max page crawled 2019-01-29 17:04:45 +01:00
Terrtia 6c7086f4eb
fix: [Crawler] first_seen 2019-01-29 16:54:39 +01:00
Terrtia 655646b7c5
fix: [hiddenServices] catch empty data 2019-01-29 16:43:56 +01:00
Terrtia ad04cf7bb2
fix: [Onion] typo 2019-01-29 16:27:12 +01:00
Terrtia 88eaaeae93
chg: [Crawler] add priority queue, fix #263 2019-01-29 16:08:59 +01:00
Terrtia c1b34bd99c
fix: [Crawler] limit max crawled pages 2019-01-29 15:38:00 +01:00
Terrtia 64ffdd52e8
fix: [Onion] loop 2019-01-29 14:39:57 +01:00
Terrtia 7a80683f13
chg: [hiddenServices] crawlers status, dynamic update 2019-01-29 14:27:50 +01:00
Terrtia b9572e0085
fix: [Crawler] typo 2019-01-29 12:19:10 +01:00
Terrtia 2dc0eca4a9
fix: [Crawler] fix crawler cache info 2019-01-29 12:09:19 +01:00
Terrtia bb301a870c
fix: [Crawler] fix onion blacklist + add crawler info 2019-01-29 12:00:14 +01:00
Terrtia 1a1fda4c47
Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2019-01-29 09:46:38 +01:00
Terrtia 535f44044d
fix: [Onion] add timeout signal 2019-01-29 09:46:03 +01:00
Thirion Aurélien 2ee7964370
Merge pull request #318 from raggadhub/master 
[Redis] by default just listen on loopback interface [security hardening]
 2019-01-29 08:40:46 +01:00
raggadhub 2d8b88af89
6381 - by default just listen on loopback interface [security hardening] 
user will change explicitly the bind option if necessary
 2019-01-27 16:03:40 +01:00
raggadhub fa1edd6e0a
6380 - by default just listen on loopback interface [security hardening] 
user will change explicitly the bind option if necessary
 2019-01-27 16:03:00 +01:00
raggadhub ce53c8ea74
6379 - by default just listen on loopback interface [security hardening] 
user will change explicitly the bind option if necessary
 2019-01-27 16:02:11 +01:00
Thirion Aurélien f4ba21e492
blacklist onion debian manpages 2019-01-14 11:08:53 +01:00
Thirion Aurélien 199177aeea
Merge pull request #311 from raggadhub/patch-2 
fix path_to_send path
 2019-01-09 15:10:04 +01:00
raggadhub 74c726d5fa
fix path_to_send path 
Hi,

Missing slash between pastes_directory and paste in path_to_send = pastes_directory+paste 

path error example:
/opt/AIL-framework/PASTESarchive/pastebin.com_pro/2019/01/07/cAgqsY2U.gz
 2019-01-09 13:15:58 +01:00
Terrtia c4c3810037
fix: [hiddenServices] typo 2019-01-08 15:51:39 +01:00
Terrtia 46c1105483
chg: [hiddenServices] auto update last crawled domains 2019-01-08 15:48:40 +01:00
Terrtia b3b75ccbea
fix: [Crawler] Restart Splash on failure, limit unbound in memory cache (maxrss) 2019-01-04 15:51:08 +01:00
Terrtia 0358b0cf58
chg: [OVERVIEW] update db map 2018-12-24 11:45:49 +01:00