CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
1,396 Commits
6 Branches
40 Tags
107 MiB
Commit Graph

1396 Commits (d9279823d51ec211705a7c395c6e72b28034c781)

Author SHA1 Message Date
Terrtia e83174327a
chg: [DB] remove absolute path 2019-04-10 17:47:40 +02:00
Terrtia d44acea04d
chg: [merge] merge update into advanced_crawler 2019-04-10 16:41:06 +02:00
Terrtia 2589fc2161
chg: [merge] merge master into update branche 2019-04-10 15:43:15 +02:00
Terrtia b4bee29a33
Merge branch 'master' into advanced_crawler 2019-04-10 15:37:42 +02:00
Terrtia 6165987ec3
chg: [DB Migration] add hash dynamic update 2019-04-10 15:36:41 +02:00
Terrtia 0519b4a437
chg: [update DB] add migration script 2019-04-08 17:04:09 +02:00
Terrtia 870fa1981c
fix: [update_thirdparty] detect and use virtualenv 2019-04-02 09:50:52 +02:00
Terrtia f6d7d2ae16
chg: [Update] add debug 2019-04-02 09:46:04 +02:00
Thirion Aurélien 976b97bcd1
Merge pull request #338 from CIRCL/pre_update 
Merge Pre update
 2019-03-26 16:51:26 +01:00
Terrtia 044933aa33
chg: [Launch] remove old launchers 2019-03-26 16:47:10 +01:00
Terrtia cee2b2486f
chg: [Launch] remove old launchers 2019-03-26 16:45:01 +01:00
Terrtia 59664efe45
Merge branch 'master' into advanced_crawler 2019-03-26 16:03:42 +01:00
Terrtia 5165a5de2f
fix: [Crawler] fix index + redis history key 2019-03-22 17:14:27 +01:00
Terrtia f64c385343
chg: [Crawler] handle port: crawling + history 2019-03-22 16:48:07 +01:00
Terrtia f4cdddbc7f
chg: [UI + item submit] remove browse paste from UI + fix pasteSubmit tagging 2019-03-21 11:50:53 +01:00
Terrtia 51eb951a47
fix: [UI terms] update tags endpoint 2019-03-21 11:34:21 +01:00
Terrtia c2bc72c72e
chg: [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items 2019-03-21 11:11:53 +01:00
Terrtia ce6475e96e
chg: [tags] remove/add tags by daterange + improve item loading time 2019-03-20 15:57:55 +01:00
Terrtia 8b5266706e
fix: [Tags] cast nb_max_page to int 2019-03-20 13:33:12 +01:00
Terrtia a9e86231ad
chg: [Tags] add backend, daterange selector 2019-03-20 13:25:02 +01:00
Terrtia 65e6eae00d
chg: [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition 2019-03-19 13:34:41 +01:00
Terrtia e9bac098c9
fix: [UI Tags] paste preview 2019-03-15 11:16:44 +01:00
Terrtia 23d9cb2265
chg: [UI Tags search] bootstrap 4 migration 2019-03-14 17:04:55 +01:00
Thirion Aurélien c87f0ffcb1
Merge pull request #336 from 5thColumn/docker 
Add indexdir volume to docker-compose
 2019-03-14 09:35:19 +01:00
Jason Hedden 9279f56048 Add indexdir volume to docker-compose 
Make the indexdir data persistent and shared with Flask for searching pastes.
 2019-03-13 17:05:54 -05:00
Alexandre Dulaunoy 85d2ea578c
Merge pull request #335 from 5thColumn/docker 
Add docker-compose support
 2019-03-13 21:38:12 +01:00
Terrtia 9e4813ca4b
chg: [UI Tags] start b4 migration + add date range selector 2019-03-12 13:37:15 +01:00
Jason Hedden 23fb91d326 Add docker-compose support 
This commit separates the AIL Framework services into individual Docker
containers that can be managed with docker-compose. It's intended to ease the
installation, development and troubleshooting procedures for all-in-one
deployments.

No changes to architecture or base code are required in this commit.

Changes to existing files:
.gitignore
  * Ignore pystemon archive directory

Dockerfile:
  * Add AIL shell environment variables
  * Install pystemon and crawler pip requirements into AIL virtual environment

Docker-compose notes:
  * All containers are participating in a service network `network_mode:
  service:flask`.  This allows the containers to share the same IP namespace
  to accommodate hard-coded localhost entries.
  * By default persistent data is saved to the following directories in the
  local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS,
  pystemon/archives

This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment
would look like:
```
git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg
	(optionally enable activate_crawler in config.cfg)
docker-compose build
docker-compose up -d
```
 2019-03-11 19:46:34 -05:00
Alexandre Dulaunoy d54c4e2d86
Merge pull request #334 from 5thColumn/pybgpranking 
Add pybgpranking to install deps
 2019-03-11 07:46:46 +01:00
Jason Hedden c825ea6b2b Add pybgpranking to install deps 
The python module DomainClassifier requires pybgpranking which is not
available in the community PyPI repository. This commit installs pybgpranking
from git using the same version required by DomainClassifier https://github.com/adulau/DomainClassifier

Fixes the following installation error when using installing_deps.sh:
```
Could not find a version that satisfies the requirement pybgpranking (from DomainClassifier->-r pip3_packages_requirement.txt (line 50)) (from versions: )
No matching distribution found for pybgpranking (from DomainClassifier->-r pip3_packages_requirement.txt (line 50))
```
 2019-03-10 22:40:00 -05:00
Terrtia d3b3942ad7
chg: [Flask UI] refractor crawler frontend + show crawled domain 2019-03-06 15:05:58 +01:00
Terrtia 6f5f880b7d
chg: [Crawler UI] add basic tooltips 2019-03-01 11:40:43 +01:00
Terrtia 93c7db86e4
chg: [Crawler UI Flask] refractor functions, templates 2019-02-28 14:56:14 +01:00
Terrtia 460f70fee6
chg: [css] add basic toggle sidebar 2019-02-27 16:58:17 +01:00
Terrtia aaf4712877
fix: [UI crawler manual] fix min input 2019-02-26 15:39:52 +01:00
Terrtia c0d72e7d2a
chg: [Crawler UI] Crawler major refractor (end) + basic UI for manual crawler 2019-02-26 14:50:48 +01:00
Terrtia 7b32d7f34e
chg: [Crawler] major refractor 2019-02-25 16:38:50 +01:00
Thirion Aurélien 77e4078f3d
Merge pull request #333 from 5thColumn/fix_modulesinformationv2 
Fix current_selected amount and action vars
 2019-02-25 08:53:03 +01:00
Jason Hedden 9eae25bdf5 Fix current_selected amount and action vars 
This commit fixes issue #268

```
  File "./bin/ModulesInformationV2.py", line 319, in _setValue
    action = current_selected_action + " " + str(current_selected_amount) + "x"
```

`current_selected_action` was defined twice. The second occurance should be `current_selected_amount`.

This also fixes the generic exception from failing when `current_selected_value` is not defined.
 2019-02-22 12:39:27 -06:00
Terrtia 60f7645ac1
chg: [Crawler] refractor 2019-02-22 17:00:24 +01:00
Terrtia 3d36ddbc1e
fix: [CVE] fix stat regex 2019-02-21 10:51:05 +01:00
Terrtia 1114aa2ffd
chg: [CVE] add stat script 2019-02-21 09:58:10 +01:00
Terrtia e5dca268a8
chg: [Crawler] refractor 2019-02-21 09:54:43 +01:00
Terrtia 0832784f7a
chg: [thirdparty_update] get b4 datatable 2019-02-19 11:56:08 +01:00
Terrtia da78d0552d
chg: [Crawler UI Tags] add tag by day + add crawler status + UI onion blacklist 2019-02-19 11:41:45 +01:00
Terrtia 32acbef4e9
chg: [Overview] add db map tag 2019-02-18 15:24:47 +01:00
Terrtia 6a8fa3d5d1
chg: [Tags] tags by day 2019-02-18 14:33:34 +01:00
Alexandre Dulaunoy 0dc27c37e0
chg: [doc] badge release updated 2019-02-17 09:31:59 +01:00
Terrtia 95d63daaf4
chg: [Crawler UI] fix css + bootstrap4 migration 2019-02-15 17:00:07 +01:00
Terrtia 709d5487b8
fix: [Onion] filter too many subdomain 2019-02-13 16:53:38 +01:00