CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity
1,356 Commits
6 Branches
43 Tags
110 MiB
Commit Graph

1356 Commits (7ecd43db99209738fadc4959f0c7231844e55dd8)

Author SHA1 Message Date
Terrtia a43467a0f2
chg: [Update v1.5] update tracked terms path 2019-04-15 13:27:46 +02:00
Terrtia 0ce4489ed2
chg: [OVERVIEW] add doc termfred DB 2019-04-15 11:46:20 +02:00
Terrtia fc1a04336c
fix: [dynamic update v1.5] make sure updates are excuted in the correct order + fix nb_seen_hash dynamic update 2019-04-15 11:01:33 +02:00
Terrtia a3167a740a
fix: [Update DB] remove indexErrors 2019-04-12 17:32:17 +02:00
Terrtia bf92a2f22f
fix: [background update] avoid multiple update on the same key 2019-04-12 16:07:40 +02:00
Terrtia f2dc3bcbf4
fix: [Global] remove absolute path from item path 2019-04-12 15:13:36 +02:00
Terrtia 3a8631d44a
chg: [Update + Launcher] use background update + add new options in launcher 2019-04-12 15:06:24 +02:00
Terrtia 5c4db5f2f1
chg: [update] add update script Redis ARDB + add background update 2019-04-11 17:49:20 +02:00
Terrtia 1ed9468881
fix: [UI showpaste] min preview 2019-04-11 13:44:20 +02:00
Terrtia 3cc614a1ad
chg: [DB] update items tags metadata 2019-04-11 11:58:06 +02:00
Terrtia e83174327a
chg: [DB] remove absolute path 2019-04-10 17:47:40 +02:00
Terrtia d44acea04d
chg: [merge] merge update into advanced_crawler 2019-04-10 16:41:06 +02:00
Terrtia 2589fc2161
chg: [merge] merge master into update branche 2019-04-10 15:43:15 +02:00
Terrtia b4bee29a33
Merge branch 'master' into advanced_crawler 2019-04-10 15:37:42 +02:00
Terrtia 6165987ec3
chg: [DB Migration] add hash dynamic update 2019-04-10 15:36:41 +02:00
Terrtia 0519b4a437
chg: [update DB] add migration script 2019-04-08 17:04:09 +02:00
Terrtia 870fa1981c
fix: [update_thirdparty] detect and use virtualenv 2019-04-02 09:50:52 +02:00
Terrtia f6d7d2ae16
chg: [Update] add debug 2019-04-02 09:46:04 +02:00
Thirion Aurélien 976b97bcd1
Merge pull request #338 from CIRCL/pre_update 
Merge Pre update
 2019-03-26 16:51:26 +01:00
Terrtia 044933aa33
chg: [Launch] remove old launchers 2019-03-26 16:47:10 +01:00
Terrtia cee2b2486f
chg: [Launch] remove old launchers 2019-03-26 16:45:01 +01:00
Terrtia 59664efe45
Merge branch 'master' into advanced_crawler 2019-03-26 16:03:42 +01:00
Terrtia 5165a5de2f
fix: [Crawler] fix index + redis history key 2019-03-22 17:14:27 +01:00
Terrtia f64c385343
chg: [Crawler] handle port: crawling + history 2019-03-22 16:48:07 +01:00
Terrtia f4cdddbc7f
chg: [UI + item submit] remove browse paste from UI + fix pasteSubmit tagging 2019-03-21 11:50:53 +01:00
Terrtia 51eb951a47
fix: [UI terms] update tags endpoint 2019-03-21 11:34:21 +01:00
Terrtia c2bc72c72e
chg: [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items 2019-03-21 11:11:53 +01:00
Terrtia ce6475e96e
chg: [tags] remove/add tags by daterange + improve item loading time 2019-03-20 15:57:55 +01:00
Terrtia 8b5266706e
fix: [Tags] cast nb_max_page to int 2019-03-20 13:33:12 +01:00
Terrtia a9e86231ad
chg: [Tags] add backend, daterange selector 2019-03-20 13:25:02 +01:00
Terrtia 65e6eae00d
chg: [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition 2019-03-19 13:34:41 +01:00
Terrtia e9bac098c9
fix: [UI Tags] paste preview 2019-03-15 11:16:44 +01:00
Terrtia 23d9cb2265
chg: [UI Tags search] bootstrap 4 migration 2019-03-14 17:04:55 +01:00
Thirion Aurélien c87f0ffcb1
Merge pull request #336 from 5thColumn/docker 
Add indexdir volume to docker-compose
 2019-03-14 09:35:19 +01:00
Jason Hedden 9279f56048 Add indexdir volume to docker-compose 
Make the indexdir data persistent and shared with Flask for searching pastes.
 2019-03-13 17:05:54 -05:00
Alexandre Dulaunoy 85d2ea578c
Merge pull request #335 from 5thColumn/docker 
Add docker-compose support
 2019-03-13 21:38:12 +01:00
Terrtia 9e4813ca4b
chg: [UI Tags] start b4 migration + add date range selector 2019-03-12 13:37:15 +01:00
Jason Hedden 23fb91d326 Add docker-compose support 
This commit separates the AIL Framework services into individual Docker
containers that can be managed with docker-compose. It's intended to ease the
installation, development and troubleshooting procedures for all-in-one
deployments.

No changes to architecture or base code are required in this commit.

Changes to existing files:
.gitignore
  * Ignore pystemon archive directory

Dockerfile:
  * Add AIL shell environment variables
  * Install pystemon and crawler pip requirements into AIL virtual environment

Docker-compose notes:
  * All containers are participating in a service network `network_mode:
  service:flask`.  This allows the containers to share the same IP namespace
  to accommodate hard-coded localhost entries.
  * By default persistent data is saved to the following directories in the
  local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS,
  pystemon/archives

This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment
would look like:
```
git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg
	(optionally enable activate_crawler in config.cfg)
docker-compose build
docker-compose up -d
```
 2019-03-11 19:46:34 -05:00
Alexandre Dulaunoy d54c4e2d86
Merge pull request #334 from 5thColumn/pybgpranking 
Add pybgpranking to install deps
 2019-03-11 07:46:46 +01:00
Jason Hedden c825ea6b2b Add pybgpranking to install deps 
The python module DomainClassifier requires pybgpranking which is not
available in the community PyPI repository. This commit installs pybgpranking
from git using the same version required by DomainClassifier https://github.com/adulau/DomainClassifier

Fixes the following installation error when using installing_deps.sh:
```
Could not find a version that satisfies the requirement pybgpranking (from DomainClassifier->-r pip3_packages_requirement.txt (line 50)) (from versions: )
No matching distribution found for pybgpranking (from DomainClassifier->-r pip3_packages_requirement.txt (line 50))
```
 2019-03-10 22:40:00 -05:00
Terrtia d3b3942ad7
chg: [Flask UI] refractor crawler frontend + show crawled domain 2019-03-06 15:05:58 +01:00
Terrtia 6f5f880b7d
chg: [Crawler UI] add basic tooltips 2019-03-01 11:40:43 +01:00
Terrtia 93c7db86e4
chg: [Crawler UI Flask] refractor functions, templates 2019-02-28 14:56:14 +01:00
Terrtia 460f70fee6
chg: [css] add basic toggle sidebar 2019-02-27 16:58:17 +01:00
Terrtia aaf4712877
fix: [UI crawler manual] fix min input 2019-02-26 15:39:52 +01:00
Terrtia c0d72e7d2a
chg: [Crawler UI] Crawler major refractor (end) + basic UI for manual crawler 2019-02-26 14:50:48 +01:00
Terrtia 7b32d7f34e
chg: [Crawler] major refractor 2019-02-25 16:38:50 +01:00
Thirion Aurélien 77e4078f3d
Merge pull request #333 from 5thColumn/fix_modulesinformationv2 
Fix current_selected amount and action vars
 2019-02-25 08:53:03 +01:00
Jason Hedden 9eae25bdf5 Fix current_selected amount and action vars 
This commit fixes issue #268

```
  File "./bin/ModulesInformationV2.py", line 319, in _setValue
    action = current_selected_action + " " + str(current_selected_amount) + "x"
```

`current_selected_action` was defined twice. The second occurance should be `current_selected_amount`.

This also fixes the generic exception from failing when `current_selected_value` is not defined.
 2019-02-22 12:39:27 -06:00
Terrtia 60f7645ac1
chg: [Crawler] refractor 2019-02-22 17:00:24 +01:00