mirror of https://github.com/CIRCL/AIL-framework
Terrtia
commit
175963e371
65
HOWTO.md
65
HOWTO.md
|
|
@ -78,21 +78,6 @@ If you want to add a new processing or analysis module in AIL, follow these simp
|
|||
2. Use [./bin/template.py](./bin/template.py) as a sample module and create a new file in bin/ with the module name used in the modules.cfg configuration.
|
||||
|
||||
|
||||
How to create a new webpage
|
||||
---------------------------
|
||||
|
||||
If you want to add a new webpage for a module in AIL, follow these simple steps:
|
||||
|
||||
1. Launch [./var/www/create_new_web_module.py](./var/www/create_new_web_module.py) and enter the name to use for your webpage (Usually, your newly created python module).
|
||||
|
||||
2. A template and flask skeleton has been created for your new webpage in [./var/www/modules/](./var/www/modules/)
|
||||
|
||||
3. Edit the created html files under the template folder as well as the Flask_* python script so that they fit your needs.
|
||||
|
||||
4. You can change the order of your module in the top navigation header in the file [./var/www/templates/header_base.html](./var/www/templates/header_base.html)
|
||||
|
||||
5. You can ignore module, and so, not display them in the top navigation header by adding the module name in the file [./var/www/templates/ignored_modules.txt](./var/www/templates/ignored_modules.txt)
|
||||
|
||||
How to contribute a module
|
||||
--------------------------
|
||||
|
||||
|
|
@ -104,30 +89,6 @@ To contribute your module, feel free to pull your contribution.
|
|||
Additional information
|
||||
======================
|
||||
|
||||
Manage modules: ModulesInformationV2.py
|
||||
---------------------------------------
|
||||
|
||||
You can do a lots of things easily with the [./bin/ModulesInformationV2](./bin/ModulesInformationV2) script:
|
||||
|
||||
- Monitor the health of other modules
|
||||
- Monitor the ressources comsumption of other modules
|
||||
- Start one or more modules
|
||||
- Kill running modules
|
||||
- Restart automatically stuck modules
|
||||
- Show the paste currently processed by a module
|
||||
|
||||
### Navigation
|
||||
|
||||
You can navigate into the interface by using arrow keys. In order to perform an action on a selected module, you can either press <ENTER> or <SPACE> to show the dialog box.
|
||||
|
||||
To change list, you can press the <TAB> key.
|
||||
|
||||
Also, you can quickly stop or start modules by clicking on the ``<K>`` or ``<S>`` symbol respectively. These are located in the _Action_ column.
|
||||
|
||||
Finally, you can quit this program by pressing either ``<q>`` or ``<C-c>``.
|
||||
|
||||
|
||||
|
||||
Crawler
|
||||
---------------------
|
||||
|
||||
|
|
@ -145,6 +106,7 @@ In AIL, you can crawl websites and Tor hidden services. Don't forget to review t
|
|||
1. Lacus URL:
|
||||
In the webinterface, go to ``Crawlers>Settings`` and click on the Edit button
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
@ -156,9 +118,26 @@ Choose the number of crawlers you want to launch
|
|||
|
||||
|
||||
|
||||
Kvrocks Migration
|
||||
---------------------
|
||||
**Important Note:
|
||||
We are currently working on a [migration script](https://github.com/ail-project/ail-framework/blob/master/bin/DB_KVROCKS_MIGRATION.py) to facilitate the migration to Kvrocks.
|
||||
Once this script is ready, AIL version 5.0 will be released.**
|
||||
|
||||
#### Old updates
|
||||
Please note that the current version of this migration script only supports migrating the database on the same server.
|
||||
(If you plan to migrate to another server, we will provide additional instructions in this section once the migration script is completed)
|
||||
|
||||
##### Python 3 Upgrade
|
||||
|
||||
To upgrade from an existing AIL installation, you have to launch [python3_upgrade.sh](./python3_upgrade.sh), this script will delete and create a new virtual environment. The script **will upgrade the packages but won't keep your previous data** (neverthless the data is copied into a directory called `old`). If you install from scratch, you don't require to launch the [python3_upgrade.sh](./python3_upgrade.sh).
|
||||
To migrate your database to Kvrocks:
|
||||
1. Launch ARDB and Kvrocks
|
||||
2. Pull from remote
|
||||
```
|
||||
git checkout master
|
||||
git pull
|
||||
```
|
||||
3. Launch the migration script:
|
||||
```
|
||||
git checkout master
|
||||
git pull
|
||||
cd bin/
|
||||
./DB_KVROCKS_MIGRATION.py
|
||||
```
|
||||
|
|
|
|||
|
|
@ -37,7 +37,8 @@ def auto_update_enabled(cfg):
|
|||
|
||||
# check if files are modify locally
|
||||
def check_if_files_modified():
|
||||
process = subprocess.run(['git', 'ls-files', '-m'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
# return True
|
||||
process = subprocess.run(['git', 'ls-files' ,'-m'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
if process.returncode == 0:
|
||||
modified_files = process.stdout
|
||||
if modified_files:
|
||||
|
|
@ -61,6 +62,7 @@ def check_if_files_modified():
|
|||
sys.exit(1)
|
||||
|
||||
def repo_is_fork():
|
||||
# return False
|
||||
print('Check if this repository is a fork:')
|
||||
process = subprocess.run(['git', 'remote', '-v'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
|
||||
|
|
@ -234,9 +236,9 @@ def get_git_upper_tags_remote(current_tag, is_fork):
|
|||
except ValueError:
|
||||
continue
|
||||
|
||||
if float(current_tag) < 4.2:
|
||||
if float(current_tag) < 5.0:
|
||||
# add tag with last commit
|
||||
if float(current_tag_val) <= float(tag_val) < float(4.2):
|
||||
if float(current_tag_val) <= float(tag_val) < float(5.0):
|
||||
dict_tags_commit[tag] = commit
|
||||
else:
|
||||
# add tag with last commit
|
||||
|
|
@ -270,13 +272,28 @@ def update_ail(current_tag, list_upper_tags_remote, current_version_path, is_for
|
|||
|
||||
update_submodules()
|
||||
|
||||
print('{}git pull:{}'.format(TERMINAL_YELLOW, TERMINAL_DEFAULT))
|
||||
process = subprocess.run(['git', 'pull'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
temp_current_tag = current_tag.replace('v', '')
|
||||
if temp_current_tag.count('.') > 1:
|
||||
temp_current_tag = temp_current_tag.rsplit('.', 1)
|
||||
temp_current_tag = ''.join(temp_current_tag)
|
||||
|
||||
if process.returncode == 0:
|
||||
output = process.stdout.decode()
|
||||
print(output)
|
||||
if float(temp_current_tag) < 5.0:
|
||||
roll_back_update('2c65194b94dab95df9b8da19c88d65239f398355')
|
||||
pulled = True
|
||||
else:
|
||||
print('{}git pull:{}'.format(TERMINAL_YELLOW, TERMINAL_DEFAULT))
|
||||
process = subprocess.run(['git', 'pull'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
if process.returncode == 0:
|
||||
output = process.stdout.decode()
|
||||
print(output)
|
||||
pulled = True
|
||||
else:
|
||||
print('{}{}{}'.format(TERMINAL_RED, process.stderr.decode(), TERMINAL_DEFAULT))
|
||||
aborting_update()
|
||||
pulled = False
|
||||
sys.exit(1)
|
||||
|
||||
if pulled:
|
||||
# CHECK IF UPDATER Update
|
||||
if float(os.stat(UPDATER_FILENAME).st_mtime) > UPDATER_LAST_MODIFICATION:
|
||||
# request updater relaunch
|
||||
|
|
@ -321,10 +338,7 @@ def update_ail(current_tag, list_upper_tags_remote, current_version_path, is_for
|
|||
print(f'{TERMINAL_YELLOW}**************** AIL Successfully Updated *****************{TERMINAL_DEFAULT}')
|
||||
print()
|
||||
sys.exit(0)
|
||||
else:
|
||||
print('{}{}{}'.format(TERMINAL_RED, process.stderr.decode(), TERMINAL_DEFAULT))
|
||||
aborting_update()
|
||||
sys.exit(1)
|
||||
|
||||
else:
|
||||
print('{}{}{}'.format(TERMINAL_RED, process.stderr.decode(), TERMINAL_DEFAULT))
|
||||
aborting_update()
|
||||
|
|
|
|||
|
|
@ -74,9 +74,7 @@
|
|||
{%endif%}
|
||||
|
||||
<!-- TODO: Add users messages -->
|
||||
<!-- {%if update_note%}
|
||||
{% include 'dashboard/update_modal.html' %}
|
||||
{%endif%} -->
|
||||
{# {% include 'dashboard/update_modal.html' %}#}
|
||||
|
||||
<div class="row my-2">
|
||||
|
||||
|
|
|
|||
|
|
@ -1,40 +1,49 @@
|
|||
<div class="modal fade" id="update_modal" tabindex="-1" role="dialog" aria-labelledby="exampleModalLabel" aria-hidden="true">
|
||||
<div class="modal-dialog modal-lg" role="document">
|
||||
<div class="modal-content">
|
||||
<div class="modal-header bg-secondary text-white">
|
||||
<h5 class="modal-title" id="exampleModalLabel">Update Note: v3.5 - Splash Manager</h5>
|
||||
<button type="button" class="close" data-dismiss="modal" aria-label="Close">
|
||||
<span aria-hidden="true">×</span>
|
||||
</button>
|
||||
</div>
|
||||
<div class="modal-body">
|
||||
<div class="alert alert-danger text-danger" role="alert">All Splash Crawlers have been removed from the core.</div>
|
||||
AIL is using a new Crawler manager to start and launch dockers and tor/web crawlers
|
||||
<div class="modal-dialog modal-lg" role="document">
|
||||
<div class="modal-content">
|
||||
<div class="modal-header bg-secondary text-white">
|
||||
<h5 class="modal-title" id="exampleModalLabel">Update Note: <b>v5.0 - Kvrocks Migration</b></h5>
|
||||
<button type="button" class="close" data-dismiss="modal" aria-label="Close">
|
||||
<span aria-hidden="true">×</span>
|
||||
</button>
|
||||
</div>
|
||||
<div class="modal-body">
|
||||
<div>
|
||||
In order to improve the performance and reliability of AIL, we have migrated our database from ARDB to <a href="https://github.com/apache/incubator-kvrocks">Kvrocks</a>,
|
||||
and the AIL crawler has been upgraded to <a href="https://github.com/ail-project/lacus">Lacus</a>.
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<ul class="list-group my-3">
|
||||
<li class="list-group-item active">Splash Manager Features:</li>
|
||||
<li class="list-group-item">Install and run Splash crawlers on another server</li>
|
||||
<li class="list-group-item">Handle proxies (Web and tor)</li>
|
||||
<li class="list-group-item">Launch/Kill Splash Dockers</li>
|
||||
<li class="list-group-item">Restart crawlers on crash</li>
|
||||
</ul>
|
||||
<div class="alert alert-danger text-danger" role="alert">Kvrocks DB Migration.</div>
|
||||
<div>
|
||||
As a result of this change, it is necessary to manually update your AIL installation.
|
||||
Instructions on how to perform the update can be found <a href="https://github.com/ail-project/ail-framework/blob/master/HOWTO.md#kvrocks-migration">here</a>.
|
||||
</div>
|
||||
|
||||
<div class="d-flex justify-content-center">
|
||||
<a class="btn btn-info" href="https://github.com/ail-project/ail-splash-manager" role="button">
|
||||
<i class="fab fa-github"></i> Install and Configure AIL-Splash-Manager
|
||||
</a>
|
||||
<br><br>
|
||||
|
||||
<div class="alert alert-danger text-danger" role="alert">AIL Crawlers Migration: Lacus</div>
|
||||
If you wish to crawl Tor or website domains, you need to install Lacus.
|
||||
|
||||
<div class="d-flex justify-content-center">
|
||||
<a class="btn btn-info" href="https://github.com/ail-project/lacus" role="button">
|
||||
<i class="fab fa-github"></i> Install and Configure Lacus
|
||||
</a>
|
||||
</div>
|
||||
How to configure the AIL crawler:
|
||||
<a href="https://github.com/ail-project/ail-framework/blob/master/HOWTO.md#crawler">HOWTO</a>
|
||||
<br><br><br>
|
||||
If you encounter any difficulties, you can open an <a href="https://github.com/ail-project/ail-framework/issues">issue</a>
|
||||
|
||||
</div>
|
||||
<div class="modal-footer">
|
||||
<button type="button" class="btn btn-secondary" data-dismiss="modal">Close</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<div class="modal-footer">
|
||||
<button type="button" class="btn btn-secondary" data-dismiss="modal">Close</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<script>
|
||||
$(window).on('load', function() {
|
||||
$('#update_modal').modal('show');
|
||||
|
|
|
|||
Loading…
Reference in New Issue