CIRCL
/
AIL-framework
mirror of https://github.com/CIRCL/AIL-framework
2
1
Fork 0
Code Issues Releases Wiki Activity

chg: [trackers] filter trackers list by type + minor fix

pull/389/head
Terrtia 2019-09-11 15:33:04 +02:00
parent ad65ed7c8e
commit 0ced25f3be
No known key found for this signature in database
GPG Key ID: 1E1B1F50D84613D0
15 changed files with 518 additions and 336 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

48
OVERVIEW.md
View File

@ -109,63 +109,65 @@ Redis and ARDB overview
| **uuid**:ltags | **tag** | | **uuid**:ltags | **tag** |
| **uuid**:ltagsgalaxies | **tag** | | **uuid**:ltagsgalaxies | **tag** |
## DB2 - New TermFreq: ## DB2 - Leak Hunter:
##### Term Tracker metadata: ##### Tracker metadata:
| Hset - Key | Field | Value | | Hset - Key | Field | Value |
| ------ | ------ | ------ | | ------ | ------ | ------ |
| tracked_term:**uuid** | tracked | **tacked word/set/regex** | | tracker:**uuid** | tracked | **tacked word/set/regex** |
| | type | **term/set/regex** | | | type | **word/set/regex** |
| | date | **date added** | | | date | **date added** |
| | user_id | **created by user_id** | | | user_id | **created by user_id** |
| | dashboard | **0/1 Display alert on dashboard** | | | dashboard | **0/1 Display alert on dashboard** |
| | level | **0/1 Tracker visibility** | | | level | **0/1 Tracker visibility** |
##### Term Tracked by user_id (visibility level: user only): ##### Tracker by user_id (visibility level: user only):
| Set - Key | Value | | Set - Key | Value |
| ------ | ------ | | ------ | ------ |
| user:tracked_term:**user_id** | **uuid - tracked term uuid** | | user:tracker:**user_id** | **uuid - tracker uuid** |
| user:tracker:**user_id**:**word/set/regex - tracker type** | **uuid - tracker uuid** |
##### Global Term Tracked (visibility level: all users): ##### Global Tracker (visibility level: all users):
| Set - Key | Value | | Set - Key | Value |
| ------ | ------ | | ------ | ------ |
| gobal:tracked_term | **uuid - tracked term uuid** | | gobal:tracker | **uuid - tracker uuid** |
| gobal:tracker:**word/set/regex - tracker type** | **uuid - tracker uuid** |
##### All Term Tracked by type: ##### All Tracker by type:
| Set - Key | Value | | Set - Key | Value |
| ------ | ------ | | ------ | ------ |
| all:tracked_term:**word/set/regex - term type** | **tracked term** | | all:tracker:**word/set/regex - tracker type** | **tracked item** |
| Set - Key | Value | | Set - Key | Value |
| ------ | ------ | | ------ | ------ |
| all:tracked_term_uuid:**term type**:**tracked term** | **uuid - tracked term uuid** | | all:tracker_uuid:**tracker type**:**tracked item** | **uuid - tracker uuid** |
##### All Term Tracked items: ##### All Tracked items:
| Set - Key | Value | | Set - Key | Value |
| ------ | ------ | | ------ | ------ |
| tracked_term:item:**uuid**:**date** | **item_id** | | tracker:item:**uuid**:**date** | **item_id** |
##### All Term Tracked tags: ##### All Tracked tags:
| Set - Key | Value | | Set - Key | Value |
| ------ | ------ | | ------ | ------ |
| tracked_term:tags:**uuid** | **tag** | | tracker:tags:**uuid** | **tag** |
##### All Term Tracked tags: ##### All Tracked mail:
| Set - Key | Value | | Set - Key | Value |
| ------ | ------ | | ------ | ------ |
| tracked_term:mail:**uuid** | **mail** | | tracker:mail:**uuid** | **mail** |
##### Refresh Tracked term: ##### Refresh Tracker:
| Key | Value | | Key | Value |
| ------ | ------ | | ------ | ------ |
| tracked_term:refresh:word | **last refreshed epoch** | | tracker:refresh:word | **last refreshed epoch** |
| tracked_term:refresh:set | - | | tracker:refresh:set | - |
| tracked_term:refresh:regex | - | | tracker:refresh:regex | - |
##### Zset Stat Tracked term: ##### Zset Stat Tracker:
| Key | Field | Value | | Key | Field | Value |
| ------ | ------ | ------ | | ------ | ------ | ------ |
| tracked_term:stat:**uuid** | **date** | **nb_seen** | | tracker:stat:**uuid** | **date** | **nb_seen** |
##### Stat token: ##### Stat token:
| Key | Field | Value | | Key | Field | Value |

4
bin/DbCleaner.py
View File

@ -1,7 +1,7 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
# -*-coding:UTF-8 -* # -*-coding:UTF-8 -*
""" """
The TermTracker Module The DbCleaner Module
=================== ===================
""" """
@ -36,8 +36,6 @@ if __name__ == "__main__":
publisher.channel = "Script" publisher.channel = "Script"
publisher.info("DbCleaner started") publisher.info("DbCleaner started")
config_section = 'TermTrackerMod'
# low priority # low priority
time.sleep(180) time.sleep(180)

10
bin/packages/Date.py
View File

@ -60,3 +60,13 @@ def get_date_range(num_day):
for i in range(0, num_day+1): for i in range(0, num_day+1):
date_list.append(date.substract_day(i)) date_list.append(date.substract_day(i))
return list(reversed(date_list)) return list(reversed(date_list))
def substract_date(date_from, date_to):
date_from = datetime.date(int(date_from[0:4]), int(date_from[4:6]), int(date_from[6:8]))
date_to = datetime.date(int(date_to[0:4]), int(date_to[4:6]), int(date_to[6:8]))
delta = date_to - date_from # timedelta
l_date = []
for i in range(delta.days + 1):
date = date_from + datetime.timedelta(i)
l_date.append( date.strftime('%Y%m%d') )
return l_date

10
bin/packages/Item.py
View File

@ -134,3 +134,13 @@ def get_item_pgp_name(item_id):
def get_item_pgp_mail(item_id): def get_item_pgp_mail(item_id):
return _get_item_correlation('pgpdump', 'mail', item_id) return _get_item_correlation('pgpdump', 'mail', item_id)
###
### GET Internal Module DESC
###
def get_item_list_desc(list_item_id):
desc_list = []
for item_id in list_item_id:
desc_list.append( {'id': item_id, 'date': get_item_date(item_id), 'tags': Tag.get_item_tags(item_id)} )
return desc_list

178
bin/packages/Term.py
View File

@ -18,6 +18,7 @@ sys.path.append(os.path.join(os.environ['AIL_FLASK'], 'modules'))
import Flask_config import Flask_config
import Date import Date
import Item
r_serv_term = Flask_config.r_serv_term r_serv_term = Flask_config.r_serv_term
email_regex = Flask_config.email_regex email_regex = Flask_config.email_regex
@ -47,11 +48,11 @@ def is_in_role(user_id, role):
def check_term_uuid_valid_access(term_uuid, user_id): def check_term_uuid_valid_access(term_uuid, user_id):
if not is_valid_uuid_v4(term_uuid): if not is_valid_uuid_v4(term_uuid):
return ({"status": "error", "reason": "Invalid uuid"}, 400) return ({"status": "error", "reason": "Invalid uuid"}, 400)
level = r_serv_term.hget('tracked_term:{}'.format(term_uuid), 'level') level = r_serv_term.hget('tracker:{}'.format(term_uuid), 'level')
if not level: if not level:
return ({"status": "error", "reason": "Unknown uuid"}, 404) return ({"status": "error", "reason": "Unknown uuid"}, 404)
if level == 0: if level == 0:
if r_serv_term.hget('tracked_term:{}'.format(term_uuid), 'user_id') != user_id: if r_serv_term.hget('tracker:{}'.format(term_uuid), 'user_id') != user_id:
if not is_in_role(user_id, 'admin'): if not is_in_role(user_id, 'admin'):
return ({"status": "error", "reason": "Unknown uuid"}, 404) return ({"status": "error", "reason": "Unknown uuid"}, 404)
return None return None
@ -91,10 +92,10 @@ def get_text_word_frequency(item_content, filtering=True):
# # TODO: create all tracked words # # TODO: create all tracked words
def get_tracked_words_list(): def get_tracked_words_list():
return list(r_serv_term.smembers('all:tracked_term:word')) return list(r_serv_term.smembers('all:tracker:word'))
def get_set_tracked_words_list(): def get_set_tracked_words_list():
set_list = r_serv_term.smembers('all:tracked_term:set') set_list = r_serv_term.smembers('all:tracker:set')
all_set_list = [] all_set_list = []
for elem in set_list: for elem in set_list:
res = elem.split(';') res = elem.split(';')
@ -104,7 +105,7 @@ def get_set_tracked_words_list():
return all_set_list return all_set_list
def get_regex_tracked_words_dict(): def get_regex_tracked_words_dict():
regex_list = r_serv_term.smembers('all:tracked_term:regex') regex_list = r_serv_term.smembers('all:tracker:regex')
dict_tracked_regex = {} dict_tracked_regex = {}
for regex in regex_list: for regex in regex_list:
dict_tracked_regex[regex] = re.compile(regex) dict_tracked_regex[regex] = re.compile(regex)
@ -113,24 +114,24 @@ def get_regex_tracked_words_dict():
def get_tracked_term_list_item(term_uuid, date_from, date_to): def get_tracked_term_list_item(term_uuid, date_from, date_to):
all_item_id = [] all_item_id = []
if date_from and date_to: if date_from and date_to:
for date in r_serv_term.zrangebyscore('tracked_term:stat:{}'.format(term_uuid), int(date_from), int(date_to)): for date in r_serv_term.zrangebyscore('tracker:stat:{}'.format(term_uuid), int(date_from), int(date_to)):
all_item_id = all_item_id + list(r_serv_term.smembers('tracked_term:item:{}:{}'.format(term_uuid, date))) all_item_id = all_item_id + list(r_serv_term.smembers('tracker:item:{}:{}'.format(term_uuid, date)))
return all_item_id return all_item_id
def is_term_tracked_in_global_level(term, term_type): def is_term_tracked_in_global_level(term, term_type):
res = r_serv_term.smembers('all:tracked_term_uuid:{}:{}'.format(term_type, term)) res = r_serv_term.smembers('all:tracker_uuid:{}:{}'.format(term_type, term))
if res: if res:
for elem_uuid in res: for elem_uuid in res:
if r_serv_term.hget('tracked_term:{}'.format(elem_uuid), 'level')=='1': if r_serv_term.hget('tracker:{}'.format(elem_uuid), 'level')=='1':
return True return True
return False return False
def is_term_tracked_in_user_level(term, term_type, user_id): def is_term_tracked_in_user_level(term, term_type, user_id):
res = r_serv_term.smembers('user:tracked_term:{}'.format(user_id)) res = r_serv_term.smembers('user:tracker:{}'.format(user_id))
if res: if res:
for elem_uuid in res: for elem_uuid in res:
if r_serv_term.hget('tracked_term:{}'.format(elem_uuid), 'tracked')== term: if r_serv_term.hget('tracker:{}'.format(elem_uuid), 'tracked')== term:
if r_serv_term.hget('tracked_term:{}'.format(elem_uuid), 'type')== term_type: if r_serv_term.hget('tracker:{}'.format(elem_uuid), 'type')== term_type:
return True return True
return False return False
@ -220,39 +221,42 @@ def add_tracked_term(term , term_type, user_id, level, tags, mails, dashboard=0)
term_uuid = str(uuid.uuid4()) term_uuid = str(uuid.uuid4())
# create metadata # create metadata
r_serv_term.hset('tracked_term:{}'.format(term_uuid), 'tracked',term) r_serv_term.hset('tracker:{}'.format(term_uuid), 'tracked',term)
r_serv_term.hset('tracked_term:{}'.format(term_uuid), 'type', term_type) r_serv_term.hset('tracker:{}'.format(term_uuid), 'type', term_type)
r_serv_term.hset('tracked_term:{}'.format(term_uuid), 'date', datetime.date.today().strftime("%Y%m%d")) r_serv_term.hset('tracker:{}'.format(term_uuid), 'date', datetime.date.today().strftime("%Y%m%d"))
r_serv_term.hset('tracked_term:{}'.format(term_uuid), 'user_id', user_id) r_serv_term.hset('tracker:{}'.format(term_uuid), 'user_id', user_id)
r_serv_term.hset('tracked_term:{}'.format(term_uuid), 'level', level) r_serv_term.hset('tracker:{}'.format(term_uuid), 'level', level)
r_serv_term.hset('tracked_term:{}'.format(term_uuid), 'dashboard', dashboard) r_serv_term.hset('tracker:{}'.format(term_uuid), 'dashboard', dashboard)
# create all term set # create all term set
r_serv_term.sadd('all:tracked_term:{}'.format(term_type), term) r_serv_term.sadd('all:tracker:{}'.format(term_type), term)
# create term - uuid map # create term - uuid map
r_serv_term.sadd('all:tracked_term_uuid:{}:{}'.format(term_type, term), term_uuid) r_serv_term.sadd('all:tracker_uuid:{}:{}'.format(term_type, term), term_uuid)
# add display level set # add display level set
if level == 0: # user only if level == 0: # user only
r_serv_term.sadd('user:tracked_term:{}'.format(user_id), term_uuid) r_serv_term.sadd('user:tracker:{}'.format(user_id), term_uuid)
r_serv_term.sadd('user:tracker:{}:{}'.format(user_id, term_type), term_uuid)
elif level == 1: # global elif level == 1: # global
r_serv_term.sadd('global:tracked_term', term_uuid) r_serv_term.sadd('global:tracker', term_uuid)
r_serv_term.sadd('global:tracker:{}'.format(term_type), term_uuid)
# create term tags list # create term tags list
for tag in tags: for tag in tags:
r_serv_term.sadd('tracked_term:tags:{}'.format(term_uuid), tag) r_serv_term.sadd('tracker:tags:{}'.format(term_uuid), tag)
# create term tags mail notification list # create term tags mail notification list
for mail in mails: for mail in mails:
r_serv_term.sadd('tracked_term:mail:{}'.format(term_uuid), mail) r_serv_term.sadd('tracker:mail:{}'.format(term_uuid), mail)
# toggle refresh module tracker list/set # toggle refresh module tracker list/set
r_serv_term.set('tracked_term:refresh:{}'.format(term_type), time.time()) r_serv_term.set('tracker:refresh:{}'.format(term_type), time.time())
return term_uuid return term_uuid
def parse_tracked_term_to_delete(dict_input, user_id): def parse_tracked_term_to_delete(dict_input, user_id):
term_uuid = dict_input.get("uuid", None)
res = check_term_uuid_valid_access(term_uuid, user_id) res = check_term_uuid_valid_access(term_uuid, user_id)
if res: if res:
return res return res
@ -261,66 +265,68 @@ def parse_tracked_term_to_delete(dict_input, user_id):
return ({"uuid": term_uuid}, 200) return ({"uuid": term_uuid}, 200)
def delete_term(term_uuid): def delete_term(term_uuid):
term = r_serv_term.hget('tracked_term:{}'.format(term_uuid), 'tracked') term = r_serv_term.hget('tracker:{}'.format(term_uuid), 'tracked')
term_type = r_serv_term.hget('tracked_term:{}'.format(term_uuid), 'type') term_type = r_serv_term.hget('tracker:{}'.format(term_uuid), 'type')
level = r_serv_term.hget('tracked_term:{}'.format(term_uuid), 'level') level = r_serv_term.hget('tracker:{}'.format(term_uuid), 'level')
r_serv_term.srem('all:tracked_term_uuid:{}:{}'.format(term_type, term), term_uuid) r_serv_term.srem('all:tracker_uuid:{}:{}'.format(term_type, term), term_uuid)
# Term not tracked by other users # Term not tracked by other users
if not r_serv_term.exists('all:tracked_term_uuid:{}:{}'.format(term_type, term)): if not r_serv_term.exists('all:tracker_uuid:{}:{}'.format(term_type, term)):
r_serv_term.srem('all:tracked_term:{}'.format(term_type), term) r_serv_term.srem('all:tracker:{}'.format(term_type), term)
# toggle refresh module tracker list/set # toggle refresh module tracker list/set
r_serv_term.set('tracked_term:refresh:{}'.format(term_type), time.time()) r_serv_term.set('tracker:refresh:{}'.format(term_type), time.time())
if level == 0: # user only if level == '0': # user only
user_id = term_type = r_serv_term.hget('tracked_term:{}'.format(term_uuid), 'user_id') user_id = term_type = r_serv_term.hget('tracker:{}'.format(term_uuid), 'user_id')
r_serv_term.srem('user:tracked_term:{}'.format(user_id), term_uuid) r_serv_term.srem('user:tracker:{}'.format(user_id), term_uuid)
elif level == 1: # global r_serv_term.srem('user:tracker:{}:{}'.format(user_id, term_type), term_uuid)
r_serv_term.srem('global:tracked_term', term_uuid) elif level == '1': # global
r_serv_term.srem('global:tracker', term_uuid)
r_serv_term.srem('global:tracker:{}'.format(term_type), term_uuid)
# delete metatadata # delete metatadata
r_serv_term.delete('tracked_term:{}'.format(term_uuid)) r_serv_term.delete('tracker:{}'.format(term_uuid))
# remove tags # remove tags
r_serv_term.delete('tracked_term:tags:{}'.format(term_uuid)) r_serv_term.delete('tracker:tags:{}'.format(term_uuid))
# remove mails # remove mails
r_serv_term.delete('tracked_term:mail:{}'.format(term_uuid)) r_serv_term.delete('tracker:mail:{}'.format(term_uuid))
# remove item set # remove item set
all_item_date = r_serv_term.zrange('tracked_term:stat:{}'.format(term_uuid), 0, -1) all_item_date = r_serv_term.zrange('tracker:stat:{}'.format(term_uuid), 0, -1)
for date in all_item_date: for date in all_item_date:
r_serv_term.delete('tracked_term:item:{}:{}'.format(term_uuid, date)) r_serv_term.delete('tracker:item:{}:{}'.format(term_uuid, date))
r_serv_term.delete('tracked_term:stat:{}'.format(term_uuid)) r_serv_term.delete('tracker:stat:{}'.format(term_uuid))
def replace_tracked_term_tags(term_uuid, tags): def replace_tracked_term_tags(term_uuid, tags):
r_serv_term.delete('tracked_term:tags:{}'.format(term_uuid)) r_serv_term.delete('tracker:tags:{}'.format(term_uuid))
for tag in tags: for tag in tags:
r_serv_term.sadd('tracked_term:tags:{}'.format(term_uuid), tag) r_serv_term.sadd('tracker:tags:{}'.format(term_uuid), tag)
def replace_tracked_term_mails(term_uuid, mails): def replace_tracked_term_mails(term_uuid, mails):
res = verify_mail_list(mails) res = verify_mail_list(mails)
if res: if res:
return res return res
else: else:
r_serv_term.delete('tracked_term:mail:{}'.format(term_uuid)) r_serv_term.delete('tracker:mail:{}'.format(term_uuid))
for mail in mails: for mail in mails:
r_serv_term.sadd('tracked_term:mail:{}'.format(term_uuid), mail) r_serv_term.sadd('tracker:mail:{}'.format(term_uuid), mail)
def get_term_uuid_list(term, term_type): def get_term_uuid_list(term, term_type):
return list(r_serv_term.smembers('all:tracked_term_uuid:{}:{}'.format(term_type, term))) return list(r_serv_term.smembers('all:tracker_uuid:{}:{}'.format(term_type, term)))
def get_term_tags(term_uuid): def get_term_tags(term_uuid):
return list(r_serv_term.smembers('tracked_term:tags:{}'.format(term_uuid))) return list(r_serv_term.smembers('tracker:tags:{}'.format(term_uuid)))
def get_term_mails(term_uuid): def get_term_mails(term_uuid):
return list(r_serv_term.smembers('tracked_term:mail:{}'.format(term_uuid))) return list(r_serv_term.smembers('tracker:mail:{}'.format(term_uuid)))
def add_tracked_item(term_uuid, item_id, item_date): def add_tracked_item(term_uuid, item_id, item_date):
# track item # track item
r_serv_term.sadd('tracked_term:item:{}:{}'.format(term_uuid, item_date), item_id) r_serv_term.sadd('tracker:item:{}:{}'.format(term_uuid, item_date), item_id)
# track nb item by date # track nb item by date
r_serv_term.zadd('tracked_term:stat:{}'.format(term_uuid), item_date, int(item_date)) r_serv_term.zadd('tracker:stat:{}'.format(term_uuid), item_date, int(item_date))
def create_token_statistics(item_date, word, nb): def create_token_statistics(item_date, word, nb):
r_serv_term.zincrby('stat_token_per_item_by_day:{}'.format(item_date), word, 1) r_serv_term.zincrby('stat_token_per_item_by_day:{}'.format(item_date), word, 1)
@ -336,7 +342,7 @@ def get_all_token_stat_history():
return r_serv_term.smembers('stat_token_history') return r_serv_term.smembers('stat_token_history')
def get_tracked_term_last_updated_by_type(term_type): def get_tracked_term_last_updated_by_type(term_type):
epoch_update = r_serv_term.get('tracked_term:refresh:{}'.format(term_type)) epoch_update = r_serv_term.get('tracker:refresh:{}'.format(term_type))
if not epoch_update: if not epoch_update:
epoch_update = 0 epoch_update = 0
return float(epoch_update) return float(epoch_update)
@ -363,6 +369,7 @@ def parse_get_tracker_term_item(dict_input, user_id):
date_from = date_to date_from = date_to
all_item_id = get_tracked_term_list_item(term_uuid, date_from, date_to) all_item_id = get_tracked_term_list_item(term_uuid, date_from, date_to)
all_item_id = Item.get_item_list_desc(all_item_id)
res_dict = {} res_dict = {}
res_dict['uuid'] = term_uuid res_dict['uuid'] = term_uuid
@ -372,7 +379,7 @@ def parse_get_tracker_term_item(dict_input, user_id):
return (res_dict, 200) return (res_dict, 200)
def get_tracked_term_first_seen(term_uuid): def get_tracked_term_first_seen(term_uuid):
res = r_serv_term.zrange('tracked_term:stat:{}'.format(term_uuid), 0, 0) res = r_serv_term.zrange('tracker:stat:{}'.format(term_uuid), 0, 0)
if res: if res:
return res[0] return res[0]
else: else:
@ -380,7 +387,7 @@ def get_tracked_term_first_seen(term_uuid):
def get_tracked_term_last_seen(term_uuid): def get_tracked_term_last_seen(term_uuid):
res = r_serv_term.zrevrange('tracked_term:stat:{}'.format(term_uuid), 0, 0) res = r_serv_term.zrevrange('tracker:stat:{}'.format(term_uuid), 0, 0)
if res: if res:
return res[0] return res[0]
else: else:
@ -388,15 +395,15 @@ def get_tracked_term_last_seen(term_uuid):
def get_term_metedata(term_uuid, user_id=False, level=False, tags=False, mails=False, sparkline=False): def get_term_metedata(term_uuid, user_id=False, level=False, tags=False, mails=False, sparkline=False):
dict_uuid = {} dict_uuid = {}
dict_uuid['term'] = r_serv_term.hget('tracked_term:{}'.format(term_uuid), 'tracked') dict_uuid['term'] = r_serv_term.hget('tracker:{}'.format(term_uuid), 'tracked')
dict_uuid['type'] = r_serv_term.hget('tracked_term:{}'.format(term_uuid), 'type') dict_uuid['type'] = r_serv_term.hget('tracker:{}'.format(term_uuid), 'type')
dict_uuid['date'] = r_serv_term.hget('tracked_term:{}'.format(term_uuid), 'date') dict_uuid['date'] = r_serv_term.hget('tracker:{}'.format(term_uuid), 'date')
dict_uuid['first_seen'] = get_tracked_term_first_seen(term_uuid) dict_uuid['first_seen'] = get_tracked_term_first_seen(term_uuid)
dict_uuid['last_seen'] = get_tracked_term_last_seen(term_uuid) dict_uuid['last_seen'] = get_tracked_term_last_seen(term_uuid)
if user_id: if user_id:
dict_uuid['user_id'] = r_serv_term.hget('tracked_term:{}'.format(term_uuid), 'user_id') dict_uuid['user_id'] = r_serv_term.hget('tracker:{}'.format(term_uuid), 'user_id')
if level: if level:
dict_uuid['level'] = r_serv_term.hget('tracked_term:{}'.format(term_uuid), 'level') dict_uuid['level'] = r_serv_term.hget('tracker:{}'.format(term_uuid), 'level')
if mails: if mails:
dict_uuid['mails'] = get_list_trackeed_term_mails(term_uuid) dict_uuid['mails'] = get_list_trackeed_term_mails(term_uuid)
if tags: if tags:
@ -406,48 +413,71 @@ def get_term_metedata(term_uuid, user_id=False, level=False, tags=False, mails=F
dict_uuid['uuid'] = term_uuid dict_uuid['uuid'] = term_uuid
return dict_uuid return dict_uuid
def get_tracked_term_sparkline(term_uuid, num_day=6): def get_tracked_term_sparkline(tracker_uuid, num_day=6):
date_range_sparkline = Date.get_date_range(num_day) date_range_sparkline = Date.get_date_range(num_day)
sparklines_value = [] sparklines_value = []
for date_day in date_range_sparkline: for date_day in date_range_sparkline:
nb_seen_this_day = r_serv_term.zscore('tracked_term:stat:{}'.format(term_uuid), date_day) nb_seen_this_day = r_serv_term.scard('tracker:item:{}:{}'.format(tracker_uuid, date_day))
if nb_seen_this_day is None: if nb_seen_this_day is None:
nb_seen_this_day = 0 nb_seen_this_day = 0
sparklines_value.append(int(nb_seen_this_day)) sparklines_value.append(int(nb_seen_this_day))
return sparklines_value return sparklines_value
def get_list_tracked_term_stats_by_day(list_tracker_uuid, num_day=31, date_from=None, date_to=None):
if date_from and date_to:
date_range = Date.substract_date(date_from, date_to)
else:
date_range = Date.get_date_range(num_day)
list_tracker_stats = []
for tracker_uuid in list_tracker_uuid:
dict_tracker_data = []
tracker = r_serv_term.hget('tracker:{}'.format(tracker_uuid), 'tracked')
for date_day in date_range:
nb_seen_this_day = r_serv_term.scard('tracker:item:{}:{}'.format(tracker_uuid, date_day))
if nb_seen_this_day is None:
nb_seen_this_day = 0
dict_tracker_data.append({"date": date_day,"value": int(nb_seen_this_day)})
list_tracker_stats.append({"name": tracker,"Data": dict_tracker_data})
return list_tracker_stats
def get_list_trackeed_term_tags(term_uuid): def get_list_trackeed_term_tags(term_uuid):
res = r_serv_term.smembers('tracked_term:tags:{}'.format(term_uuid)) res = r_serv_term.smembers('tracker:tags:{}'.format(term_uuid))
if res: if res:
return list(res) return list(res)
else: else:
return [] return []
def get_list_trackeed_term_mails(term_uuid): def get_list_trackeed_term_mails(term_uuid):
res = r_serv_term.smembers('tracked_term:mail:{}'.format(term_uuid)) res = r_serv_term.smembers('tracker:mail:{}'.format(term_uuid))
if res: if res:
return list(res) return list(res)
else: else:
return [] return []
def get_user_tracked_term_uuid(user_id): def get_user_tracked_term_uuid(user_id, filter_type=None):
return list(r_serv_term.smembers('user:tracked_term:{}'.format(user_id))) if filter_type:
return list(r_serv_term.smembers('user:tracker:{}:{}'.format(user_id,filter_type)))
else:
return list(r_serv_term.smembers('user:tracker:{}'.format(user_id)))
def get_global_tracked_term_uuid(): def get_global_tracked_term_uuid(filter_type=None):
return list(r_serv_term.smembers('global:tracked_term')) if filter_type:
return list(r_serv_term.smembers('global:tracker:{}'.format(filter_type)))
else:
return list(r_serv_term.smembers('global:tracker'))
def get_all_user_tracked_terms(user_id): def get_all_user_tracked_terms(user_id, filter_type=None):
all_user_term = [] all_user_term = []
all_user_term_uuid = get_user_tracked_term_uuid(user_id) all_user_term_uuid = get_user_tracked_term_uuid(user_id, filter_type=filter_type)
for term_uuid in all_user_term_uuid: for term_uuid in all_user_term_uuid:
all_user_term.append(get_term_metedata(term_uuid, tags=True, mails=True)) all_user_term.append(get_term_metedata(term_uuid, tags=True, mails=True, sparkline=True))
return all_user_term return all_user_term
def get_all_global_tracked_terms(): def get_all_global_tracked_terms(filter_type=None):
all_user_term = [] all_user_term = []
all_user_term_uuid = get_global_tracked_term_uuid() all_user_term_uuid = get_global_tracked_term_uuid(filter_type=filter_type)
for term_uuid in all_user_term_uuid: for term_uuid in all_user_term_uuid:
all_user_term.append(get_term_metedata(term_uuid, user_id=True, tags=True, mails=True)) all_user_term.append(get_term_metedata(term_uuid, user_id=True, tags=True, mails=True, sparkline=True))
return all_user_term return all_user_term

216
var/www/modules/hunter/Flask_hunter.py Normal file
View File

@ -0,0 +1,216 @@
#!/usr/bin/env python3
# -*-coding:UTF-8 -*
'''
Flask functions and routes for tracked items
'''
import json
import redis
import datetime
import calendar
import flask
from flask import Flask, render_template, jsonify, request, Blueprint, url_for, redirect, Response
from Role_Manager import login_admin, login_analyst
from flask_login import login_required, current_user
import re
from pprint import pprint
import Levenshtein
# ---------------------------------------------------------------
import Paste
import Term
# ============ VARIABLES ============
import Flask_config
app = Flask_config.app
cfg = Flask_config.cfg
baseUrl = Flask_config.baseUrl
r_serv_term = Flask_config.r_serv_term
r_serv_cred = Flask_config.r_serv_cred
r_serv_db = Flask_config.r_serv_db
bootstrap_label = Flask_config.bootstrap_label
hunter = Blueprint('hunter', __name__, template_folder='templates')
# ============ FUNCTIONS ============
# ============ ROUTES ============
@hunter.route("/trackers")
@login_required
@login_analyst
def tracked_menu():
user_id = current_user.get_id()
user_term = Term.get_all_user_tracked_terms(user_id)
global_term = Term.get_all_global_tracked_terms()
return render_template("trackersManagement.html", user_term=user_term, global_term=global_term, bootstrap_label=bootstrap_label)
@hunter.route("/trackers/word")
@login_required
@login_analyst
def tracked_menu_word():
filter_type = 'word'
user_id = current_user.get_id()
user_term = Term.get_all_user_tracked_terms(user_id, filter_type='word')
global_term = Term.get_all_global_tracked_terms(filter_type='word')
return render_template("trackersManagement.html", user_term=user_term, global_term=global_term, bootstrap_label=bootstrap_label, filter_type=filter_type)
@hunter.route("/trackers/set")
@login_required
@login_analyst
def tracked_menu_set():
filter_type = 'set'
user_id = current_user.get_id()
user_term = Term.get_all_user_tracked_terms(user_id, filter_type=filter_type)
global_term = Term.get_all_global_tracked_terms(filter_type=filter_type)
return render_template("trackersManagement.html", user_term=user_term, global_term=global_term, bootstrap_label=bootstrap_label, filter_type=filter_type)
@hunter.route("/trackers/regex")
@login_required
@login_analyst
def tracked_menu_regex():
filter_type = 'regex'
user_id = current_user.get_id()
user_term = Term.get_all_user_tracked_terms(user_id, filter_type=filter_type)
global_term = Term.get_all_global_tracked_terms(filter_type=filter_type)
return render_template("trackersManagement.html", user_term=user_term, global_term=global_term, bootstrap_label=bootstrap_label, filter_type=filter_type)
@hunter.route("/tracker/add", methods=['GET', 'POST'])
@login_required
@login_analyst
def add_tracked_menu():
if request.method == 'POST':
term = request.form.get("term")
term_type = request.form.get("tracker_type")
nb_words = request.form.get("nb_word", 1)
level = request.form.get("level", 0)
tags = request.form.get("tags", [])
mails = request.form.get("mails", [])
if level == 'on':
level = 1
if mails:
mails = mails.split()
if tags:
tags = tags.split()
input_dict = {"term": term, "type": term_type, "nb_words": nb_words, "tags": tags, "mails": mails, "level": level}
user_id = current_user.get_id()
res = Term.parse_json_term_to_add(input_dict, user_id)
if res[1] == 200:
return redirect(url_for('hunter.tracked_menu'))
else:
## TODO: use modal
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
else:
return render_template("Add_tracker.html")
@hunter.route("/tracker/show_tracker")
@login_required
@login_analyst
def show_tracker():
user_id = current_user.get_id()
term_uuid = request.args.get('uuid', None)
res = Term.check_term_uuid_valid_access(term_uuid, user_id)
if res: # invalid access
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
date_from = request.args.get('date_from')
date_to = request.args.get('date_to')
if date_from:
date_from = date_from.replace('-', '')
if date_to:
date_to = date_to.replace('-', '')
tracker_metadata = Term.get_term_metedata(term_uuid, user_id=True, level=True, tags=True, mails=True, sparkline=True)
if date_from:
res = Term.parse_get_tracker_term_item({'uuid': term_uuid, 'date_from': date_from, 'date_to': date_to}, user_id)
if res[1] !=200:
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
tracker_metadata['items'] = res[0]['items']
tracker_metadata['date_from'] = res[0]['date_from']
tracker_metadata['date_to'] = res[0]['date_to']
else:
tracker_metadata['items'] = []
tracker_metadata['date_from'] = ''
tracker_metadata['date_to'] = ''
return render_template("showTracker.html", tracker_metadata=tracker_metadata, bootstrap_label=bootstrap_label)
@hunter.route("/tracker/update_tracker_tags", methods=['POST'])
@login_required
@login_analyst
def update_tracker_tags():
user_id = current_user.get_id()
term_uuid = request.form.get('uuid')
res = Term.check_term_uuid_valid_access(term_uuid, user_id)
if res: # invalid access
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
tags = request.form.get('tags')
if tags:
tags = tags.split()
else:
tags = []
Term.replace_tracked_term_tags(term_uuid, tags)
return redirect(url_for('hunter.show_tracker', uuid=term_uuid))
@hunter.route("/tracker/update_tracker_mails", methods=['POST'])
@login_required
@login_analyst
def update_tracker_mails():
user_id = current_user.get_id()
term_uuid = request.form.get('uuid')
res = Term.check_term_uuid_valid_access(term_uuid, user_id)
if res: # invalid access
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
mails = request.form.get('mails')
if mails:
mails = mails.split()
else:
mails = []
res = Term.replace_tracked_term_mails(term_uuid, mails)
if res: # invalid mail
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
return redirect(url_for('hunter.show_tracker', uuid=term_uuid))
@hunter.route("/tracker/delete", methods=['GET'])
@login_required
@login_analyst
def delete_tracker():
user_id = current_user.get_id()
term_uuid = request.args.get('uuid')
res = Term.parse_tracked_term_to_delete({'uuid': term_uuid}, user_id)
if res[1] !=200:
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
return redirect(url_for('hunter.tracked_menu'))
@hunter.route("/tracker/get_json_tracker_stats", methods=['GET'])
@login_required
@login_analyst
def get_json_tracker_stats():
date_from = request.args.get('date_from')
date_to = request.args.get('date_to')
if date_from:
date_from = date_from.replace('-', '')
if date_to:
date_to = date_to.replace('-', '')
tracker_uuid = request.args.get('uuid')
if date_from and date_to:
res = Term.get_list_tracked_term_stats_by_day([tracker_uuid], date_from=date_from, date_to=date_to)
else:
res = Term.get_list_tracked_term_stats_by_day([tracker_uuid])
return jsonify(res)
# ========= REGISTRATION =========
app.register_blueprint(hunter, url_prefix=baseUrl)

4
var/www/modules/terms/templates/Add_tracker.html → var/www/modules/hunter/templates/Add_tracker.html
View File

@ -23,7 +23,7 @@
<div class="container-fluid"> <div class="container-fluid">
<div class="row"> <div class="row">
{% include 'crawler/menu_sidebar.html' %} {% include 'hunter/menu_sidebar.html' %}
<div class="col-12 col-lg-10" id="core_content"> <div class="col-12 col-lg-10" id="core_content">
@ -34,7 +34,7 @@
<div class="card-body"> <div class="card-body">
<p class="card-text">Enter a domain and choose what kind of data you want.</p> <p class="card-text">Enter a domain and choose what kind of data you want.</p>
<form action="{{ url_for('terms.add_tracked_term_menu') }}" method='post'> <form action="{{ url_for('hunter.add_tracked_menu') }}" method='post'>
<div class="row"> <div class="row">
<div class="col-12 col-xl-9"> <div class="col-12 col-xl-9">

136
var/www/modules/terms/templates/showTrackerTerm.html → var/www/modules/hunter/templates/showTracker.html
View File

@ -11,21 +11,21 @@
<link href="{{ url_for('static', filename='css/bootstrap4.min.css') }}" rel="stylesheet"> <link href="{{ url_for('static', filename='css/bootstrap4.min.css') }}" rel="stylesheet">
<link href="{{ url_for('static', filename='css/font-awesome.min.css') }}" rel="stylesheet"> <link href="{{ url_for('static', filename='css/font-awesome.min.css') }}" rel="stylesheet">
<link href="{{ url_for('static', filename='css/daterangepicker.min.css') }}" rel="stylesheet"> <link href="{{ url_for('static', filename='css/daterangepicker.min.css') }}" rel="stylesheet">
<link href="{{ url_for('static', filename='css/dataTables.bootstrap.min.css') }}" rel="stylesheet">
<!-- JS --> <!-- JS -->
<script src="{{ url_for('static', filename='js/jquery.js')}}"></script> <script src="{{ url_for('static', filename='js/jquery.js')}}"></script>
<script src="{{ url_for('static', filename='js/popper.min.js')}}"></script> <script src="{{ url_for('static', filename='js/popper.min.js')}}"></script>
<script src="{{ url_for('static', filename='js/bootstrap4.min.js')}}"></script> <script src="{{ url_for('static', filename='js/bootstrap4.min.js')}}"></script>
<script src="{{ url_for('static', filename='js/jquery.dataTables.min.js')}}"></script>
<script src="{{ url_for('static', filename='js/dataTables.bootstrap.min.js')}}"></script>
<script language="javascript" src="{{ url_for('static', filename='js/d3.min.js') }}"></script> <script language="javascript" src="{{ url_for('static', filename='js/d3.min.js') }}"></script>
<script src="{{ url_for('static', filename='js/plugins/d3/sparklines.js')}}"></script>
<script src="{{ url_for('static', filename='js/plugins/d3/graphlinesgroup.js')}}"></script>
<script language="javascript" src="{{ url_for('static', filename='js/moment.min.js') }}"></script> <script language="javascript" src="{{ url_for('static', filename='js/moment.min.js') }}"></script>
<script language="javascript" src="{{ url_for('static', filename='js/jquery.daterangepicker.min.js') }}"></script> <script language="javascript" src="{{ url_for('static', filename='js/jquery.daterangepicker.min.js') }}"></script>
<style> <style>
.line_sparkline {
fill: none;
stroke: #000;
stroke-width: 2.0px;
}
.btn-link { .btn-link {
color: #000000 color: #000000
} }
@ -42,13 +42,13 @@
<div class="container-fluid"> <div class="container-fluid">
<div class="row"> <div class="row">
{% include 'tracker/menu_sidebar.html' %} {% include 'hunter/menu_sidebar.html' %}
<div class="col-12 col-lg-10" id="core_content"> <div class="col-12 col-lg-10" id="core_content">
<div class="card my-3"> <div class="card my-3">
<div class="card-header" style="background-color:#d9edf7;font-size: 15px"> <div class="card-header" style="background-color:#d9edf7;font-size: 15px">
<h4 class="text-secondary">{{ term_metadata['uuid'] }} </h4> <h4 class="text-secondary">{{ tracker_metadata['uuid'] }} </h4>
<ul class="list-group mb-2"> <ul class="list-group mb-2">
<li class="list-group-item py-0"> <li class="list-group-item py-0">
<div class="row"> <div class="row">
@ -57,7 +57,7 @@
<thead> <thead>
<tr> <tr>
<th>Type</th> <th>Type</th>
<th>Tracked</th> <th>Tracker</th>
<th>Date added</th> <th>Date added</th>
<th>Level</th> <th>Level</th>
<th>Created by</th> <th>Created by</th>
@ -69,23 +69,23 @@
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td>{{ term_metadata['type'] }}</td> <td>{{ tracker_metadata['type'] }}</td>
<td>{{ term_metadata['term'] }}</td> <td>{{ tracker_metadata['term'] }}</td>
<td>{{ term_metadata['date'][0:4] }}/{{ term_metadata['date'][4:6] }}/{{ term_metadata['date'][6:8] }}</td> <td>{{ tracker_metadata['date'][0:4] }}/{{ tracker_metadata['date'][4:6] }}/{{ tracker_metadata['date'][6:8] }}</td>
<td>{{ term_metadata['level'] }}</td> <td>{{ tracker_metadata['level'] }}</td>
<td>{{ term_metadata['user_id'] }}</td> <td>{{ tracker_metadata['user_id'] }}</td>
<td> <td>
{% if term_metadata['first_seen'] %} {% if tracker_metadata['first_seen'] %}
{{ term_metadata['first_seen'][0:4] }}/{{ term_metadata['first_seen'][4:6] }}/{{ term_metadata['first_seen'][6:8] }} {{ tracker_metadata['first_seen'][0:4] }}/{{ tracker_metadata['first_seen'][4:6] }}/{{ tracker_metadata['first_seen'][6:8] }}
{% endif %} {% endif %}
</td> </td>
<td> <td>
{% if term_metadata['last_seen'] %} {% if tracker_metadata['last_seen'] %}
{{ term_metadata['last_seen'][0:4] }}/{{ term_metadata['last_seen'][4:6] }}/{{ term_metadata['last_seen'][6:8] }} {{ tracker_metadata['last_seen'][0:4] }}/{{ tracker_metadata['last_seen'][4:6] }}/{{ tracker_metadata['last_seen'][6:8] }}
{% endif %} {% endif %}
</td> </td>
<td> <td>
{% for tag in term_metadata['tags'] %} {% for tag in tracker_metadata['tags'] %}
<a href="{{ url_for('Tags.Tags_page') }}?ltags={{ tag }}"> <a href="{{ url_for('Tags.Tags_page') }}?ltags={{ tag }}">
<span class="badge badge-{{ bootstrap_label[loop.index0 % 5] }}">{{ tag }}</span> <span class="badge badge-{{ bootstrap_label[loop.index0 % 5] }}">{{ tag }}</span>
</a> </a>
@ -93,7 +93,7 @@
</td> </td>
<td> <td>
{% for mail in term_metadata['mails'] %} {% for mail in tracker_metadata['mails'] %}
{{ mail }}<br> {{ mail }}<br>
{% endfor %} {% endfor %}
</td> </td>
@ -109,15 +109,15 @@
</ul> </ul>
<div id="div_edit_tags"> <div id="div_edit_tags">
<form action="{{ url_for('terms.update_tracker_tags') }}" method='post'> <form action="{{ url_for('hunter.update_tracker_tags') }}" method='post'>
<input name="uuid" type="text" value="{{term_metadata['uuid']}}" hidden> <input name="uuid" type="text" value="{{tracker_metadata['uuid']}}" hidden>
<div>All Tags added for this tracker, space separated: </div> <div>All Tags added for this tracker, space separated: </div>
<div class="input-group mb-2 mr-sm-2"> <div class="input-group mb-2 mr-sm-2">
<div class="input-group-prepend"> <div class="input-group-prepend">
<div class="input-group-text"><i class="fas fa-tag"></i></div> <div class="input-group-text"><i class="fas fa-tag"></i></div>
</div> </div>
<input id="tags" name="tags" class="form-control" placeholder="Tags (optional, space separated)" type="text" <input id="tags" name="tags" class="form-control" placeholder="Tags (optional, space separated)" type="text"
value="{% for tag in term_metadata['tags'] %}{{tag}} {% endfor %}"> value="{% for tag in tracker_metadata['tags'] %}{{tag}} {% endfor %}">
</div> </div>
<button class="btn btn-info"> <button class="btn btn-info">
@ -128,15 +128,15 @@
</div> </div>
<div id="div_edit_mails"> <div id="div_edit_mails">
<form action="{{ url_for('terms.update_tracker_mails') }}" method='post'> <form action="{{ url_for('hunter.update_tracker_mails') }}" method='post'>
<input name="uuid" type="text" value="{{term_metadata['uuid']}}" hidden> <input name="uuid" type="text" value="{{tracker_metadata['uuid']}}" hidden>
<div>All E-Mails to Notify for this tracker, space separated: </div> <div>All E-Mails to Notify for this tracker, space separated: </div>
<div class="input-group mb-2 mr-sm-2"> <div class="input-group mb-2 mr-sm-2">
<div class="input-group-prepend"> <div class="input-group-prepend">
<div class="input-group-text"><i class="fas fa-at"></i></div> <div class="input-group-text"><i class="fas fa-at"></i></div>
</div> </div>
<input id="mails" name="mails" class="form-control" placeholder="E-Mails Notification (optional, space separated)" type="text" <input id="mails" name="mails" class="form-control" placeholder="E-Mails Notification (optional, space separated)" type="text"
value="{% for mail in term_metadata['mails'] %}{{mail}} {% endfor %}"> value="{% for mail in tracker_metadata['mails'] %}{{mail}} {% endfor %}">
</div> </div>
<button class="btn btn-info"> <button class="btn btn-info">
@ -146,28 +146,29 @@
</div> </div>
<a href="{{ url_for('hashDecoded.downloadHash') }}?hash={{hash}}" target="blank" class="float-right" style="font-size: 15px"> <a href="{{ url_for('hunter.delete_tracker') }}?uuid={{tracker_metadata['uuid']}}" class="float-right" style="font-size: 15px">
<button class='btn btn-danger'><i class="fas fa-trash-alt"></i> <button class='btn btn-danger'><i class="fas fa-trash-alt"></i>
</button> </button>
</a> </a>
</div> </div>
</div> </div>
<div id="graphline" class="text-center"></div>
<div class="card mb-3 mt-1"> <div class="card mb-5 mt-1">
<div class="card-body"> <div class="card-body">
<div class="row mb-3"> <div class="row mb-3">
<div class="col-md-6"> <div class="col-md-6">
<div class="input-group" id="date-range-from"> <div class="input-group" id="date-range-from">
<div class="input-group-prepend"><span class="input-group-text"><i class="far fa-calendar-alt" aria-hidden="true"></i></span></div> <div class="input-group-prepend"><span class="input-group-text"><i class="far fa-calendar-alt" aria-hidden="true"></i></span></div>
<input class="form-control" id="date-range-from-input" placeholder="yyyy-mm-dd" value="{{ term_metadata['date_from'] }}" name="date_from" autocomplete="off"> <input class="form-control" id="date-range-from-input" placeholder="yyyy-mm-dd" value="{{ tracker_metadata['date_from'] }}" name="date_from" autocomplete="off">
</div> </div>
</div> </div>
<div class="col-md-6"> <div class="col-md-6">
<div class="input-group" id="date-range-to"> <div class="input-group" id="date-range-to">
<div class="input-group-prepend"><span class="input-group-text"><i class="far fa-calendar-alt" aria-hidden="true"></i></span></div> <div class="input-group-prepend"><span class="input-group-text"><i class="far fa-calendar-alt" aria-hidden="true"></i></span></div>
<input class="form-control" id="date-range-to-input" placeholder="yyyy-mm-dd" value="{{ term_metadata['date_to'] }}" name="date_to" autocomplete="off"> <input class="form-control" id="date-range-to-input" placeholder="yyyy-mm-dd" value="{{ tracker_metadata['date_to'] }}" name="date_to" autocomplete="off">
</div> </div>
</div> </div>
</div> </div>
@ -179,27 +180,40 @@
</div> </div>
</div> </div>
{%if term_metadata['items']%} {%if tracker_metadata['items']%}
<div class="mt-4">
<table class="table table-bordered table-hover" id="myTable_"> <table class="table table-bordered table-hover" id="myTable_">
<thead class="thead-dark"> <thead class="thead-dark">
<tr> <tr>
<th>Item</th> <th>Date</th>
<th>Item Id</th>
</tr> </tr>
</thead> </thead>
<tbody> <tbody>
{% for item in term_metadata['items'] %} {% for item in tracker_metadata['items'] %}
<tr> <tr>
<td> <td>
<a target="_blank" href="{{ url_for('showsavedpastes.showsavedpaste') }}?paste={{item}}"> {{item['date'][0:4]}}/{{item['date'][4:6]}}/{{item['date'][6:8]}}
<div style="line-height:0.9;">{{ item }}</div> </td>
<td>
<a class="text-secondary" target="_blank" href="{{ url_for('showsavedpastes.showsavedpaste') }}?paste={{item['id']}}">
<div style="line-height:0.9;">{{ item['id'] }}</div>
</a> </a>
<div class="mb-2">
{% for tag in item['tags'] %}
<a href="{{ url_for('Tags.Tags_page') }}?ltags={{ tag }}">
<span class="badge badge-{{ bootstrap_label[loop.index0 % 5] }} pull-left">{{ tag }}</span>
</a>
{% endfor %}
</div>
</td> </td>
</tr> </tr>
{% endfor %} {% endfor %}
</tbody> </tbody>
</table> </table>
</div>
{% endif %} {% endif %}
</div> </div>
@ -239,7 +253,18 @@ $(document).ready(function(){
} }
}); });
sparklines("sparkline", {{ term_metadata['sparkline'] }}); $('#myTable_').DataTable({
"aLengthMenu": [[5, 10, 15, -1], [5, 10, 15, "All"]],
"iDisplayLength": 10,
"order": [[ 0, "desc" ]]
});
sparkline("sparkline", {{ tracker_metadata['sparkline'] }}, {});
let div_width = $("#graphline").width();
$.getJSON( "{{ url_for('hunter.get_json_tracker_stats') }}?uuid={{ tracker_metadata['uuid'] }}{%if tracker_metadata['date_from']%}&date_from={{ tracker_metadata['date_from'] }}{%endif%}{%if tracker_metadata['date_to']%}&date_to={{ tracker_metadata['date_to'] }}{%endif%}",
function( data ) {multilines_group("graphline", data, {"width": div_width});}
);
}); });
function toggle_sidebar(){ function toggle_sidebar(){
@ -269,49 +294,10 @@ function edit_mails(){
function getItems() { function getItems() {
var date_from = $('#date-range-from-input').val(); var date_from = $('#date-range-from-input').val();
var date_to =$('#date-range-to-input').val(); var date_to =$('#date-range-to-input').val();
window.location.replace("{{ url_for('terms.show_term_tracker') }}?uuid={{ term_metadata['uuid'] }}&date_from="+date_from+"&date_to="+date_to); window.location.replace("{{ url_for('hunter.show_tracker') }}?uuid={{ tracker_metadata['uuid'] }}&date_from="+date_from+"&date_to="+date_to);
} }
</script> </script>
<script>
//var data = [6,3,3,2,5,3,9];
// a sparklines plot
function sparklines(id, points) {
var width_spark = 100, height_spark = 60;
var data = []
for (i = 0; i < points.length; i++) {
data[i] = {
'x': i,
'y': +points[i]
}
}
var x = d3.scaleLinear()
.range([0, width_spark - 10])
.domain([0,5]);
var y = d3.scaleLinear()
.range([height_spark, 0])
.domain([0,10]);
var line = d3.line()
.x(function(d) {return x(d.x)})
.y(function(d) {return y(d.y)});
d3.select("#"+id).append('svg')
.attr('width', width_spark)
.attr('height', height_spark)
.append('path')
.attr('class','line_sparkline')
.datum(data)
.attr('d', line);
}
</script>
</body> </body>

44
var/www/modules/terms/templates/tracker_term_management.html → var/www/modules/hunter/templates/trackersManagement.html
View File

@ -5,7 +5,7 @@
<meta charset="utf-8"> <meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Terms Management</title> <title>Tracker Management</title>
<link rel="icon" href="{{ url_for('static', filename='image/ail-icon.png') }}"> <link rel="icon" href="{{ url_for('static', filename='image/ail-icon.png') }}">
<!-- Core CSS --> <!-- Core CSS -->
@ -17,6 +17,8 @@
<script src="{{ url_for('static', filename='js/bootstrap4.min.js')}}"></script> <script src="{{ url_for('static', filename='js/bootstrap4.min.js')}}"></script>
<script src="{{ url_for('static', filename='js/jquery.dataTables.min.js')}}"></script> <script src="{{ url_for('static', filename='js/jquery.dataTables.min.js')}}"></script>
<script src="{{ url_for('static', filename='js/dataTables.bootstrap.min.js')}}"></script> <script src="{{ url_for('static', filename='js/dataTables.bootstrap.min.js')}}"></script>
<script src="{{ url_for('static', filename='js/d3.min.js') }}"></script>
<script src="{{ url_for('static', filename='js/plugins/d3/sparklines.js')}}"></script>
<style> <style>
@ -38,23 +40,24 @@
<div class="container-fluid"> <div class="container-fluid">
<div class="row"> <div class="row">
{% include 'tracker/menu_sidebar.html' %} {% include 'hunter/menu_sidebar.html' %}
<div class="col-12 col-lg-10" id="core_content"> <div class="col-12 col-lg-10" id="core_content">
<div class="card my-3"> <div class="card my-3">
<div class="card-header"> <div class="card-header">
<h5 class="card-title">Your Tracked Terms</h5> <h5 class="card-title">Your {{filter_type}} Trackers</h5>
</div> </div>
<div class="card-body"> <div class="card-body">
<table id="table_user_terms" class="table table-striped table-bordered"> <table id="table_user_trackers" class="table table-striped table-bordered">
<thead class="bg-dark text-white"> <thead class="bg-dark text-white">
<tr> <tr>
<th>Type</th> <th>Type</th>
<th>Tracked Term</th> <th>Tracker</th>
<th>First seen</th> <th>First seen</th>
<th>Last seen</th> <th>Last seen</th>
<th>Email notification</th> <th>Email notification</th>
<th>sparkline</th>
</tr> </tr>
</thead> </thead>
<tbody style="font-size: 15px;"> <tbody style="font-size: 15px;">
@ -62,11 +65,11 @@
<tr> <tr>
<td>{{dict_uuid['type']}}</td> <td>{{dict_uuid['type']}}</td>
<td> <td>
<span><a target="_blank" href="{{ url_for('terms.show_term_tracker') }}?uuid={{ dict_uuid['uuid'] }}">{{dict_uuid['term']}}</a></span> <span><a target="_blank" href="{{ url_for('hunter.show_tracker') }}?uuid={{ dict_uuid['uuid'] }}">{{dict_uuid['term']}}</a></span>
<div> <div>
{% for tag in dict_uuid['tags'] %} {% for tag in dict_uuid['tags'] %}
<a href="{{ url_for('Tags.Tags_page') }}?ltags={{ tag }}"> <a href="{{ url_for('Tags.Tags_page') }}?ltags={{ tag }}">
<span class="label label-{{ bootstrap_label[loop.index0 % 5] }} pull-left">{{ tag }}</span> <span class="badge badge-{{ bootstrap_label[loop.index0 % 5] }} pull-left">{{ tag }}</span>
</a> </a>
{% endfor %} {% endfor %}
</div> </div>
@ -86,6 +89,7 @@
{{ mail }}<br> {{ mail }}<br>
{% endfor %} {% endfor %}
</td> </td>
<td id="sparklines_{{ dict_uuid['uuid'] }}" style="text-align:center;"></td>
</tr> </tr>
{% endfor %} {% endfor %}
</tbody> </tbody>
@ -95,17 +99,18 @@
<div class="card my-3"> <div class="card my-3">
<div class="card-header"> <div class="card-header">
<h5 class="card-title">Global Tracked Terms</h5> <h5 class="card-title">Global {{filter_type}} Trackers</h5>
</div> </div>
<div class="card-body"> <div class="card-body">
<table id="table_global_terms" class="table table-striped table-bordered"> <table id="table_global_trackers" class="table table-striped table-bordered">
<thead class="bg-dark text-white"> <thead class="bg-dark text-white">
<tr> <tr>
<th>Type</th> <th>Type</th>
<th>Tracked Term</th> <th>Tracker</th>
<th>First seen</th> <th>First seen</th>
<th>Last seen</th> <th>Last seen</th>
<th>Email notification</th> <th>Email notification</th>
<th>sparkline</th>
</tr> </tr>
</thead> </thead>
<tbody style="font-size: 15px;"> <tbody style="font-size: 15px;">
@ -113,7 +118,7 @@
<tr> <tr>
<td>{{dict_uuid['type']}}</td> <td>{{dict_uuid['type']}}</td>
<td> <td>
<span><a target="_blank" href="{{ url_for('terms.show_term_tracker') }}?uuid={{ dict_uuid['uuid'] }}">{{dict_uuid['term']}}</a></span> <span><a target="_blank" href="{{ url_for('hunter.show_tracker') }}?uuid={{ dict_uuid['uuid'] }}">{{dict_uuid['term']}}</a></span>
<div> <div>
{% for tag in dict_uuid['tags'] %} {% for tag in dict_uuid['tags'] %}
<a href="{{ url_for('Tags.Tags_page') }}?ltags={{ tag }}"> <a href="{{ url_for('Tags.Tags_page') }}?ltags={{ tag }}">
@ -137,6 +142,7 @@
{{ mail }}<br> {{ mail }}<br>
{% endfor %} {% endfor %}
</td> </td>
<td id="sparklines_{{ dict_uuid['uuid'] }}" style="text-align:center;"></td>
</tr> </tr>
{% endfor %} {% endfor %}
</tbody> </tbody>
@ -150,16 +156,28 @@
<script> <script>
$(document).ready(function(){ $(document).ready(function(){
$('#table_user_terms').DataTable({ $("#page-Tracker").addClass("active");
$("#nav_tracker_{{filter_type}}").addClass("active");
$('#table_user_trackers').DataTable({
"aLengthMenu": [[5, 10, 15, -1], [5, 10, 15, "All"]], "aLengthMenu": [[5, 10, 15, -1], [5, 10, 15, "All"]],
"iDisplayLength": 10, "iDisplayLength": 10,
"order": [[ 0, "desc" ]] "order": [[ 0, "desc" ]]
}); });
$('#table_global_terms').DataTable({ $('#table_global_trackers').DataTable({
"aLengthMenu": [[5, 10, 15, -1], [5, 10, 15, "All"]], "aLengthMenu": [[5, 10, 15, -1], [5, 10, 15, "All"]],
"iDisplayLength": 10, "iDisplayLength": 10,
"order": [[ 0, "desc" ]] "order": [[ 0, "desc" ]]
}); });
{% for dict_uuid in user_term %}
sparkline("sparklines_{{ dict_uuid['uuid'] }}", {{ dict_uuid['sparkline'] }}, {height: 40});
{% endfor %}
{% for dict_uuid in global_term %}
sparkline("sparklines_{{ dict_uuid['uuid'] }}", {{ dict_uuid['sparkline'] }}, {height: 40});
{% endfor %}
}); });
function toggle_sidebar(){ function toggle_sidebar(){

105
var/www/modules/terms/Flask_terms.py
View File

@ -151,111 +151,6 @@ def save_tag_to_auto_push(list_tag):
# ============ ROUTES ============ # ============ ROUTES ============
@terms.route("/tracker_term")
def tracked_term_menu():
user_id = current_user.get_id()
user_term = Term.get_all_user_tracked_terms(user_id)
global_term = Term.get_all_global_tracked_terms()
return render_template("tracker_term_management.html", user_term=user_term, global_term=global_term, bootstrap_label=bootstrap_label)
@terms.route("/tracker/add", methods=['GET', 'POST'])
@login_required
@login_analyst
def add_tracked_term_menu():
if request.method == 'POST':
term = request.form.get("term")
term_type = request.form.get("tracker_type")
nb_words = request.form.get("nb_word", 1)
level = request.form.get("level", 1)
tags = request.form.get("tags", [])
mails = request.form.get("mails", [])
if mails:
mails = mails.split()
if tags:
tags = tags.split()
input_dict = {"term": term, "type": term_type, "nb_words": nb_words, "tags": tags, "mails": mails}
user_id = current_user.get_id()
res = Term.parse_json_term_to_add(input_dict, user_id)
if res[1] == 200:
return redirect(url_for('terms.tracked_term_menu'))
else:
## TODO: use modal
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
else:
return render_template("Add_tracker.html")
@terms.route("/tracker/show_term_tracker")
@login_required
@login_analyst
def show_term_tracker():
user_id = current_user.get_id()
term_uuid = request.args.get('uuid', None)
res = Term.check_term_uuid_valid_access(term_uuid, user_id)
if res: # invalid access
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
date_from = request.args.get('date_from')
date_to = request.args.get('date_to')
if date_from:
date_from = date_from.replace('-', '')
if date_to:
date_to = date_to.replace('-', '')
term_metadata = Term.get_term_metedata(term_uuid, user_id=True, level=True, tags=True, mails=True, sparkline=True)
if date_from:
res = Term.parse_get_tracker_term_item({'uuid': term_uuid, 'date_from': date_from, 'date_to': date_to}, user_id)
if res[1] !=200:
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
term_metadata['items'] = res[0]['items']
term_metadata['date_from'] = res[0]['date_from']
term_metadata['date_to'] = res[0]['date_to']
else:
term_metadata['items'] = []
term_metadata['date_from'] = ''
term_metadata['date_to'] = ''
return render_template("showTrackerTerm.html", term_metadata=term_metadata, bootstrap_label=bootstrap_label)
@terms.route("/tracker/update_tracker_tags", methods=['POST'])
@login_required
@login_analyst
def update_tracker_tags():
user_id = current_user.get_id()
term_uuid = request.form.get('uuid')
res = Term.check_term_uuid_valid_access(term_uuid, user_id)
if res: # invalid access
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
tags = request.form.get('tags')
if tags:
tags = tags.split()
else:
tags = []
Term.replace_tracked_term_tags(term_uuid, tags)
return redirect(url_for('terms.show_term_tracker', uuid=term_uuid))
@terms.route("/tracker/update_tracker_mails", methods=['POST'])
@login_required
@login_analyst
def update_tracker_mails():
user_id = current_user.get_id()
term_uuid = request.form.get('uuid')
res = Term.check_term_uuid_valid_access(term_uuid, user_id)
if res: # invalid access
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
mails = request.form.get('mails')
if mails:
mails = mails.split()
else:
mails = []
res = Term.replace_tracked_term_mails(term_uuid, mails)
if res: # invalid mail
return Response(json.dumps(res[0], indent=2, sort_keys=True), mimetype='application/json'), res[1]
return redirect(url_for('terms.show_term_tracker', uuid=term_uuid))
@terms.route("/terms_plot_tool/") @terms.route("/terms_plot_tool/")
@login_required @login_required

3
var/www/modules/terms/templates/header_terms.html
View File

@ -1,7 +1,6 @@
<li id='page-termsfrequency'><a class="dropdown-toggle" data-toggle="dropdown" href="{{ url_for('terms.terms_management') }}"><i class="fa fa-eye"></i> Terms frequency <li id='page-termsfrequency'><a class="dropdown-toggle" data-toggle="dropdown" href="{{ url_for('terms.credentials_tracker') }}"><i class="fa fa-eye"></i> Terms frequency
<span class="caret"></span></a> <span class="caret"></span></a>
<ul class="dropdown-menu"> <ul class="dropdown-menu">
<li><a href="{{ url_for('terms.terms_management') }}"><i class="fa fa-gear "> </i> Terms managements</a></li>
<li><a href="{{ url_for('terms.credentials_tracker') }}"><i class="glyphicon glyphicon-screenshot"> </i> Credentials seeker</a></li> <li><a href="{{ url_for('terms.credentials_tracker') }}"><i class="glyphicon glyphicon-screenshot"> </i> Credentials seeker</a></li>
<li><a href="{{ url_for('terms.terms_plot_top') }}"><i class="glyphicon glyphicon-fire"> </i> Terms plot top</a></li> <li><a href="{{ url_for('terms.terms_plot_top') }}"><i class="glyphicon glyphicon-fire"> </i> Terms plot top</a></li>
<li><a href="{{ url_for('terms.terms_plot_tool') }}"><i class="fa fa-wrench"> </i> Terms plot tool</a></li> <li><a href="{{ url_for('terms.terms_plot_tool') }}"><i class="fa fa-wrench"> </i> Terms plot tool</a></li>

2
var/www/static/js/plugins/d3/sparklines.js vendored
View File

@ -20,7 +20,7 @@ const sparkline = (container_id, data, options) => {
let maxY = d3.max(data, function(d) { return d } ); let maxY = d3.max(data, function(d) { return d } );
let x = d3.scaleLinear() let x = d3.scaleLinear()
.range([0, width_spark - 10]) .range([0, width_spark])
.domain([0,maxX]); .domain([0,maxX]);
let y = d3.scaleLinear() let y = d3.scaleLinear()

42
var/www/templates/hunter/menu_sidebar.html Normal file
View File

@ -0,0 +1,42 @@
<div class="col-12 col-lg-2 p-0 bg-light border-right" id="side_menu">
<button type="button" class="btn btn-outline-secondary mt-1 ml-3" onclick="toggle_sidebar()">
<i class="fas fa-align-left"></i>
<span>Toggle Sidebar</span>
</button>
<nav class="navbar navbar-expand navbar-light bg-light flex-md-column flex-row align-items-start py-2" id="nav_menu">
<h5 class="d-flex text-muted w-100">
<span>Trackers </span>
<a class="ml-auto" href="{{url_for('hunter.add_tracked_menu')}}">
<i class="fas fa-plus-circle ml-auto"></i>
</a>
</h5>
<ul class="nav flex-md-column flex-row navbar-nav justify-content-between w-100"> <!--nav-pills-->
<li class="nav-item">
<a class="nav-link" href="{{url_for('hunter.tracked_menu')}}" id="nav_tracker_">
<i class="fas fa-ruler-combined"></i>
<span>All Trackers</span>
</a>
</li>
<li class="nav-item">
<a class="nav-link" href="{{url_for('hunter.tracked_menu_word')}}" id="nav_tracker_word">
<i class="fas fa-font"></i>
<span>Tracked Words</span>
</a>
</li>
<li class="nav-item">
<a class="nav-link" href="{{url_for('hunter.tracked_menu_set')}}" id="nav_tracker_set">
<i class="fas fa-layer-group"></i>
<span>Tracked Set</span>
</a>
</li>
<li class="nav-item">
<a class="nav-link" href="{{url_for('hunter.tracked_menu_regex')}}" id="nav_tracker_regex">
<i class="fas fa-ruler"></i>
<span>Tracked Regex</span>
</a>
</li>
</ul>
</nav>
</div>

2
var/www/templates/nav_bar.html
View File

@ -19,7 +19,7 @@
<a class="nav-link" id="page-Browse-Items" href="{{ url_for('Tags.Tags_page') }}" aria-disabled="true"><i class="fas fa-tag"></i> Browse Items</a> <a class="nav-link" id="page-Browse-Items" href="{{ url_for('Tags.Tags_page') }}" aria-disabled="true"><i class="fas fa-tag"></i> Browse Items</a>
</li> </li>
<li class="nav-item mr-3"> <li class="nav-item mr-3">
<a class="nav-link" id="page-Tracker" href="{{ url_for('terms.tracked_term_menu') }}" aria-disabled="true"><i class="fas fa-crosshairs"></i> Leaks Hunter</a> <a class="nav-link" id="page-Tracker" href="{{ url_for('hunter.tracked_menu') }}" aria-disabled="true"><i class="fas fa-crosshairs"></i> Leaks Hunter</a>
</li> </li>
<li class="nav-item mr-3"> <li class="nav-item mr-3">
<a class="nav-link" id="page-Crawler" href="{{ url_for('hiddenServices.dashboard') }}" tabindex="-1" aria-disabled="true"><i class="fas fa-spider"></i> Crawlers</a> <a class="nav-link" id="page-Crawler" href="{{ url_for('hiddenServices.dashboard') }}" tabindex="-1" aria-disabled="true"><i class="fas fa-spider"></i> Crawlers</a>

24
var/www/templates/tracker/menu_sidebar.html
View File

@ -1,24 +0,0 @@
<div class="col-12 col-lg-2 p-0 bg-light border-right" id="side_menu">
<button type="button" class="btn btn-outline-secondary mt-1 ml-3" onclick="toggle_sidebar()">
<i class="fas fa-align-left"></i>
<span>Toggle Sidebar</span>
</button>
<nav class="navbar navbar-expand navbar-light bg-light flex-md-column flex-row align-items-start py-2" id="nav_menu">
<h5 class="d-flex text-muted w-100">
<span>Terms Tracker </span>
<a class="ml-auto" href="{{url_for('terms.add_tracked_term_menu')}}">
<i class="fas fa-plus-circle ml-auto"></i>
</a>
</h5>
<ul class="nav flex-md-column flex-row navbar-nav justify-content-between w-100"> <!--nav-pills-->
<li class="nav-item">
<a class="nav-link" href="{{url_for('terms.tracked_term_menu')}}" id="nav_dashboard">
<i class="fas fa-search"></i>
<span>Tracked Terms</span>
</a>
</li>
</ul>
</nav>
</div>