MISP
/
misp-dashboard
mirror of https://github.com/MISP/misp-dashboard
2
0
Fork 0
Code Issues Releases Wiki Activity

Merge branch 'master' of github.com:MISP/misp-dashboard into HEAD

diagnosticTool
mokaddem 2019-06-14 09:30:26 +02:00
parent ca75838896 16682247d9
commit c34d94cf7f
21 changed files with 325 additions and 152 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

101
README.md
View File

@ -1,23 +1,23 @@
# misp-dashboard
A dashboard showing live data and statistics from the ZMQ feeds of one or more [MISP](https://www.misp-project.org/) instances. The dashboard
can be used as a real-time situational awareness tool to gather threat intelligence information. The misp-dashboard includes
a gamification tool to show the contributions of each organisations and how they are ranked over time. The dashboard can be used for
SOC (Security Operation Center), security team or during cyber exercise to keep track of what's going on your various MISP instances.
A dashboard showing live data and statistics from the ZMQ feeds of one or more [MISP](https://www.misp-project.org/) instances.
The dashboard can be used as a real-time situational awareness tool to gather threat intelligence information.
The misp-dashboard includes a [gamification](https://en.wikipedia.org/wiki/Gamification#Criticism) tool to show the contributions of each organisation and how they are ranked over time.
The dashboard can be used for SOCs (Security Operation Centers), security teams or during cyber exercises to keep track of what is being processed on your various MISP instances.
# Features
## Live Dashboard
- Possibility to subscribe to multiple ZMQ feeds
- Shows direct contribution made by organisations
- Shows live resolvable posted locations
- Possibility to subscribe to multiple ZMQ feeds from different MISP instances
- Shows immediate contributions made by organisations
- Displays live resolvable posted geo-locations
![Dashboard live](./screenshots/dashboard-live.png)
## Geolocalisation Dashboard
- Provides historical geolocalised information to support security teams, CSIRTs or SOC finding threats in their constituency
- Provides historical geolocalised information to support security teams, CSIRTs or SOCs in finding threats within their constituency
- Possibility to get geospatial information from specific regions
![Dashbaord geo](./screenshots/dashboard-geo.png)
@ -25,25 +25,25 @@ SOC (Security Operation Center), security team or during cyber exercise to keep
## Contributors Dashboard
__Shows__:
- The monthly rank of all organisation
- The monthly rank of all organisations
- The last organisation that contributed (dynamic updates)
- The contribution level of all organisation
- Each category of contribution per organisation
- The contribution level of all organisations
- Each category of contributions per organisation
- The current ranking of the selected organisation (dynamic updates)
__Includes__:
- Gamification of the platform:
- [Gamification](https://en.wikipedia.org/wiki/Gamification#Criticism) of the platform:
- Two different levels of ranking with unique icons
- Exclusive obtainable badges for source code contributors and donator
![Dashboard contributor](./screenshots/dashboard-contributors2.png)
![Dashboard contributor2](./screenshots/dashboard-contributors3.png)
![Dashboard contributors](./screenshots/dashboard-contributors2.png)
![Dashboard contributors2](./screenshots/dashboard-contributors3.png)
## Users Dashboard
- Shows when and how the platform is used:
- Login punchcard and overtime
- Login punchcard and contributions over time
- Contribution vs login
![Dashboard users](./screenshots/dashboard-users.png)
@ -57,7 +57,7 @@ __Includes__:
![Dashboard users](./screenshots/dashboard-trendings.png)
# Installation
- Launch ```./install_dependencies.sh``` from the MISP-Dashboard directory
- Launch ```./install_dependencies.sh``` from the MISP-Dashboard directory ([idempotent-ish](https://en.wikipedia.org/wiki/Idempotence))
- Update the configuration file ```config.cfg``` so that it matches your system
- Fields that you may change:
- RedisGlobal -> host
@ -68,7 +68,7 @@ __Includes__:
# Updating by pulling
- Re-launch ```./install_dependencies.sh``` to fetch new required dependencies
- Re-update your configuration file ```config.cfg```
- Re-update your configuration file ```config.cfg``` by comparing eventual changes in ```config.cfg.default```
:warning: Make sure no zmq python3 scripts are running. They block the update.
@ -92,7 +92,7 @@ OSError: [Errno 26] Text file busy: '/home/steve/code/misp-dashboard/DASHENV/bin
```
# Starting the System
:warning: You do not need to run it as root. Normal privileges are fine.
:warning: You should not run it as root. Normal privileges are fine.
- Be sure to have a running redis server
- e.g. ```redis-server --port 6250```
@ -102,7 +102,7 @@ OSError: [Errno 26] Text file busy: '/home/steve/code/misp-dashboard/DASHENV/bin
- Start the Flask server ```./server.py &```
- Access the interface at ```http://localhost:8001/```
Alternatively, you can run the ```start_all.sh``` script to run the commands described above.
__Alternatively__, you can run the ```start_all.sh``` script to run the commands described above.
# Debug
@ -117,7 +117,7 @@ export FLASK_APP=server.py
flask run --host=0.0.0.0 --port=8001 # <- Be careful here, this exposes it on ALL ip addresses. Ideally if run locally --host=127.0.0.1
```
OR, just toggle the debug flag in start_all.sh script.
OR, just toggle the debug flag in start_all.sh or config.cfg.
Happy hacking ;)
@ -140,6 +140,25 @@ The misp-dashboard being stateless in regards to MISP, it can only process data
The most revelant example could be the user login punchcard. If your MISP doesn't have the option ``Plugin.ZeroMQ_audit_notifications_enable`` set to ``true``, the punchcard will be empty.
## Dashboard not showing results - No module named zmq
When the misp-dashboard does not show results then first check if the zmq module within MISP is properly installed.
In **Administration**, **Plugin Settings**, **ZeroMQ** check that **Plugin.ZeroMQ_enable** is set to **True**.
Publish a test event from MISP to ZMQ via **Event Actions**, **Publish event to ZMQ**.
Verify the logfiles
```
${PATH_TO_MISP}/app/tmp/log/mispzmq.error.log
${PATH_TO_MISP}/app/tmp/log/mispzmq.log
```
If there's an error **ModuleNotFoundError: No module named 'zmq'** then install pyzmq.
```
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install pyzmq
```
# zmq_subscriber options
```usage: zmq_subscriber.py [-h] [-n ZMQNAME] [-u ZMQURL]
@ -155,7 +174,7 @@ optional arguments:
# Deploy in production using mod_wsgi
Install Apache's mod-wsgi for Python3
Install Apache mod-wsgi for Python3
```bash
sudo apt-get install libapache2-mod-wsgi-py3
@ -170,7 +189,7 @@ The following NEW packages will be installed:
libapache2-mod-wsgi-py3
```
Configuration file `/etc/apache2/sites-available/misp-dashboard.conf` assumes that `misp-dashboard` is cloned into `var/www/misp-dashboard`. It runs as user `misp` in this example. Change the permissions to folder and files accordingly.
Configuration file `/etc/apache2/sites-available/misp-dashboard.conf` assumes that `misp-dashboard` is cloned into `/var/www/misp-dashboard`. It runs as user `misp` in this example. Change the permissions to your custom folder and files accordingly.
```
<VirtualHost *:8001>
@ -218,33 +237,35 @@ Configuration file `/etc/apache2/sites-available/misp-dashboard.conf` assumes th
# License
~~~~
Copyright (C) 2017-2019 CIRCL - Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d'Intérêt Economique)
Copyright (c) 2017-2019 Sami Mokaddem
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
~~~~
Images and logos are handmade for:
- rankingMISPOrg/
- rankingMISPMonthly/
- MISPHonorableIcons/
Note that:
- Part of ```MISPHonorableIcons/1.svg``` comes from [octicons.github.com](https://octicons.github.com/icon/git-pull-request/) (CC0 - No Rights Reserved)
- Part of ```MISPHonorableIcons/2.svg``` comes from [Zeptozephyr](https://zeptozephyr.deviantart.com/art/Vectored-Portal-Icons-207347804) (CC0 - No Rights Reserved)
- Part of ```MISPHonorableIcons/3.svg``` comes from [octicons.github.com](https://octicons.github.com/icon/git-pull-request/) (CC0 - No Rights Reserved)
- Part of ```MISPHonorableIcons/4.svg``` comes from [Zeptozephyr](https://zeptozephyr.deviantart.com/art/Vectored-Portal-Icons-207347804) & [octicons.github.com](https://octicons.github.com/icon/git-pull-request/) (CC0 - No Rights Reserved)
- Part of ```MISPHonorableIcons/5.svg``` comes from [Zeptozephyr](https://zeptozephyr.deviantart.com/art/Vectored-Portal-Icons-207347804) & [octicons.github.com](https://octicons.github.com/icon/git-pull-request/) (CC0 - No Rights Reserved)
```
Copyright (C) 2017-2018 CIRCL - Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d'Intérêt Economique)
Copyright (c) 2017-2018 Sami Mokaddem
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
```

8
clean.py
View File

@ -1,11 +1,11 @@
#!/usr/bin/env python3
from pprint import pprint
import os
import redis
import configparser
import argparse
import configparser
import os
from pprint import pprint
import redis
RED="\033[91m"
GREEN="\033[92m"

5
config/config.cfg.default
View File

@ -1,6 +1,7 @@
[Server]
host = localhost
port = 8001
debug = False
[Dashboard]
#hours
@ -33,7 +34,9 @@ additional_help_text = ["Sightings multiplies earned points by 2", "Editing an a
[Log]
directory=logs
filename=logs.log
dispatcher_filename=zmq_dispatcher.log
subscriber_filename=zmq_subscriber.log
helpers_filename=helpers.log
[RedisGlobal]
host=localhost

10
give_honors_to_org.py
View File

@ -1,9 +1,13 @@
#!/usr/bin/env python3.5
import os, sys, json
import datetime, time
import redis
import configparser
import datetime
import json
import os
import sys
import time
import redis
import util
from helpers import contributor_helper

25
helpers/contributor_helper.py
View File

@ -1,16 +1,20 @@
import util
from util import getZrange
import math, random
import time
import os
import configparser
import json
import datetime
import json
import logging
import math
import os
import random
import sys
import time
import redis
import util
from util import getZrange
from . import users_helper
KEYDAY = "CONTRIB_DAY" # To be used by other module
KEYALLORG = "CONTRIB_ALL_ORG" # To be used by other module
@ -30,11 +34,16 @@ class Contributor_helper:
#logger
logDir = cfg.get('Log', 'directory')
logfilename = cfg.get('Log', 'filename')
logfilename = cfg.get('Log', 'helpers_filename')
logPath = os.path.join(logDir, logfilename)
if not os.path.exists(logDir):
os.makedirs(logDir)
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
try:
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
except PermissionError as error:
print(error)
print("Please fix the above and try again.")
sys.exit(126)
self.logger = logging.getLogger(__name__)
#honorBadge

45
helpers/geo_helper.py
View File

@ -1,18 +1,22 @@
import math, random
import os
import datetime
import json
import datetime, time
import logging
import json
import redis
import math
import os
import random
import sys
import time
from collections import OrderedDict
import geoip2.database
import phonenumbers, pycountry
from phonenumbers import geocoder
import redis
import geoip2.database
import phonenumbers
import pycountry
import util
from helpers import live_helper
from phonenumbers import geocoder
class InvalidCoordinate(Exception):
pass
@ -29,11 +33,16 @@ class Geo_helper:
#logger
logDir = cfg.get('Log', 'directory')
logfilename = cfg.get('Log', 'filename')
logfilename = cfg.get('Log', 'helpers_filename')
logPath = os.path.join(logDir, logfilename)
if not os.path.exists(logDir):
os.makedirs(logDir)
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
try:
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
except PermissionError as error:
print(error)
print("Please fix the above and try again.")
sys.exit(126)
self.logger = logging.getLogger(__name__)
self.keyCategCoord = "GEO_COORD"
@ -43,7 +52,12 @@ class Geo_helper:
self.PATH_TO_JSON = cfg.get('RedisMap', 'path_countrycode_to_coord_JSON')
self.CHANNELDISP = cfg.get('RedisMap', 'channelDisp')
self.reader = geoip2.database.Reader(self.PATH_TO_DB)
try:
self.reader = geoip2.database.Reader(self.PATH_TO_DB)
except PermissionError as error:
print(error)
print("Please fix the above and try again.")
sys.exit(126)
self.country_to_iso = { country.name: country.alpha_2 for country in pycountry.countries}
with open(self.PATH_TO_JSON) as f:
self.country_code_to_coord = json.load(f)
@ -125,7 +139,7 @@ class Geo_helper:
self.live_helper.add_to_stream_log_cache('Map', j_to_send)
self.logger.info('Published: {}'.format(json.dumps(to_send)))
except ValueError:
self.logger.warning("can't resolve ip")
self.logger.warning("Can't resolve IP: " + str(supposed_ip))
except geoip2.errors.AddressNotFoundError:
self.logger.warning("Address not in Database")
except InvalidCoordinate:
@ -181,7 +195,12 @@ class Geo_helper:
now = datetime.datetime.now()
today_str = util.getDateStrFormat(now)
keyname = "{}:{}".format(keyCateg, today_str)
self.serv_redis_db.geoadd(keyname, lon, lat, content)
try:
self.serv_redis_db.geoadd(keyname, lon, lat, content)
except redis.exceptions.ResponseError as error:
print(error)
print("Please fix the above, and make sure you use a redis version that supports the GEOADD command.")
print("To test for support: echo \"help GEOADD\"| redis-cli")
self.logger.debug('Added to redis: keyname={}, lon={}, lat={}, content={}'.format(keyname, lon, lat, content))
def push_to_redis_zset(self, keyCateg, toAdd, endSubkey="", count=1):
now = datetime.datetime.now()

17
helpers/live_helper.py
View File

@ -1,8 +1,10 @@
import os
import datetime
import json
import random
import datetime, time
import logging
import os
import random
import sys
import time
class Live_helper:
@ -16,11 +18,16 @@ class Live_helper:
# logger
logDir = cfg.get('Log', 'directory')
logfilename = cfg.get('Log', 'filename')
logfilename = cfg.get('Log', 'helpers_filename')
logPath = os.path.join(logDir, logfilename)
if not os.path.exists(logDir):
os.makedirs(logDir)
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
try:
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
except PermissionError as error:
print(error)
print("Please fix the above and try again.")
sys.exit(126)
self.logger = logging.getLogger(__name__)
def publish_log(self, zmq_name, name, content, channel=None):

21
helpers/trendings_helper.py
View File

@ -1,13 +1,17 @@
import math, random
import os
import json
import copy
import datetime, time
import datetime
import json
import logging
import math
import os
import random
import sys
import time
from collections import OrderedDict
import util
class Trendings_helper:
def __init__(self, serv_redis_db, cfg):
self.serv_redis_db = serv_redis_db
@ -23,11 +27,16 @@ class Trendings_helper:
#logger
logDir = cfg.get('Log', 'directory')
logfilename = cfg.get('Log', 'filename')
logfilename = cfg.get('Log', 'helpers_filename')
logPath = os.path.join(logDir, logfilename)
if not os.path.exists(logDir):
os.makedirs(logDir)
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
try:
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
except PermissionError as error:
print(error)
print("Please fix the above and try again.")
sys.exit(126)
self.logger = logging.getLogger(__name__)
''' SETTER '''

25
helpers/users_helper.py
View File

@ -1,10 +1,14 @@
import math, random
import os
import datetime
import json
import datetime, time
import logging
import math
import os
import random
import sys
import time
import util
from . import contributor_helper
@ -20,11 +24,16 @@ class Users_helper:
#logger
logDir = cfg.get('Log', 'directory')
logfilename = cfg.get('Log', 'filename')
logfilename = cfg.get('Log', 'helpers_filename')
logPath = os.path.join(logDir, logfilename)
if not os.path.exists(logDir):
os.makedirs(logDir)
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
try:
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
except PermissionError as error:
print(error)
print("Please fix the above and try again.")
sys.exit(126)
self.logger = logging.getLogger(__name__)
def add_user_login(self, timestamp, org, email=''):
@ -63,11 +72,11 @@ class Users_helper:
else:
break # timestamps should be sorted, no need to process anymore
return to_return
# return: All dates for all orgs, if date is not supplied, return for all dates
def getUserLogins(self, date=None):
# get all orgs and retreive their timestamps
# get all orgs and retrieve their timestamps
dates = []
for org in self.getAllOrg():
keyname = "{}:{}".format(self.keyOrgLog, org)
@ -169,7 +178,7 @@ class Users_helper:
data = [data[6]]+data[:6]
return data
# return: a dico of the form {login: [[timestamp, count], ...], contrib: [[timestamp, 1/0], ...]}
# return: a dico of the form {login: [[timestamp, count], ...], contrib: [[timestamp, 1/0], ...]}
# either for all orgs or the supplied one
def getUserLoginsAndContribOvertime(self, date, org=None, prev_days=6):
dico_hours_contrib = {}

22
install_dependencies.sh
View File

@ -1,12 +1,21 @@
#!/bin/bash
set -e
## disable -e for production systems
#set -e
## Debug mode
#set -x
sudo apt-get install python3-virtualenv virtualenv screen redis-server unzip -y
if [ -z "$VIRTUAL_ENV" ]; then
virtualenv -p python3 DASHENV
virtualenv -p python3 DASHENV ; DASH_VENV=$?
if [[ "$DASH_VENV" != "0" ]]; then
echo "Something went wrong with either the update or install of the virtualenv."
echo "Please investigate manually."
exit $DASH_VENV
fi
. ./DASHENV/bin/activate
fi
@ -35,7 +44,14 @@ mkdir -p css fonts js
popd
mkdir -p temp
wget http://www.misp-project.org/assets/images/misp-small.png -O static/pics/MISP.png
NET_WGET=$(wget --no-cache -q https://www.misp-project.org/assets/images/misp-small.png -O static/pics/MISP.png; echo $?)
if [[ "$NET_WGET" != "0" ]]; then
echo "The first wget we tried failed, please investigate manually."
exit $NET_WGET
fi
wget https://www.misp-project.org/favicon.ico -O static/favicon.ico
# jquery
JQVERSION="3.2.1"

12
retreive_map_pic.py → retrieve_map_pic.py
View File

@ -1,13 +1,15 @@
#!/usr/bin/env python3.5
import redis
import requests
import shutil
import json
import math
import sys, os
import os
import shlex
import shutil
import sys
import time
from subprocess import PIPE, Popen
import shlex
import redis
import requests
URL_OPEN_MAP = "http://tile.openstreetmap.org/{zoom}/{x}/{y}.png"
MAP_DIR = "static/maps/"

47
server.py
View File

@ -1,21 +1,22 @@
#!/usr/bin/env python3
from flask import Flask, render_template, request, Response, jsonify, stream_with_context
import json
import redis
import random, math
import configparser
import datetime
import errno
import json
import logging
import math
import os
import random
from time import gmtime as now
from time import sleep, strftime
import datetime
import os
import logging
import redis
import util
from helpers import geo_helper
from helpers import contributor_helper
from helpers import users_helper
from helpers import trendings_helper
from helpers import live_helper
from flask import (Flask, Response, jsonify, render_template, request,
send_from_directory, stream_with_context)
from helpers import (contributor_helper, geo_helper, live_helper,
trendings_helper, users_helper)
configfile = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'config/config.cfg')
cfg = configparser.ConfigParser()
@ -26,6 +27,7 @@ logger.setLevel(logging.ERROR)
server_host = cfg.get("Server", "host")
server_port = cfg.getint("Server", "port")
server_debug = cfg.get("Server", "debug")
app = Flask(__name__)
@ -94,6 +96,12 @@ class LogItem():
to_add = util.getFields(self.feed, field)
to_ret[i] = to_add if to_add is not None else ''
# Number to keep them sorted (jsonify sort keys)
for item in range(len(LogItem.FIELDNAME_ORDER)):
try:
to_ret[item] = self.fields[item]
except IndexError: # not enough field in rcv item
to_ret[item] = ''
return to_ret
@ -176,6 +184,10 @@ def index():
zoomlevel=cfg.getint('Dashboard' ,'zoomlevel')
)
@app.route('/favicon.ico')
def favicon():
return send_from_directory(os.path.join(app.root_path, 'static'),
'favicon.ico', mimetype='image/vnd.microsoft.icon')
@app.route("/geo")
def geo():
@ -640,4 +652,13 @@ def getGenericTrendingOvertime():
return jsonify(data)
if __name__ == '__main__':
app.run(host=server_host, port=server_port, threaded=True)
try:
app.run(host=server_host,
port=server_port,
debug=server_debug,
threaded=True)
except OSError as error:
if error.errno == 98:
print("\n\n\nAddress already in use, the defined port is: " + str(server_port))
else:
print(str(error))

25
start_all.sh
View File

@ -20,7 +20,22 @@ else
exit 1
fi
[ ! -f "`which redis-server`" ] && echo "'redis-server' is not installed/not on PATH. Please fix and run again." && exit 1
if [[ -f "/etc/redhat-release" ]]; then
echo "You are running a RedHat flavour. Detecting scl potential..."
if [[ -f "/usr/bin/scl" ]]; then
echo "scl detected, checking for redis-server"
SCL_REDIS=$(scl -l|grep rh-redis)
if [[ ! -z $SCL_REDIS ]]; then
echo "We detected: ${SCL_REDIS} acting accordingly"
REDIS_RUN="/usr/bin/scl enable ${SCL_REDIS}"
fi
else
echo "redis-server seems not to be install in scl, perhaps system-wide, testing."
[ ! -f "`which redis-server`" ] && echo "'redis-server' is not installed/not on PATH. Please fix and run again." && exit 1
fi
else
[ ! -f "`which redis-server`" ] && echo "'redis-server' is not installed/not on PATH. Please fix and run again." && exit 1
fi
netstat -an |grep LISTEN |grep 6250 |grep -v tcp6 ; check_redis_port=$?
netstat -an |grep LISTEN |grep 8001 |grep -v tcp6 ; check_dashboard_port=$?
@ -37,8 +52,12 @@ conf_dir="config/"
sleep 0.1
if [ "${check_redis_port}" == "1" ]; then
echo -e $GREEN"\t* Launching Redis servers"$DEFAULT
redis-server ${conf_dir}6250.conf &
echo -e $GREEN"\t* Launching Redis servers"$DEFAULT
if [[ ! -z $REDIS_RUN ]]; then
$REDIS_RUN "redis-server ${conf_dir}6250.conf" &
else
redis-server ${conf_dir}6250.conf &
fi
else
echo -e $RED"\t* NOT starting Redis server, made a very unrealiable check on port 6250, and something seems to be there… please double check if this is good!"$DEFAULT
fi

6
static/js/contrib.js
View File

@ -346,7 +346,8 @@ function addLastContributor(datatable, data, update) {
} else {
last_added_contrib = org;
var date = new Date(data.epoch*1000);
date.toString = function() {return this.toTimeString().slice(0,-15) +' '+ this.toLocaleDateString(); };
//date.toString = function() {return this.toTimeString().slice(0,-15) +' '+ this.toLocaleDateString(); };
date = date.getFullYear() + "-" + String(date.getMonth()).padStart(2, "0") + "-" + String(date.getDay()).padStart(2, "0") + "@" + String(date.getHours()).padStart(2, "0") + ":" + String(date.getMinutes()).padStart(2, "0");
var to_add = [
date,
data.pnts,
@ -383,7 +384,8 @@ function addAwards(datatableAwards, json, playAnim) {
var award = createTrophyImg(json.award[1][1], 40, categ);
}
var date = new Date(json.epoch*1000);
date.toString = function() {return this.toTimeString().slice(0,-15) +' '+ this.toLocaleDateString(); };
//date.toString = function() {return this.toTimeString().slice(0,-15) +' '+ this.toLocaleDateString(); };
date = date.getFullYear() + "-" + String(date.getMonth()).padStart(2, "0") + "-" + String(date.getDay()).padStart(2, "0") + "@" + String(date.getHours()).padStart(2, "0") + ":" + String(date.getMinutes()).padStart(2, "0");
var to_add = [
date,
createImg(json.logo_path, 32),

2
static/js/trendings.js
View File

@ -145,7 +145,7 @@ function getTextColour(rgb) {
}
}
// If json (from tag), only retreive the name> otherwise return the supplied arg.
// If json (from tag), only retrieve the name> otherwise return the supplied arg.
function getOnlyName(potentialJson) {
try {
jsonLabel = JSON.parse(potentialJson);

10
tests/test_geo.py
View File

@ -1,8 +1,13 @@
#!/usr/bin/env python3.5
import configparser
import redis
import sys,os
import datetime
import os
import sys
import redis
from helpers import geo_helper
sys.path.append('..')
configfile = 'test_config.cfg'
@ -14,7 +19,6 @@ serv_redis_db = redis.StrictRedis(
port=6260,
db=1)
from helpers import geo_helper
geo_helper = geo_helper.Geo_helper(serv_redis_db, cfg)
categ = 'Network Activity'

11
tests/test_trendings.py
View File

@ -1,8 +1,14 @@
#!/usr/bin/env python3.5
import configparser
import datetime
import os
import sys
import time
import redis
import sys,os
import datetime, time
from helpers import trendings_helper
sys.path.append('..')
configfile = 'test_config.cfg'
@ -14,7 +20,6 @@ serv_redis_db = redis.StrictRedis(
port=6260,
db=1)
from helpers import trendings_helper
trendings_helper = trendings_helper.Trendings_helper(serv_redis_db, cfg)

11
tests/test_users.py
View File

@ -1,8 +1,14 @@
#!/usr/bin/env python3.5
import configparser
import datetime
import os
import sys
import time
import redis
import sys,os
import datetime, time
from helpers import users_helper
sys.path.append('..')
configfile = 'test_config.cfg'
@ -14,7 +20,6 @@ serv_redis_db = redis.StrictRedis(
port=6260,
db=1)
from helpers import users_helper
users_helper = users_helper.Users_helper(serv_redis_db, cfg)

3
util.py
View File

@ -1,5 +1,6 @@
import datetime
import time
from collections import defaultdict
import datetime, time
ONE_DAY = 60*60*24

41
zmq_dispatcher.py
View File

@ -1,34 +1,38 @@
#!/usr/bin/env python3
import time, datetime
import copy
import logging
import zmq
import redis
import random
import configparser
import argparse
import os
import sys
import configparser
import copy
import datetime
import json
import logging
import os
import random
import sys
import time
import redis
import zmq
import util
from helpers import geo_helper
from helpers import contributor_helper
from helpers import users_helper
from helpers import trendings_helper
from helpers import live_helper
from helpers import (contributor_helper, geo_helper, live_helper,
trendings_helper, users_helper)
configfile = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'config/config.cfg')
cfg = configparser.ConfigParser()
cfg.read(configfile)
logDir = cfg.get('Log', 'directory')
logfilename = cfg.get('Log', 'filename')
logfilename = cfg.get('Log', 'dispatcher_filename')
logPath = os.path.join(logDir, logfilename)
if not os.path.exists(logDir):
os.makedirs(logDir)
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
try:
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
except PermissionError as error:
print(error)
print("Please fix the above and try again.")
sys.exit(126)
logger = logging.getLogger('zmq_dispatcher')
LISTNAME = cfg.get('RedisLIST', 'listName')
@ -286,4 +290,7 @@ if __name__ == "__main__":
parser.add_argument('-s', '--sleep', required=False, dest='sleeptime', type=int, help='The number of second to wait before checking redis list size', default=5)
args = parser.parse_args()
main(args.sleeptime)
try:
main(args.sleeptime)
except (redis.exceptions.ResponseError, KeyboardInterrupt) as error:
print(error)

30
zmq_subscriber.py
View File

@ -1,24 +1,31 @@
#!/usr/bin/env python3
import time, datetime
import zmq
import logging
import redis
import configparser
import argparse
import configparser
import datetime
import json
import logging
import os
import sys
import json
import time
import redis
import zmq
configfile = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'config/config.cfg')
cfg = configparser.ConfigParser()
cfg.read(configfile)
logDir = cfg.get('Log', 'directory')
logfilename = cfg.get('Log', 'filename')
logfilename = cfg.get('Log', 'subscriber_filename')
logPath = os.path.join(logDir, logfilename)
if not os.path.exists(logDir):
os.makedirs(logDir)
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
try:
logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
except PermissionError as error:
print(error)
print("Please fix the above and try again.")
sys.exit(126)
logger = logging.getLogger('zmq_subscriber')
ZMQ_URL = cfg.get('RedisGlobal', 'zmq_url')
@ -57,9 +64,12 @@ def main(zmqName):
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='A zmq subscriber. It subscribes to a ZNQ then redispatch it to the misp-dashboard')
parser = argparse.ArgumentParser(description='A zmq subscriber. It subscribes to a ZMQ then redispatch it to the misp-dashboard')
parser.add_argument('-n', '--name', required=False, dest='zmqname', help='The ZMQ feed name', default="MISP Standard ZMQ")
parser.add_argument('-u', '--url', required=False, dest='zmqurl', help='The URL to connect to', default=ZMQ_URL)
args = parser.parse_args()
main(args.zmqname)
try:
main(args.zmqname)
except redis.exceptions.ResponseError as error:
print(error)