diff --git a/README.md b/README.md index 34bde55..0e8b9e9 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,23 @@ # misp-dashboard -A dashboard showing live data and statistics from the ZMQ feeds of one or more [MISP](https://www.misp-project.org/) instances. The dashboard -can be used as a real-time situational awareness tool to gather threat intelligence information. The misp-dashboard includes -a gamification tool to show the contributions of each organisations and how they are ranked over time. The dashboard can be used for -SOC (Security Operation Center), security team or during cyber exercise to keep track of what's going on your various MISP instances. +A dashboard showing live data and statistics from the ZMQ feeds of one or more [MISP](https://www.misp-project.org/) instances. +The dashboard can be used as a real-time situational awareness tool to gather threat intelligence information. +The misp-dashboard includes a [gamification](https://en.wikipedia.org/wiki/Gamification#Criticism) tool to show the contributions of each organisation and how they are ranked over time. +The dashboard can be used for SOCs (Security Operation Centers), security teams or during cyber exercises to keep track of what is being processed on your various MISP instances. # Features ## Live Dashboard -- Possibility to subscribe to multiple ZMQ feeds -- Shows direct contribution made by organisations -- Shows live resolvable posted locations +- Possibility to subscribe to multiple ZMQ feeds from different MISP instances +- Shows immediate contributions made by organisations +- Displays live resolvable posted geo-locations ![Dashboard live](./screenshots/dashboard-live.png) ## Geolocalisation Dashboard -- Provides historical geolocalised information to support security teams, CSIRTs or SOC finding threats in their constituency +- Provides historical geolocalised information to support security teams, CSIRTs or SOCs in finding threats within their constituency - Possibility to get geospatial information from specific regions ![Dashbaord geo](./screenshots/dashboard-geo.png) @@ -25,25 +25,25 @@ SOC (Security Operation Center), security team or during cyber exercise to keep ## Contributors Dashboard __Shows__: -- The monthly rank of all organisation +- The monthly rank of all organisations - The last organisation that contributed (dynamic updates) -- The contribution level of all organisation -- Each category of contribution per organisation +- The contribution level of all organisations +- Each category of contributions per organisation - The current ranking of the selected organisation (dynamic updates) __Includes__: -- Gamification of the platform: +- [Gamification](https://en.wikipedia.org/wiki/Gamification#Criticism) of the platform: - Two different levels of ranking with unique icons - Exclusive obtainable badges for source code contributors and donator -![Dashboard contributor](./screenshots/dashboard-contributors2.png) -![Dashboard contributor2](./screenshots/dashboard-contributors3.png) +![Dashboard contributors](./screenshots/dashboard-contributors2.png) +![Dashboard contributors2](./screenshots/dashboard-contributors3.png) ## Users Dashboard - Shows when and how the platform is used: - - Login punchcard and overtime + - Login punchcard and contributions over time - Contribution vs login ![Dashboard users](./screenshots/dashboard-users.png) @@ -57,7 +57,7 @@ __Includes__: ![Dashboard users](./screenshots/dashboard-trendings.png) # Installation -- Launch ```./install_dependencies.sh``` from the MISP-Dashboard directory +- Launch ```./install_dependencies.sh``` from the MISP-Dashboard directory ([idempotent-ish](https://en.wikipedia.org/wiki/Idempotence)) - Update the configuration file ```config.cfg``` so that it matches your system - Fields that you may change: - RedisGlobal -> host @@ -68,7 +68,7 @@ __Includes__: # Updating by pulling - Re-launch ```./install_dependencies.sh``` to fetch new required dependencies -- Re-update your configuration file ```config.cfg``` +- Re-update your configuration file ```config.cfg``` by comparing eventual changes in ```config.cfg.default``` :warning: Make sure no zmq python3 scripts are running. They block the update. @@ -92,7 +92,7 @@ OSError: [Errno 26] Text file busy: '/home/steve/code/misp-dashboard/DASHENV/bin ``` # Starting the System -:warning: You do not need to run it as root. Normal privileges are fine. +:warning: You should not run it as root. Normal privileges are fine. - Be sure to have a running redis server - e.g. ```redis-server --port 6250``` @@ -102,7 +102,7 @@ OSError: [Errno 26] Text file busy: '/home/steve/code/misp-dashboard/DASHENV/bin - Start the Flask server ```./server.py &``` - Access the interface at ```http://localhost:8001/``` -Alternatively, you can run the ```start_all.sh``` script to run the commands described above. +__Alternatively__, you can run the ```start_all.sh``` script to run the commands described above. # Debug @@ -117,7 +117,7 @@ export FLASK_APP=server.py flask run --host=0.0.0.0 --port=8001 # <- Be careful here, this exposes it on ALL ip addresses. Ideally if run locally --host=127.0.0.1 ``` -OR, just toggle the debug flag in start_all.sh script. +OR, just toggle the debug flag in start_all.sh or config.cfg. Happy hacking ;) @@ -140,6 +140,25 @@ The misp-dashboard being stateless in regards to MISP, it can only process data The most revelant example could be the user login punchcard. If your MISP doesn't have the option ``Plugin.ZeroMQ_audit_notifications_enable`` set to ``true``, the punchcard will be empty. +## Dashboard not showing results - No module named zmq +When the misp-dashboard does not show results then first check if the zmq module within MISP is properly installed. + +In **Administration**, **Plugin Settings**, **ZeroMQ** check that **Plugin.ZeroMQ_enable** is set to **True**. + +Publish a test event from MISP to ZMQ via **Event Actions**, **Publish event to ZMQ**. + +Verify the logfiles +``` +${PATH_TO_MISP}/app/tmp/log/mispzmq.error.log +${PATH_TO_MISP}/app/tmp/log/mispzmq.log +``` + +If there's an error **ModuleNotFoundError: No module named 'zmq'** then install pyzmq. + +``` +$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install pyzmq +``` + # zmq_subscriber options ```usage: zmq_subscriber.py [-h] [-n ZMQNAME] [-u ZMQURL] @@ -155,7 +174,7 @@ optional arguments: # Deploy in production using mod_wsgi -Install Apache's mod-wsgi for Python3 +Install Apache mod-wsgi for Python3 ```bash sudo apt-get install libapache2-mod-wsgi-py3 @@ -170,7 +189,7 @@ The following NEW packages will be installed: libapache2-mod-wsgi-py3 ``` -Configuration file `/etc/apache2/sites-available/misp-dashboard.conf` assumes that `misp-dashboard` is cloned into `var/www/misp-dashboard`. It runs as user `misp` in this example. Change the permissions to folder and files accordingly. +Configuration file `/etc/apache2/sites-available/misp-dashboard.conf` assumes that `misp-dashboard` is cloned into `/var/www/misp-dashboard`. It runs as user `misp` in this example. Change the permissions to your custom folder and files accordingly. ``` @@ -218,33 +237,35 @@ Configuration file `/etc/apache2/sites-available/misp-dashboard.conf` assumes th # License +~~~~ + Copyright (C) 2017-2019 CIRCL - Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d'Intérêt Economique) + Copyright (c) 2017-2019 Sami Mokaddem + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see . +~~~~ + Images and logos are handmade for: + - rankingMISPOrg/ - rankingMISPMonthly/ - MISPHonorableIcons/ Note that: + - Part of ```MISPHonorableIcons/1.svg``` comes from [octicons.github.com](https://octicons.github.com/icon/git-pull-request/) (CC0 - No Rights Reserved) - Part of ```MISPHonorableIcons/2.svg``` comes from [Zeptozephyr](https://zeptozephyr.deviantart.com/art/Vectored-Portal-Icons-207347804) (CC0 - No Rights Reserved) - Part of ```MISPHonorableIcons/3.svg``` comes from [octicons.github.com](https://octicons.github.com/icon/git-pull-request/) (CC0 - No Rights Reserved) - Part of ```MISPHonorableIcons/4.svg``` comes from [Zeptozephyr](https://zeptozephyr.deviantart.com/art/Vectored-Portal-Icons-207347804) & [octicons.github.com](https://octicons.github.com/icon/git-pull-request/) (CC0 - No Rights Reserved) - Part of ```MISPHonorableIcons/5.svg``` comes from [Zeptozephyr](https://zeptozephyr.deviantart.com/art/Vectored-Portal-Icons-207347804) & [octicons.github.com](https://octicons.github.com/icon/git-pull-request/) (CC0 - No Rights Reserved) -``` -Copyright (C) 2017-2018 CIRCL - Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d'Intérêt Economique) -Copyright (c) 2017-2018 Sami Mokaddem - - -This program is free software: you can redistribute it and/or modify -it under the terms of the GNU Affero General Public License as published by -the Free Software Foundation, either version 3 of the License, or -(at your option) any later version. - -This program is distributed in the hope that it will be useful, -but WITHOUT ANY WARRANTY; without even the implied warranty of -MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -GNU Affero General Public License for more details. - -You should have received a copy of the GNU Affero General Public License -along with this program. If not, see . -``` diff --git a/clean.py b/clean.py index 061321d..0db079c 100755 --- a/clean.py +++ b/clean.py @@ -1,11 +1,11 @@ #!/usr/bin/env python3 -from pprint import pprint -import os -import redis -import configparser import argparse +import configparser +import os +from pprint import pprint +import redis RED="\033[91m" GREEN="\033[92m" diff --git a/config/config.cfg.default b/config/config.cfg.default index 1d18adb..878e71e 100644 --- a/config/config.cfg.default +++ b/config/config.cfg.default @@ -1,6 +1,7 @@ [Server] host = localhost port = 8001 +debug = False [Dashboard] #hours @@ -33,7 +34,9 @@ additional_help_text = ["Sightings multiplies earned points by 2", "Editing an a [Log] directory=logs -filename=logs.log +dispatcher_filename=zmq_dispatcher.log +subscriber_filename=zmq_subscriber.log +helpers_filename=helpers.log [RedisGlobal] host=localhost diff --git a/give_honors_to_org.py b/give_honors_to_org.py index f154f73..9279e64 100755 --- a/give_honors_to_org.py +++ b/give_honors_to_org.py @@ -1,9 +1,13 @@ #!/usr/bin/env python3.5 -import os, sys, json -import datetime, time -import redis import configparser +import datetime +import json +import os +import sys +import time + +import redis import util from helpers import contributor_helper diff --git a/helpers/contributor_helper.py b/helpers/contributor_helper.py index 7b80234..bc4813b 100644 --- a/helpers/contributor_helper.py +++ b/helpers/contributor_helper.py @@ -1,16 +1,20 @@ -import util -from util import getZrange -import math, random -import time -import os import configparser -import json import datetime +import json import logging +import math +import os +import random +import sys +import time + import redis import util +from util import getZrange + from . import users_helper + KEYDAY = "CONTRIB_DAY" # To be used by other module KEYALLORG = "CONTRIB_ALL_ORG" # To be used by other module @@ -30,11 +34,16 @@ class Contributor_helper: #logger logDir = cfg.get('Log', 'directory') - logfilename = cfg.get('Log', 'filename') + logfilename = cfg.get('Log', 'helpers_filename') logPath = os.path.join(logDir, logfilename) if not os.path.exists(logDir): os.makedirs(logDir) - logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) + try: + logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) + except PermissionError as error: + print(error) + print("Please fix the above and try again.") + sys.exit(126) self.logger = logging.getLogger(__name__) #honorBadge diff --git a/helpers/geo_helper.py b/helpers/geo_helper.py index d45e6f3..ededda2 100644 --- a/helpers/geo_helper.py +++ b/helpers/geo_helper.py @@ -1,18 +1,22 @@ -import math, random -import os +import datetime import json -import datetime, time import logging -import json -import redis +import math +import os +import random +import sys +import time from collections import OrderedDict -import geoip2.database -import phonenumbers, pycountry -from phonenumbers import geocoder +import redis +import geoip2.database +import phonenumbers +import pycountry import util from helpers import live_helper +from phonenumbers import geocoder + class InvalidCoordinate(Exception): pass @@ -29,11 +33,16 @@ class Geo_helper: #logger logDir = cfg.get('Log', 'directory') - logfilename = cfg.get('Log', 'filename') + logfilename = cfg.get('Log', 'helpers_filename') logPath = os.path.join(logDir, logfilename) if not os.path.exists(logDir): os.makedirs(logDir) - logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) + try: + logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) + except PermissionError as error: + print(error) + print("Please fix the above and try again.") + sys.exit(126) self.logger = logging.getLogger(__name__) self.keyCategCoord = "GEO_COORD" @@ -43,7 +52,12 @@ class Geo_helper: self.PATH_TO_JSON = cfg.get('RedisMap', 'path_countrycode_to_coord_JSON') self.CHANNELDISP = cfg.get('RedisMap', 'channelDisp') - self.reader = geoip2.database.Reader(self.PATH_TO_DB) + try: + self.reader = geoip2.database.Reader(self.PATH_TO_DB) + except PermissionError as error: + print(error) + print("Please fix the above and try again.") + sys.exit(126) self.country_to_iso = { country.name: country.alpha_2 for country in pycountry.countries} with open(self.PATH_TO_JSON) as f: self.country_code_to_coord = json.load(f) @@ -125,7 +139,7 @@ class Geo_helper: self.live_helper.add_to_stream_log_cache('Map', j_to_send) self.logger.info('Published: {}'.format(json.dumps(to_send))) except ValueError: - self.logger.warning("can't resolve ip") + self.logger.warning("Can't resolve IP: " + str(supposed_ip)) except geoip2.errors.AddressNotFoundError: self.logger.warning("Address not in Database") except InvalidCoordinate: @@ -181,7 +195,12 @@ class Geo_helper: now = datetime.datetime.now() today_str = util.getDateStrFormat(now) keyname = "{}:{}".format(keyCateg, today_str) - self.serv_redis_db.geoadd(keyname, lon, lat, content) + try: + self.serv_redis_db.geoadd(keyname, lon, lat, content) + except redis.exceptions.ResponseError as error: + print(error) + print("Please fix the above, and make sure you use a redis version that supports the GEOADD command.") + print("To test for support: echo \"help GEOADD\"| redis-cli") self.logger.debug('Added to redis: keyname={}, lon={}, lat={}, content={}'.format(keyname, lon, lat, content)) def push_to_redis_zset(self, keyCateg, toAdd, endSubkey="", count=1): now = datetime.datetime.now() diff --git a/helpers/live_helper.py b/helpers/live_helper.py index 071423b..8a5c2dc 100644 --- a/helpers/live_helper.py +++ b/helpers/live_helper.py @@ -1,8 +1,10 @@ -import os +import datetime import json -import random -import datetime, time import logging +import os +import random +import sys +import time class Live_helper: @@ -16,11 +18,16 @@ class Live_helper: # logger logDir = cfg.get('Log', 'directory') - logfilename = cfg.get('Log', 'filename') + logfilename = cfg.get('Log', 'helpers_filename') logPath = os.path.join(logDir, logfilename) if not os.path.exists(logDir): os.makedirs(logDir) - logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) + try: + logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) + except PermissionError as error: + print(error) + print("Please fix the above and try again.") + sys.exit(126) self.logger = logging.getLogger(__name__) def publish_log(self, zmq_name, name, content, channel=None): diff --git a/helpers/trendings_helper.py b/helpers/trendings_helper.py index 61e1479..4b55423 100644 --- a/helpers/trendings_helper.py +++ b/helpers/trendings_helper.py @@ -1,13 +1,17 @@ -import math, random -import os -import json import copy -import datetime, time +import datetime +import json import logging +import math +import os +import random +import sys +import time from collections import OrderedDict import util + class Trendings_helper: def __init__(self, serv_redis_db, cfg): self.serv_redis_db = serv_redis_db @@ -23,11 +27,16 @@ class Trendings_helper: #logger logDir = cfg.get('Log', 'directory') - logfilename = cfg.get('Log', 'filename') + logfilename = cfg.get('Log', 'helpers_filename') logPath = os.path.join(logDir, logfilename) if not os.path.exists(logDir): os.makedirs(logDir) - logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) + try: + logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) + except PermissionError as error: + print(error) + print("Please fix the above and try again.") + sys.exit(126) self.logger = logging.getLogger(__name__) ''' SETTER ''' diff --git a/helpers/users_helper.py b/helpers/users_helper.py index 5fe8b90..ba3ab22 100644 --- a/helpers/users_helper.py +++ b/helpers/users_helper.py @@ -1,10 +1,14 @@ -import math, random -import os +import datetime import json -import datetime, time import logging +import math +import os +import random +import sys +import time import util + from . import contributor_helper @@ -20,11 +24,16 @@ class Users_helper: #logger logDir = cfg.get('Log', 'directory') - logfilename = cfg.get('Log', 'filename') + logfilename = cfg.get('Log', 'helpers_filename') logPath = os.path.join(logDir, logfilename) if not os.path.exists(logDir): os.makedirs(logDir) - logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) + try: + logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) + except PermissionError as error: + print(error) + print("Please fix the above and try again.") + sys.exit(126) self.logger = logging.getLogger(__name__) def add_user_login(self, timestamp, org, email=''): @@ -63,11 +72,11 @@ class Users_helper: else: break # timestamps should be sorted, no need to process anymore return to_return - + # return: All dates for all orgs, if date is not supplied, return for all dates def getUserLogins(self, date=None): - # get all orgs and retreive their timestamps + # get all orgs and retrieve their timestamps dates = [] for org in self.getAllOrg(): keyname = "{}:{}".format(self.keyOrgLog, org) @@ -169,7 +178,7 @@ class Users_helper: data = [data[6]]+data[:6] return data - # return: a dico of the form {login: [[timestamp, count], ...], contrib: [[timestamp, 1/0], ...]} + # return: a dico of the form {login: [[timestamp, count], ...], contrib: [[timestamp, 1/0], ...]} # either for all orgs or the supplied one def getUserLoginsAndContribOvertime(self, date, org=None, prev_days=6): dico_hours_contrib = {} diff --git a/install_dependencies.sh b/install_dependencies.sh index e9beb8e..3a0747d 100755 --- a/install_dependencies.sh +++ b/install_dependencies.sh @@ -1,12 +1,21 @@ #!/bin/bash -set -e +## disable -e for production systems +#set -e + +## Debug mode #set -x sudo apt-get install python3-virtualenv virtualenv screen redis-server unzip -y if [ -z "$VIRTUAL_ENV" ]; then - virtualenv -p python3 DASHENV + virtualenv -p python3 DASHENV ; DASH_VENV=$? + + if [[ "$DASH_VENV" != "0" ]]; then + echo "Something went wrong with either the update or install of the virtualenv." + echo "Please investigate manually." + exit $DASH_VENV + fi . ./DASHENV/bin/activate fi @@ -35,7 +44,14 @@ mkdir -p css fonts js popd mkdir -p temp -wget http://www.misp-project.org/assets/images/misp-small.png -O static/pics/MISP.png +NET_WGET=$(wget --no-cache -q https://www.misp-project.org/assets/images/misp-small.png -O static/pics/MISP.png; echo $?) + +if [[ "$NET_WGET" != "0" ]]; then + echo "The first wget we tried failed, please investigate manually." + exit $NET_WGET +fi + +wget https://www.misp-project.org/favicon.ico -O static/favicon.ico # jquery JQVERSION="3.2.1" diff --git a/retreive_map_pic.py b/retrieve_map_pic.py similarity index 98% rename from retreive_map_pic.py rename to retrieve_map_pic.py index 55929a1..e8dd22e 100755 --- a/retreive_map_pic.py +++ b/retrieve_map_pic.py @@ -1,13 +1,15 @@ #!/usr/bin/env python3.5 -import redis -import requests -import shutil import json import math -import sys, os +import os +import shlex +import shutil +import sys import time from subprocess import PIPE, Popen -import shlex + +import redis +import requests URL_OPEN_MAP = "http://tile.openstreetmap.org/{zoom}/{x}/{y}.png" MAP_DIR = "static/maps/" diff --git a/server.py b/server.py index 9246d5f..3ee4344 100755 --- a/server.py +++ b/server.py @@ -1,21 +1,22 @@ #!/usr/bin/env python3 -from flask import Flask, render_template, request, Response, jsonify, stream_with_context -import json -import redis -import random, math import configparser +import datetime +import errno +import json +import logging +import math +import os +import random from time import gmtime as now from time import sleep, strftime -import datetime -import os -import logging + +import redis import util -from helpers import geo_helper -from helpers import contributor_helper -from helpers import users_helper -from helpers import trendings_helper -from helpers import live_helper +from flask import (Flask, Response, jsonify, render_template, request, + send_from_directory, stream_with_context) +from helpers import (contributor_helper, geo_helper, live_helper, + trendings_helper, users_helper) configfile = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'config/config.cfg') cfg = configparser.ConfigParser() @@ -26,6 +27,7 @@ logger.setLevel(logging.ERROR) server_host = cfg.get("Server", "host") server_port = cfg.getint("Server", "port") +server_debug = cfg.get("Server", "debug") app = Flask(__name__) @@ -94,6 +96,12 @@ class LogItem(): to_add = util.getFields(self.feed, field) to_ret[i] = to_add if to_add is not None else '' + # Number to keep them sorted (jsonify sort keys) + for item in range(len(LogItem.FIELDNAME_ORDER)): + try: + to_ret[item] = self.fields[item] + except IndexError: # not enough field in rcv item + to_ret[item] = '' return to_ret @@ -176,6 +184,10 @@ def index(): zoomlevel=cfg.getint('Dashboard' ,'zoomlevel') ) +@app.route('/favicon.ico') +def favicon(): + return send_from_directory(os.path.join(app.root_path, 'static'), + 'favicon.ico', mimetype='image/vnd.microsoft.icon') @app.route("/geo") def geo(): @@ -640,4 +652,13 @@ def getGenericTrendingOvertime(): return jsonify(data) if __name__ == '__main__': - app.run(host=server_host, port=server_port, threaded=True) + try: + app.run(host=server_host, + port=server_port, + debug=server_debug, + threaded=True) + except OSError as error: + if error.errno == 98: + print("\n\n\nAddress already in use, the defined port is: " + str(server_port)) + else: + print(str(error)) diff --git a/start_all.sh b/start_all.sh index ef00155..08b1800 100755 --- a/start_all.sh +++ b/start_all.sh @@ -20,7 +20,22 @@ else exit 1 fi -[ ! -f "`which redis-server`" ] && echo "'redis-server' is not installed/not on PATH. Please fix and run again." && exit 1 +if [[ -f "/etc/redhat-release" ]]; then + echo "You are running a RedHat flavour. Detecting scl potential..." + if [[ -f "/usr/bin/scl" ]]; then + echo "scl detected, checking for redis-server" + SCL_REDIS=$(scl -l|grep rh-redis) + if [[ ! -z $SCL_REDIS ]]; then + echo "We detected: ${SCL_REDIS} acting accordingly" + REDIS_RUN="/usr/bin/scl enable ${SCL_REDIS}" + fi + else + echo "redis-server seems not to be install in scl, perhaps system-wide, testing." + [ ! -f "`which redis-server`" ] && echo "'redis-server' is not installed/not on PATH. Please fix and run again." && exit 1 + fi +else + [ ! -f "`which redis-server`" ] && echo "'redis-server' is not installed/not on PATH. Please fix and run again." && exit 1 +fi netstat -an |grep LISTEN |grep 6250 |grep -v tcp6 ; check_redis_port=$? netstat -an |grep LISTEN |grep 8001 |grep -v tcp6 ; check_dashboard_port=$? @@ -37,8 +52,12 @@ conf_dir="config/" sleep 0.1 if [ "${check_redis_port}" == "1" ]; then - echo -e $GREEN"\t* Launching Redis servers"$DEFAULT - redis-server ${conf_dir}6250.conf & + echo -e $GREEN"\t* Launching Redis servers"$DEFAULT + if [[ ! -z $REDIS_RUN ]]; then + $REDIS_RUN "redis-server ${conf_dir}6250.conf" & + else + redis-server ${conf_dir}6250.conf & + fi else echo -e $RED"\t* NOT starting Redis server, made a very unrealiable check on port 6250, and something seems to be there… please double check if this is good!"$DEFAULT fi diff --git a/static/js/contrib.js b/static/js/contrib.js index 99f3fec..9e8fd67 100644 --- a/static/js/contrib.js +++ b/static/js/contrib.js @@ -346,7 +346,8 @@ function addLastContributor(datatable, data, update) { } else { last_added_contrib = org; var date = new Date(data.epoch*1000); - date.toString = function() {return this.toTimeString().slice(0,-15) +' '+ this.toLocaleDateString(); }; + //date.toString = function() {return this.toTimeString().slice(0,-15) +' '+ this.toLocaleDateString(); }; + date = date.getFullYear() + "-" + String(date.getMonth()).padStart(2, "0") + "-" + String(date.getDay()).padStart(2, "0") + "@" + String(date.getHours()).padStart(2, "0") + ":" + String(date.getMinutes()).padStart(2, "0"); var to_add = [ date, data.pnts, @@ -383,7 +384,8 @@ function addAwards(datatableAwards, json, playAnim) { var award = createTrophyImg(json.award[1][1], 40, categ); } var date = new Date(json.epoch*1000); - date.toString = function() {return this.toTimeString().slice(0,-15) +' '+ this.toLocaleDateString(); }; + //date.toString = function() {return this.toTimeString().slice(0,-15) +' '+ this.toLocaleDateString(); }; + date = date.getFullYear() + "-" + String(date.getMonth()).padStart(2, "0") + "-" + String(date.getDay()).padStart(2, "0") + "@" + String(date.getHours()).padStart(2, "0") + ":" + String(date.getMinutes()).padStart(2, "0"); var to_add = [ date, createImg(json.logo_path, 32), diff --git a/static/js/trendings.js b/static/js/trendings.js index d87aeae..b2ea959 100644 --- a/static/js/trendings.js +++ b/static/js/trendings.js @@ -145,7 +145,7 @@ function getTextColour(rgb) { } } -// If json (from tag), only retreive the name> otherwise return the supplied arg. +// If json (from tag), only retrieve the name> otherwise return the supplied arg. function getOnlyName(potentialJson) { try { jsonLabel = JSON.parse(potentialJson); diff --git a/tests/test_geo.py b/tests/test_geo.py index 94b9801..4c75b3d 100755 --- a/tests/test_geo.py +++ b/tests/test_geo.py @@ -1,8 +1,13 @@ #!/usr/bin/env python3.5 import configparser -import redis -import sys,os import datetime +import os +import sys + +import redis + +from helpers import geo_helper + sys.path.append('..') configfile = 'test_config.cfg' @@ -14,7 +19,6 @@ serv_redis_db = redis.StrictRedis( port=6260, db=1) -from helpers import geo_helper geo_helper = geo_helper.Geo_helper(serv_redis_db, cfg) categ = 'Network Activity' diff --git a/tests/test_trendings.py b/tests/test_trendings.py index 4cce0e7..4716e05 100755 --- a/tests/test_trendings.py +++ b/tests/test_trendings.py @@ -1,8 +1,14 @@ #!/usr/bin/env python3.5 import configparser +import datetime +import os +import sys +import time + import redis -import sys,os -import datetime, time + +from helpers import trendings_helper + sys.path.append('..') configfile = 'test_config.cfg' @@ -14,7 +20,6 @@ serv_redis_db = redis.StrictRedis( port=6260, db=1) -from helpers import trendings_helper trendings_helper = trendings_helper.Trendings_helper(serv_redis_db, cfg) diff --git a/tests/test_users.py b/tests/test_users.py index 61e7b41..2eb6f71 100755 --- a/tests/test_users.py +++ b/tests/test_users.py @@ -1,8 +1,14 @@ #!/usr/bin/env python3.5 import configparser +import datetime +import os +import sys +import time + import redis -import sys,os -import datetime, time + +from helpers import users_helper + sys.path.append('..') configfile = 'test_config.cfg' @@ -14,7 +20,6 @@ serv_redis_db = redis.StrictRedis( port=6260, db=1) -from helpers import users_helper users_helper = users_helper.Users_helper(serv_redis_db, cfg) diff --git a/util.py b/util.py index 78ded1b..8e0661f 100644 --- a/util.py +++ b/util.py @@ -1,5 +1,6 @@ +import datetime +import time from collections import defaultdict -import datetime, time ONE_DAY = 60*60*24 diff --git a/zmq_dispatcher.py b/zmq_dispatcher.py index 1359d03..afe7cf5 100755 --- a/zmq_dispatcher.py +++ b/zmq_dispatcher.py @@ -1,34 +1,38 @@ #!/usr/bin/env python3 -import time, datetime -import copy -import logging -import zmq -import redis -import random -import configparser import argparse -import os -import sys +import configparser +import copy +import datetime import json +import logging +import os +import random +import sys +import time + +import redis +import zmq import util -from helpers import geo_helper -from helpers import contributor_helper -from helpers import users_helper -from helpers import trendings_helper -from helpers import live_helper +from helpers import (contributor_helper, geo_helper, live_helper, + trendings_helper, users_helper) configfile = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'config/config.cfg') cfg = configparser.ConfigParser() cfg.read(configfile) logDir = cfg.get('Log', 'directory') -logfilename = cfg.get('Log', 'filename') +logfilename = cfg.get('Log', 'dispatcher_filename') logPath = os.path.join(logDir, logfilename) if not os.path.exists(logDir): os.makedirs(logDir) -logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) +try: + logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) +except PermissionError as error: + print(error) + print("Please fix the above and try again.") + sys.exit(126) logger = logging.getLogger('zmq_dispatcher') LISTNAME = cfg.get('RedisLIST', 'listName') @@ -286,4 +290,7 @@ if __name__ == "__main__": parser.add_argument('-s', '--sleep', required=False, dest='sleeptime', type=int, help='The number of second to wait before checking redis list size', default=5) args = parser.parse_args() - main(args.sleeptime) + try: + main(args.sleeptime) + except (redis.exceptions.ResponseError, KeyboardInterrupt) as error: + print(error) diff --git a/zmq_subscriber.py b/zmq_subscriber.py index b642333..ad9e548 100755 --- a/zmq_subscriber.py +++ b/zmq_subscriber.py @@ -1,24 +1,31 @@ #!/usr/bin/env python3 -import time, datetime -import zmq -import logging -import redis -import configparser import argparse +import configparser +import datetime +import json +import logging import os import sys -import json +import time + +import redis +import zmq configfile = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'config/config.cfg') cfg = configparser.ConfigParser() cfg.read(configfile) logDir = cfg.get('Log', 'directory') -logfilename = cfg.get('Log', 'filename') +logfilename = cfg.get('Log', 'subscriber_filename') logPath = os.path.join(logDir, logfilename) if not os.path.exists(logDir): os.makedirs(logDir) -logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) +try: + logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO) +except PermissionError as error: + print(error) + print("Please fix the above and try again.") + sys.exit(126) logger = logging.getLogger('zmq_subscriber') ZMQ_URL = cfg.get('RedisGlobal', 'zmq_url') @@ -57,9 +64,12 @@ def main(zmqName): if __name__ == "__main__": - parser = argparse.ArgumentParser(description='A zmq subscriber. It subscribes to a ZNQ then redispatch it to the misp-dashboard') + parser = argparse.ArgumentParser(description='A zmq subscriber. It subscribes to a ZMQ then redispatch it to the misp-dashboard') parser.add_argument('-n', '--name', required=False, dest='zmqname', help='The ZMQ feed name', default="MISP Standard ZMQ") parser.add_argument('-u', '--url', required=False, dest='zmqurl', help='The URL to connect to', default=ZMQ_URL) args = parser.parse_args() - main(args.zmqname) + try: + main(args.zmqname) + except redis.exceptions.ResponseError as error: + print(error)