Merge branch 'master' of github.com:MISP/misp-dashboard into HEAD

README.md

@@ -1,23 +1,23 @@
 # misp-dashboard
 
-A dashboard showing live data and statistics from the ZMQ feeds of one or more [MISP](https://www.misp-project.org/) instances. The dashboard
+A dashboard showing live data and statistics from the ZMQ feeds of one or more [MISP](https://www.misp-project.org/) instances.
-can be used as a real-time situational awareness tool to gather threat intelligence information. The misp-dashboard includes
+The dashboard can be used as a real-time situational awareness tool to gather threat intelligence information.
-a gamification tool to show the contributions of each organisations and how they are ranked over time. The dashboard can be used for
+The misp-dashboard includes a [gamification](https://en.wikipedia.org/wiki/Gamification#Criticism) tool to show the contributions of each organisation and how they are ranked over time.
-SOC (Security Operation Center), security team or during cyber exercise to keep track of what's going on your various MISP instances.
+The dashboard can be used for SOCs (Security Operation Centers), security teams or during cyber exercises to keep track of what is being processed on your various MISP instances.
 
 # Features
 
 ## Live Dashboard
 
-- Possibility to subscribe to multiple ZMQ feeds
+- Possibility to subscribe to multiple ZMQ feeds from different MISP instances
-- Shows direct contribution made by organisations
+- Shows immediate contributions made by organisations
-- Shows live resolvable posted locations
+- Displays live resolvable posted geo-locations
 
 ![Dashboard live](./screenshots/dashboard-live.png)
 
 ## Geolocalisation Dashboard
 
-- Provides historical geolocalised information to support security teams, CSIRTs or SOC finding threats in their constituency
+- Provides historical geolocalised information to support security teams, CSIRTs or SOCs in finding threats within their constituency
 - Possibility to get geospatial information from specific regions
 
 ![Dashbaord geo](./screenshots/dashboard-geo.png)
@@ -25,25 +25,25 @@ SOC (Security Operation Center), security team or during cyber exercise to keep
 ## Contributors Dashboard
 
 __Shows__:
-- The monthly rank of all organisation
+- The monthly rank of all organisations
 - The last organisation that contributed (dynamic updates)
-- The contribution level of all organisation
+- The contribution level of all organisations
-- Each category of contribution per organisation
+- Each category of contributions per organisation
 - The current ranking of the selected organisation (dynamic updates)
 
 __Includes__:
 
-- Gamification of the platform:
+- [Gamification](https://en.wikipedia.org/wiki/Gamification#Criticism) of the platform:
   - Two different levels of ranking with unique icons
   - Exclusive obtainable badges for source code contributors and donator
 
-![Dashboard contributor](./screenshots/dashboard-contributors2.png)
+![Dashboard contributors](./screenshots/dashboard-contributors2.png)
-![Dashboard contributor2](./screenshots/dashboard-contributors3.png)
+![Dashboard contributors2](./screenshots/dashboard-contributors3.png)
 
 ## Users Dashboard
 
 - Shows when and how the platform is used:
-    - Login punchcard and overtime
+    - Login punchcard and contributions over time
     - Contribution vs login
 
 ![Dashboard users](./screenshots/dashboard-users.png)
@@ -57,7 +57,7 @@ __Includes__:
 ![Dashboard users](./screenshots/dashboard-trendings.png)
 
 # Installation
-- Launch ```./install_dependencies.sh``` from the MISP-Dashboard directory
+- Launch ```./install_dependencies.sh``` from the MISP-Dashboard directory ([idempotent-ish](https://en.wikipedia.org/wiki/Idempotence))
 - Update the configuration file ```config.cfg``` so that it matches your system
   - Fields that you may change:
     - RedisGlobal -> host
@@ -68,7 +68,7 @@ __Includes__:
 
 # Updating by pulling
 - Re-launch ```./install_dependencies.sh``` to fetch new required dependencies
-- Re-update your configuration file ```config.cfg```
+- Re-update your configuration file ```config.cfg``` by comparing eventual changes in ```config.cfg.default```
 
 :warning: Make sure no zmq python3 scripts are running. They block the update.
 
@@ -92,7 +92,7 @@ OSError: [Errno 26] Text file busy: '/home/steve/code/misp-dashboard/DASHENV/bin
 ```
 
 # Starting the System
-:warning: You do not need to run it as root. Normal privileges are fine.
+:warning: You should not run it as root. Normal privileges are fine.
 
 - Be sure to have a running redis server
     - e.g. ```redis-server --port 6250```
@@ -102,7 +102,7 @@ OSError: [Errno 26] Text file busy: '/home/steve/code/misp-dashboard/DASHENV/bin
 - Start the Flask server ```./server.py &```
 - Access the interface at ```http://localhost:8001/```
 
-Alternatively, you can run the ```start_all.sh``` script to run the commands described above.
+__Alternatively__, you can run the ```start_all.sh``` script to run the commands described above.
 
 # Debug
 
@@ -117,7 +117,7 @@ export FLASK_APP=server.py
 flask run --host=0.0.0.0 --port=8001 # <- Be careful here, this exposes it on ALL ip addresses. Ideally if run locally --host=127.0.0.1
 ```
 
-OR, just toggle the debug flag in start_all.sh script.
+OR, just toggle the debug flag in start_all.sh or config.cfg.
 
 Happy hacking ;)
 
@@ -140,6 +140,25 @@ The misp-dashboard being stateless in regards to MISP, it can only process data
 
 The most revelant example could be the user login punchcard. If your MISP doesn't have the option ``Plugin.ZeroMQ_audit_notifications_enable`` set to ``true``, the punchcard will be empty.
 
+## Dashboard not showing results - No module named zmq
+When the misp-dashboard does not show results then first check if the zmq module within MISP is properly installed. 
+
+In **Administration**, **Plugin Settings**, **ZeroMQ** check that **Plugin.ZeroMQ_enable** is set to **True**.
+
+Publish a test event from MISP to ZMQ via **Event Actions**, **Publish event to ZMQ**.
+
+Verify the logfiles
+```
+${PATH_TO_MISP}/app/tmp/log/mispzmq.error.log
+${PATH_TO_MISP}/app/tmp/log/mispzmq.log
+```
+
+If there's an error **ModuleNotFoundError: No module named 'zmq'** then install pyzmq.
+
+```
+$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install pyzmq
+```
+
 # zmq_subscriber options
 ```usage: zmq_subscriber.py [-h] [-n ZMQNAME] [-u ZMQURL]
 
@@ -155,7 +174,7 @@ optional arguments:
 
 # Deploy in production using mod_wsgi
 
-Install Apache's mod-wsgi for Python3
+Install Apache mod-wsgi for Python3
 
 ```bash
 sudo apt-get install libapache2-mod-wsgi-py3
@@ -170,7 +189,7 @@ The following NEW packages will be installed:
   libapache2-mod-wsgi-py3
 ```
 
-Configuration file `/etc/apache2/sites-available/misp-dashboard.conf` assumes that `misp-dashboard` is cloned into `var/www/misp-dashboard`. It runs as user `misp` in this example. Change the permissions to folder and files accordingly.
+Configuration file `/etc/apache2/sites-available/misp-dashboard.conf` assumes that `misp-dashboard` is cloned into `/var/www/misp-dashboard`. It runs as user `misp` in this example. Change the permissions to your custom folder and files accordingly.
 
 ```
 <VirtualHost *:8001>
@@ -218,33 +237,35 @@ Configuration file `/etc/apache2/sites-available/misp-dashboard.conf` assumes th
 
 # License
 
+~~~~
+    Copyright (C) 2017-2019 CIRCL - Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d'Intérêt Economique)
+    Copyright (c) 2017-2019 Sami Mokaddem
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+~~~~
+
 Images and logos are handmade for:
+
 - rankingMISPOrg/
 - rankingMISPMonthly/
 - MISPHonorableIcons/
 
 Note that:
+
 - Part of ```MISPHonorableIcons/1.svg``` comes from [octicons.github.com](https://octicons.github.com/icon/git-pull-request/) (CC0 - No Rights Reserved)
 - Part of ```MISPHonorableIcons/2.svg``` comes from [Zeptozephyr](https://zeptozephyr.deviantart.com/art/Vectored-Portal-Icons-207347804) (CC0 - No Rights Reserved)
 - Part of ```MISPHonorableIcons/3.svg``` comes from [octicons.github.com](https://octicons.github.com/icon/git-pull-request/) (CC0 - No Rights Reserved)
 - Part of ```MISPHonorableIcons/4.svg``` comes from [Zeptozephyr](https://zeptozephyr.deviantart.com/art/Vectored-Portal-Icons-207347804) & [octicons.github.com](https://octicons.github.com/icon/git-pull-request/) (CC0 - No Rights Reserved)
 - Part of ```MISPHonorableIcons/5.svg``` comes from [Zeptozephyr](https://zeptozephyr.deviantart.com/art/Vectored-Portal-Icons-207347804) & [octicons.github.com](https://octicons.github.com/icon/git-pull-request/) (CC0 - No Rights Reserved)
 
-```
-Copyright (C) 2017-2018 CIRCL - Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d'Intérêt Economique)
-Copyright (c) 2017-2018 Sami Mokaddem
-
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with this program.  If not, see <http://www.gnu.org/licenses/>.
-```

clean.py

@@ -1,11 +1,11 @@
 #!/usr/bin/env python3
 
-from pprint import pprint
-import os
-import redis
-import configparser
 import argparse
+import configparser
+import os
+from pprint import pprint
 
+import redis
 
 RED="\033[91m"
 GREEN="\033[92m"

config/config.cfg.default

@@ -1,6 +1,7 @@
 [Server]
 host = localhost
 port = 8001
+debug = False
 
 [Dashboard]
 #hours
@@ -33,7 +34,9 @@ additional_help_text = ["Sightings multiplies earned points by 2", "Editing an a
 
 [Log]
 directory=logs
-filename=logs.log
+dispatcher_filename=zmq_dispatcher.log
+subscriber_filename=zmq_subscriber.log
+helpers_filename=helpers.log
 
 [RedisGlobal]
 host=localhost

give_honors_to_org.py

@@ -1,9 +1,13 @@
 #!/usr/bin/env python3.5
 
-import os, sys, json
-import datetime, time
-import redis
 import configparser
+import datetime
+import json
+import os
+import sys
+import time
+
+import redis
 
 import util
 from helpers import contributor_helper

helpers/contributor_helper.py

@@ -1,16 +1,20 @@
-import util
-from util import getZrange
-import math, random
-import time
-import os
 import configparser
-import json
 import datetime
+import json
 import logging
+import math
+import os
+import random
+import sys
+import time
+
 import redis
 
 import util
+from util import getZrange
+
 from . import users_helper
+
 KEYDAY = "CONTRIB_DAY" # To be used by other module
 KEYALLORG = "CONTRIB_ALL_ORG" # To be used by other module
 
@@ -30,11 +34,16 @@ class Contributor_helper:
 
         #logger
         logDir = cfg.get('Log', 'directory')
-        logfilename = cfg.get('Log', 'filename')
+        logfilename = cfg.get('Log', 'helpers_filename')
         logPath = os.path.join(logDir, logfilename)
         if not os.path.exists(logDir):
             os.makedirs(logDir)
-        logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+        try:
+            logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+        except PermissionError as error:
+            print(error)
+            print("Please fix the above and try again.")
+            sys.exit(126)
         self.logger = logging.getLogger(__name__)
 
         #honorBadge

helpers/geo_helper.py

@@ -1,18 +1,22 @@
-import math, random
+import datetime
-import os
 import json
-import datetime, time
 import logging
-import json
+import math
-import redis
+import os
+import random
+import sys
+import time
 from collections import OrderedDict
 
-import geoip2.database
+import redis
-import phonenumbers, pycountry
-from phonenumbers import geocoder
 
+import geoip2.database
+import phonenumbers
+import pycountry
 import util
 from helpers import live_helper
+from phonenumbers import geocoder
+
 
 class InvalidCoordinate(Exception):
     pass
@@ -29,11 +33,16 @@ class Geo_helper:
 
         #logger
         logDir = cfg.get('Log', 'directory')
-        logfilename = cfg.get('Log', 'filename')
+        logfilename = cfg.get('Log', 'helpers_filename')
         logPath = os.path.join(logDir, logfilename)
         if not os.path.exists(logDir):
             os.makedirs(logDir)
-        logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+        try:
+            logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+        except PermissionError as error:
+            print(error)
+            print("Please fix the above and try again.")
+            sys.exit(126)
         self.logger = logging.getLogger(__name__)
 
         self.keyCategCoord = "GEO_COORD"
@@ -43,7 +52,12 @@ class Geo_helper:
         self.PATH_TO_JSON = cfg.get('RedisMap', 'path_countrycode_to_coord_JSON')
         self.CHANNELDISP = cfg.get('RedisMap', 'channelDisp')
 
-        self.reader = geoip2.database.Reader(self.PATH_TO_DB)
+        try:
+            self.reader = geoip2.database.Reader(self.PATH_TO_DB)
+        except PermissionError as error:
+            print(error)
+            print("Please fix the above and try again.")
+            sys.exit(126)
         self.country_to_iso = { country.name: country.alpha_2 for country in pycountry.countries}
         with open(self.PATH_TO_JSON) as f:
             self.country_code_to_coord = json.load(f)
@@ -125,7 +139,7 @@ class Geo_helper:
             self.live_helper.add_to_stream_log_cache('Map', j_to_send)
             self.logger.info('Published: {}'.format(json.dumps(to_send)))
         except ValueError:
-            self.logger.warning("can't resolve ip")
+            self.logger.warning("Can't resolve IP: " + str(supposed_ip))
         except geoip2.errors.AddressNotFoundError:
             self.logger.warning("Address not in Database")
         except InvalidCoordinate:
@@ -181,7 +195,12 @@ class Geo_helper:
         now = datetime.datetime.now()
         today_str = util.getDateStrFormat(now)
         keyname = "{}:{}".format(keyCateg, today_str)
-        self.serv_redis_db.geoadd(keyname, lon, lat, content)
+        try:
+            self.serv_redis_db.geoadd(keyname, lon, lat, content)
+        except redis.exceptions.ResponseError as error:
+            print(error)
+            print("Please fix the above, and make sure you use a redis version that supports the GEOADD command.")
+            print("To test for support: echo \"help GEOADD\"| redis-cli")
         self.logger.debug('Added to redis: keyname={}, lon={}, lat={}, content={}'.format(keyname, lon, lat, content))
     def push_to_redis_zset(self, keyCateg, toAdd, endSubkey="", count=1):
         now = datetime.datetime.now()

helpers/live_helper.py

@@ -1,8 +1,10 @@
-import os
+import datetime
 import json
-import random
-import datetime, time
 import logging
+import os
+import random
+import sys
+import time
 
 
 class Live_helper:
@@ -16,11 +18,16 @@ class Live_helper:
 
         # logger
         logDir = cfg.get('Log', 'directory')
-        logfilename = cfg.get('Log', 'filename')
+        logfilename = cfg.get('Log', 'helpers_filename')
         logPath = os.path.join(logDir, logfilename)
         if not os.path.exists(logDir):
             os.makedirs(logDir)
-        logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+        try:
+            logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+        except PermissionError as error:
+            print(error)
+            print("Please fix the above and try again.")
+            sys.exit(126)
         self.logger = logging.getLogger(__name__)
 
     def publish_log(self, zmq_name, name, content, channel=None):

helpers/trendings_helper.py

@@ -1,13 +1,17 @@
-import math, random
-import os
-import json
 import copy
-import datetime, time
+import datetime
+import json
 import logging
+import math
+import os
+import random
+import sys
+import time
 from collections import OrderedDict
 
 import util
 
+
 class Trendings_helper:
     def __init__(self, serv_redis_db, cfg):
         self.serv_redis_db = serv_redis_db
@@ -23,11 +27,16 @@ class Trendings_helper:
 
         #logger
         logDir = cfg.get('Log', 'directory')
-        logfilename = cfg.get('Log', 'filename')
+        logfilename = cfg.get('Log', 'helpers_filename')
         logPath = os.path.join(logDir, logfilename)
         if not os.path.exists(logDir):
             os.makedirs(logDir)
-        logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+        try:
+            logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+        except PermissionError as error:
+            print(error)
+            print("Please fix the above and try again.")
+            sys.exit(126)
         self.logger = logging.getLogger(__name__)
 
     ''' SETTER '''

helpers/users_helper.py

@@ -1,10 +1,14 @@
-import math, random
+import datetime
-import os
 import json
-import datetime, time
 import logging
+import math
+import os
+import random
+import sys
+import time
 
 import util
+
 from . import contributor_helper
 
 
@@ -20,11 +24,16 @@ class Users_helper:
 
         #logger
         logDir = cfg.get('Log', 'directory')
-        logfilename = cfg.get('Log', 'filename')
+        logfilename = cfg.get('Log', 'helpers_filename')
         logPath = os.path.join(logDir, logfilename)
         if not os.path.exists(logDir):
             os.makedirs(logDir)
-        logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+        try:
+            logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+        except PermissionError as error:
+            print(error)
+            print("Please fix the above and try again.")
+            sys.exit(126)
         self.logger = logging.getLogger(__name__)
 
     def add_user_login(self, timestamp, org, email=''):
@@ -63,11 +72,11 @@ class Users_helper:
                 else:
                     break # timestamps should be sorted, no need to process anymore
         return to_return
-                    
+
 
     # return: All dates for all orgs, if date is not supplied, return for all dates
     def getUserLogins(self, date=None):
-        # get all orgs and retreive their timestamps
+        # get all orgs and retrieve their timestamps
         dates = []
         for org in self.getAllOrg():
             keyname = "{}:{}".format(self.keyOrgLog, org)
@@ -169,7 +178,7 @@ class Users_helper:
         data = [data[6]]+data[:6]
         return data
 
-    # return: a dico of the form {login: [[timestamp, count], ...], contrib: [[timestamp, 1/0], ...]} 
+    # return: a dico of the form {login: [[timestamp, count], ...], contrib: [[timestamp, 1/0], ...]}
     #         either for all orgs or the supplied one
     def getUserLoginsAndContribOvertime(self, date, org=None, prev_days=6):
         dico_hours_contrib = {}

install_dependencies.sh

@@ -1,12 +1,21 @@
 #!/bin/bash
 
-set -e
+## disable -e for production systems
+#set -e
+
+## Debug mode
 #set -x
 
 sudo apt-get install python3-virtualenv virtualenv screen redis-server unzip -y
 
 if [ -z "$VIRTUAL_ENV" ]; then
-    virtualenv -p python3 DASHENV
+    virtualenv -p python3 DASHENV ; DASH_VENV=$?
+
+    if [[ "$DASH_VENV" != "0" ]]; then
+      echo "Something went wrong with either the update or install of the virtualenv."
+      echo "Please investigate manually."
+      exit $DASH_VENV
+    fi
 
     . ./DASHENV/bin/activate
 fi
@@ -35,7 +44,14 @@ mkdir -p css fonts js
 popd
 mkdir -p temp
 
-wget http://www.misp-project.org/assets/images/misp-small.png -O static/pics/MISP.png
+NET_WGET=$(wget --no-cache -q https://www.misp-project.org/assets/images/misp-small.png -O static/pics/MISP.png; echo $?)
+
+if [[ "$NET_WGET" != "0" ]]; then
+  echo "The first wget we tried failed, please investigate manually."
+  exit $NET_WGET
+fi
+
+wget https://www.misp-project.org/favicon.ico -O static/favicon.ico
 
 # jquery
 JQVERSION="3.2.1"

retrieve_map_pic.py

@@ -1,13 +1,15 @@
 #!/usr/bin/env python3.5
-import redis
-import requests
-import shutil
 import json
 import math
-import sys, os
+import os
+import shlex
+import shutil
+import sys
 import time
 from subprocess import PIPE, Popen
-import shlex
+
+import redis
+import requests
 
 URL_OPEN_MAP = "http://tile.openstreetmap.org/{zoom}/{x}/{y}.png"
 MAP_DIR = "static/maps/"

server.py

@@ -1,21 +1,22 @@
 #!/usr/bin/env python3
-from flask import Flask, render_template, request, Response, jsonify, stream_with_context
-import json
-import redis
-import random, math
 import configparser
+import datetime
+import errno
+import json
+import logging
+import math
+import os
+import random
 from time import gmtime as now
 from time import sleep, strftime
-import datetime
+
-import os
+import redis
-import logging
 
 import util
-from helpers import geo_helper
+from flask import (Flask, Response, jsonify, render_template, request,
-from helpers import contributor_helper
+                   send_from_directory, stream_with_context)
-from helpers import users_helper
+from helpers import (contributor_helper, geo_helper, live_helper,
-from helpers import trendings_helper
+                     trendings_helper, users_helper)
-from helpers import live_helper
 
 configfile = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'config/config.cfg')
 cfg = configparser.ConfigParser()
@@ -26,6 +27,7 @@ logger.setLevel(logging.ERROR)
 
 server_host = cfg.get("Server", "host")
 server_port = cfg.getint("Server", "port")
+server_debug = cfg.get("Server", "debug")
 
 app = Flask(__name__)
 
@@ -94,6 +96,12 @@ class LogItem():
                 to_add = util.getFields(self.feed, field)
             to_ret[i] = to_add if to_add is not None else ''
 
+        # Number to keep them sorted (jsonify sort keys)
+        for item in range(len(LogItem.FIELDNAME_ORDER)):
+            try:
+                to_ret[item] = self.fields[item]
+            except IndexError: # not enough field in rcv item
+                to_ret[item] = ''
         return to_ret
 
 
@@ -176,6 +184,10 @@ def index():
             zoomlevel=cfg.getint('Dashboard' ,'zoomlevel')
             )
 
+@app.route('/favicon.ico')
+def favicon():
+    return send_from_directory(os.path.join(app.root_path, 'static'),
+                               'favicon.ico', mimetype='image/vnd.microsoft.icon')
 
 @app.route("/geo")
 def geo():
@@ -640,4 +652,13 @@ def getGenericTrendingOvertime():
     return jsonify(data)
 
 if __name__ == '__main__':
-    app.run(host=server_host, port=server_port, threaded=True)
+    try:
+        app.run(host=server_host,
+            port=server_port,
+            debug=server_debug,
+            threaded=True)
+    except OSError as error:
+        if error.errno == 98:
+            print("\n\n\nAddress already in use, the defined port is: " + str(server_port))
+        else:
+            print(str(error))

start_all.sh

@@ -20,7 +20,22 @@ else
     exit 1
 fi
 
-[ ! -f "`which redis-server`" ] && echo "'redis-server' is not installed/not on PATH. Please fix and run again." && exit 1
+if [[ -f "/etc/redhat-release" ]]; then
+  echo "You are running a RedHat flavour. Detecting scl potential..."
+  if [[ -f "/usr/bin/scl" ]]; then
+    echo "scl detected, checking for redis-server"
+    SCL_REDIS=$(scl -l|grep rh-redis)
+    if [[ ! -z $SCL_REDIS ]]; then
+      echo "We detected: ${SCL_REDIS} acting accordingly"
+      REDIS_RUN="/usr/bin/scl enable ${SCL_REDIS}"
+    fi
+  else
+    echo "redis-server seems not to be install in scl, perhaps system-wide, testing."
+    [ ! -f "`which redis-server`" ] && echo "'redis-server' is not installed/not on PATH. Please fix and run again." && exit 1
+  fi
+else
+  [ ! -f "`which redis-server`" ] && echo "'redis-server' is not installed/not on PATH. Please fix and run again." && exit 1
+fi
 
 netstat -an |grep LISTEN |grep 6250 |grep -v tcp6 ; check_redis_port=$?
 netstat -an |grep LISTEN |grep 8001 |grep -v tcp6 ; check_dashboard_port=$?
@@ -37,8 +52,12 @@ conf_dir="config/"
 
 sleep 0.1
 if [ "${check_redis_port}" == "1" ]; then
-    echo -e $GREEN"\t* Launching Redis servers"$DEFAULT
+  echo -e $GREEN"\t* Launching Redis servers"$DEFAULT
-    redis-server ${conf_dir}6250.conf &
+    if [[ ! -z $REDIS_RUN ]]; then
+      $REDIS_RUN "redis-server ${conf_dir}6250.conf" &
+    else
+      redis-server ${conf_dir}6250.conf &
+    fi
 else
     echo -e $RED"\t* NOT starting Redis server, made a very unrealiable check on port 6250, and something seems to be there… please double check if this is good!"$DEFAULT
 fi

static/js/contrib.js

@@ -346,7 +346,8 @@ function addLastContributor(datatable, data, update) {
     } else {
         last_added_contrib = org;
         var date = new Date(data.epoch*1000);
-        date.toString = function() {return this.toTimeString().slice(0,-15) +' '+ this.toLocaleDateString(); };
+        //date.toString = function() {return this.toTimeString().slice(0,-15) +' '+ this.toLocaleDateString(); };
+        date = date.getFullYear() + "-" + String(date.getMonth()).padStart(2, "0") + "-" + String(date.getDay()).padStart(2, "0") + "@" + String(date.getHours()).padStart(2, "0") + ":" + String(date.getMinutes()).padStart(2, "0");
         var to_add = [
             date,
             data.pnts,
@@ -383,7 +384,8 @@ function addAwards(datatableAwards, json, playAnim) {
         var award = createTrophyImg(json.award[1][1], 40, categ);
     }
     var date = new Date(json.epoch*1000);
-    date.toString = function() {return this.toTimeString().slice(0,-15) +' '+ this.toLocaleDateString(); };
+    //date.toString = function() {return this.toTimeString().slice(0,-15) +' '+ this.toLocaleDateString(); };
+    date = date.getFullYear() + "-" + String(date.getMonth()).padStart(2, "0") + "-" + String(date.getDay()).padStart(2, "0") + "@" + String(date.getHours()).padStart(2, "0") + ":" + String(date.getMinutes()).padStart(2, "0");
     var to_add = [
         date,
         createImg(json.logo_path, 32),

static/js/trendings.js

@@ -145,7 +145,7 @@ function getTextColour(rgb) {
     }
 }
 
-// If json (from tag), only retreive the name> otherwise return the supplied arg.
+// If json (from tag), only retrieve the name> otherwise return the supplied arg.
 function getOnlyName(potentialJson) {
     try {
         jsonLabel = JSON.parse(potentialJson);

tests/test_geo.py

@@ -1,8 +1,13 @@
 #!/usr/bin/env python3.5
 import configparser
-import redis
-import sys,os
 import datetime
+import os
+import sys
+
+import redis
+
+from helpers import geo_helper
+
 sys.path.append('..')
 
 configfile = 'test_config.cfg'
@@ -14,7 +19,6 @@ serv_redis_db = redis.StrictRedis(
         port=6260,
         db=1)
 
-from helpers import geo_helper
 geo_helper = geo_helper.Geo_helper(serv_redis_db, cfg)
 
 categ = 'Network Activity'

tests/test_trendings.py

@@ -1,8 +1,14 @@
 #!/usr/bin/env python3.5
 import configparser
+import datetime
+import os
+import sys
+import time
+
 import redis
-import sys,os
+
-import datetime, time
+from helpers import trendings_helper
+
 sys.path.append('..')
 
 configfile = 'test_config.cfg'
@@ -14,7 +20,6 @@ serv_redis_db = redis.StrictRedis(
         port=6260,
         db=1)
 
-from helpers import trendings_helper
 trendings_helper = trendings_helper.Trendings_helper(serv_redis_db, cfg)
 
 

tests/test_users.py

@@ -1,8 +1,14 @@
 #!/usr/bin/env python3.5
 import configparser
+import datetime
+import os
+import sys
+import time
+
 import redis
-import sys,os
+
-import datetime, time
+from helpers import users_helper
+
 sys.path.append('..')
 
 configfile = 'test_config.cfg'
@@ -14,7 +20,6 @@ serv_redis_db = redis.StrictRedis(
         port=6260,
         db=1)
 
-from helpers import users_helper
 users_helper = users_helper.Users_helper(serv_redis_db, cfg)
 
 

util.py

@@ -1,5 +1,6 @@
+import datetime
+import time
 from collections import defaultdict
-import datetime, time
 
 ONE_DAY = 60*60*24
 

zmq_dispatcher.py

@@ -1,34 +1,38 @@
 #!/usr/bin/env python3
 
-import time, datetime
-import copy
-import logging
-import zmq
-import redis
-import random
-import configparser
 import argparse
-import os
+import configparser
-import sys
+import copy
+import datetime
 import json
+import logging
+import os
+import random
+import sys
+import time
+
+import redis
+import zmq
 
 import util
-from helpers import geo_helper
+from helpers import (contributor_helper, geo_helper, live_helper,
-from helpers import contributor_helper
+                     trendings_helper, users_helper)
-from helpers import users_helper
-from helpers import trendings_helper
-from helpers import live_helper
 
 configfile = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'config/config.cfg')
 cfg = configparser.ConfigParser()
 cfg.read(configfile)
 
 logDir = cfg.get('Log', 'directory')
-logfilename = cfg.get('Log', 'filename')
+logfilename = cfg.get('Log', 'dispatcher_filename')
 logPath = os.path.join(logDir, logfilename)
 if not os.path.exists(logDir):
     os.makedirs(logDir)
-logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+try:
+    logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+except PermissionError as error:
+    print(error)
+    print("Please fix the above and try again.")
+    sys.exit(126)
 logger = logging.getLogger('zmq_dispatcher')
 
 LISTNAME = cfg.get('RedisLIST', 'listName')
@@ -286,4 +290,7 @@ if __name__ == "__main__":
     parser.add_argument('-s', '--sleep', required=False, dest='sleeptime', type=int, help='The number of second to wait before checking redis list size', default=5)
     args = parser.parse_args()
 
-    main(args.sleeptime)
+    try:
+        main(args.sleeptime)
+    except (redis.exceptions.ResponseError, KeyboardInterrupt) as error:
+        print(error)

zmq_subscriber.py

@@ -1,24 +1,31 @@
 #!/usr/bin/env python3
 
-import time, datetime
-import zmq
-import logging
-import redis
-import configparser
 import argparse
+import configparser
+import datetime
+import json
+import logging
 import os
 import sys
-import json
+import time
+
+import redis
+import zmq
 
 configfile = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'config/config.cfg')
 cfg = configparser.ConfigParser()
 cfg.read(configfile)
 logDir = cfg.get('Log', 'directory')
-logfilename = cfg.get('Log', 'filename')
+logfilename = cfg.get('Log', 'subscriber_filename')
 logPath = os.path.join(logDir, logfilename)
 if not os.path.exists(logDir):
     os.makedirs(logDir)
-logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+try:
+    logging.basicConfig(filename=logPath, filemode='a', level=logging.INFO)
+except PermissionError as error:
+    print(error)
+    print("Please fix the above and try again.")
+    sys.exit(126)
 logger = logging.getLogger('zmq_subscriber')
 
 ZMQ_URL = cfg.get('RedisGlobal', 'zmq_url')
@@ -57,9 +64,12 @@ def main(zmqName):
 
 if __name__ == "__main__":
 
-    parser = argparse.ArgumentParser(description='A zmq subscriber. It subscribes to a ZNQ then redispatch it to the misp-dashboard')
+    parser = argparse.ArgumentParser(description='A zmq subscriber. It subscribes to a ZMQ then redispatch it to the misp-dashboard')
     parser.add_argument('-n', '--name', required=False, dest='zmqname', help='The ZMQ feed name', default="MISP Standard ZMQ")
     parser.add_argument('-u', '--url', required=False, dest='zmqurl', help='The URL to connect to', default=ZMQ_URL)
     args = parser.parse_args()
 
-    main(args.zmqname)
+    try:
+        main(args.zmqname)
+    except redis.exceptions.ResponseError as error:
+        print(error)