Update TRANSFORM_HUB_DISCLAIMER.md

Fix timeout

README.md

@@ -1,4 +1,5 @@
 
+
 ![logo](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/logo.png)
 
 This is a [Maltego](https://www.paterva.com/web7/) [MISP](https://www.misp-project.org) integration tool allowing you to view (read-only) data from a MISP instance. 
@@ -17,11 +18,13 @@ The currently supported entities are: `AS`, `DNSName`, `Domain`, `EmailAddress`,
 For MITRE ATT&CK pivoting, feel free to start with an `Attack Technique`, `Software`, `Threat Actor`, or `MISPGalaxy`. Create your entity, enter a keyword such as `%gama%` and use the `Search in MISP` transform to get started. 
 
 ## Installation
-### Remote Transform Installation
-This is coming soon. An entry will appear in the Transform Hub of Maltego, on which you can simply press the "install" button, and no local code needs to be installed. Your transforms will however go throught Paterva's servers and ours. See the [Transform Hub Disclaimer](https://github.com/MISP/MISP-maltego/blob/master/TRANSFORM_HUB_DISCLAIMER.md) for more information.
+### Transform Hub
+Open the Transform Hub, locate **ATT&CK - MISP** and press the **Install** button. 
 
-**For MISP specific transforms this requires your MISP server to be reachable from the internet!
-ATT&CK transforms do not require a MISP server or API key to be configured.**
+Your transforms will go through Paterva's servers and ours. See the [Transform Hub Disclaimer](https://github.com/MISP/MISP-maltego/blob/master/TRANSFORM_HUB_DISCLAIMER.md) for more information.
+
+- ATT&CK transforms do not require a MISP server or API key to be configured.
+- MISP transforms requires your MISP server to be reachable from the internet! To enter your MISP server URL and key click **Details** on the Transform Hub item and then **Settings** at the bottom right. 
 
 ### Local Transform Installation
 If you trust nobody, or just want to connect to your local MISP server you can install everything as local transforms.

SECURITY.md

@@ -0,0 +1 @@
+For security issues, please refer to the instructions in the main MISP repository: https://github.com/MISP/MISP/

TRANSFORM_HUB_DISCLAIMER.md

@@ -3,13 +3,13 @@
 
 The public Transform Distribution Server (TDS) is located on the Internet and is free for all to use. It’s a convenient way to immediately start writing remote transforms. Since this server is located on Paterva’s infrastructure data (entity, and settings) will be flowing from the Maltego GUI to this server. Paterva states they DO NOT store the details of your transforms (entities, MISP URL, API KEY).
 
-Finally it will flow further to a server managed by the MISP-maltego developer(s), where the transform code runs. We also DO NOT store or look at the details of your transforms (entities, MISP URL, API KEY). As you can see in the code (open source), this data is only used live in memory to provide the transform functionality. The only reasons why we would be seeing this data is by accident; while troubleshooting or by unintentional mis-configuration.
+Finally it will flow further to a server managed by the MISP-maltego developer(s), where the transform code runs and for MISP transforms connects to YOUR server. We DO NOT store or look at the details of your transforms (entities, MISP URL, API KEY). As you can see in the code (open source), this data is only used live in memory to provide the transform functionality. The only reasons why we would be seeing this data is by accident; while troubleshooting or by unintentional mis-configuration.
 
 We do keep standard HTTP logs for troubleshooting and anonymous statistics, although these contain the IP addresses of Paterva's TDS server, and not yours.
 
 **DO NOT use these Transform Hub transforms if you do not agree or if this is in violation with your MISP community.**
 
-**If so, feel free to use the MISP-Maltego transforms locally, where all the code runs on your own system. Installation instructions can be found [here](https://github.com/MISP/MISP-maltego/blob/master/doc/README.md#installation).**
+**If so, feel free to use the MISP-Maltego transforms locally, where all the code runs on your own system. Installation instructions can be found [here](https://github.com/MISP/MISP-maltego/blob/master/README.md#installation).**
 
 You can also run this on your own iTDS server if you have the license. Have a look at the [Dockerfile](https://github.com/MISP/MISP-maltego/blob/master/Dockerfile) for more info.
 

ansible/nginx.conf

@@ -21,6 +21,12 @@ server {
 	root /var/www/html;
 	server_name _;
 
+	location /munin/ {
+		alias /var/cache/munin/www/;
+		index index.html;
+		allow 127.0.0.1;
+		deny all;
+	}
 
 	location / {
 		proxy_set_header X-Real-IP $remote_addr;
@@ -32,4 +38,4 @@ server {
 	location @redirect {
 		return 302 https://github.com/MISP/MISP-maltego;
 	}
-}
+}

ansible/plume.yaml

@@ -168,6 +168,29 @@
       name: plume
       state: started
 
+  # MONITORING
+  #############
+  - name: install munin
+    package:
+      name: ['munin', 'munin-node', 'munin-plugins-extra']
+
+  - name: munin - enabling plugins
+    file:
+      state: link
+      src: '/usr/share/munin/plugins/{{item}}'
+      dest: '/etc/munin/plugins/{{item}}'
+    loop:
+      - nginx_request
+      - nginx_status
+    notify: restart munin-node
+
+  - name: munin - service active and running
+    service:
+      name: munin-node
+      state: started
+      enabled: yes
+
+
   # FIREWALLING
   #############
   - name: firewall logging
@@ -225,3 +248,8 @@
       service:
         name: nginx
         state: restarted
+
+    - name: restart munin-node
+      service:
+        name: munin-node
+        state: restarted

ansible/redeploy.yaml

@@ -0,0 +1,59 @@
+---
+# Install MISP-maltego remote transform using ansible.
+
+- hosts: all
+  become: yes
+  vars:
+    misp_maltego_version: 1.4.5  # TODO change this !!!
+    host_locale: en_US.UTF-8
+    host_locale_dict: {
+      LANG: "{{ host_locale }}",
+      LC_COLLATE: "{{ host_locale }}",
+      LC_CTYPE: "{{ host_locale }}",
+      LC_MESSAGES: "{{ host_locale }}",
+      LC_MONETARY: "{{ host_locale }}",
+      LC_NUMERIC: "{{ host_locale }}",
+      LC_TIME: "{{ host_locale }}",
+      LC_ALL: "{{ host_locale }}",
+    }
+
+  tasks:
+  # use the public pip package
+  - name: install MISP-maltego
+    pip:
+      executable: pip3
+      name: ['MISP-maltego']
+      state: latest
+    notify: restart plume
+
+  # use local git repo instead, useful for development
+  # - name: bundle MISP-maltego
+  #   delegate_to: 127.0.0.1
+  #   command: 
+  #     cmd: python3 setup.py sdist
+  #     chdir: ../
+  #   become: no
+  # - name: copy MISP-maltego
+  #   copy:
+  #     src: ../dist/MISP_maltego-{{misp_maltego_version}}.tar.gz
+  #     dest: /usr/local/src/
+  # - name: install MISP-maltego
+  #   pip:
+  #     executable: /usr/bin/pip3
+  #     name: file:///usr/local/src/MISP_maltego-{{misp_maltego_version}}.tar.gz
+  #     state: forcereinstall
+  #   environment: "{{host_locale_dict}}"
+  #   notify: restart plume
+
+  # - name: remove local MISP-maltego bundle
+  #   delegate_to: 127.0.0.1
+  #   file:
+  #     path: ../dist/MISP_maltego-{{misp_maltego_version}}.tar.gz
+  #     state: absent
+  #   become: no
+
+  handlers:
+    - name: restart plume
+      service:
+        name: plume
+        state: restarted

k8s.yaml

@@ -1,43 +0,0 @@
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: misp-maltego
-spec:
-  selector:
-    matchLabels:
-      app: misp-maltego
-  replicas: 1
-  template:
-    metadata:
-      labels:
-        app: misp-maltego
-    spec:
-      containers:
-        - name: misp-maltego
-          image: cvandeplas/misp-maltego
-          imagePullPolicy: Always
-          ports:
-            - name: listen-port
-              containerPort: 8080
-          resources:
-            requests:
-              cpu: 10m
-              memory: 4Gi
-            limits:
-              cpu: 1
-              memory: 4Gi
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: misp-maltego
-  annotations:
-    dev.okteto.com/auto-ingress: "true"
-spec:
-  type: ClusterIP  
-  ports:
-  - name: "misp-maltego"
-    port: 8080
-  selector:
-    app: misp-maltego

setup.py

@@ -9,7 +9,7 @@ setup(
     name='MISP_maltego',
     author='Christophe Vandeplas',
     # also update version in util.py
-    version='1.4.5',
+    version='1.4.6',
     author_email='christophe@vandeplas.com',
     maintainer='Christophe Vandeplas',
     url='https://github.com/MISP/MISP-maltego',
@@ -36,7 +36,7 @@ setup(
     python_requires='>=3.5',
     install_requires=[
         'canari>=3.3.10,<4',
-        'PyMISP>=2.4.114'
+        'PyMISP>=2.4.127'
     ],
     dependency_links=[
         # custom links for the install_requires

src/MISP_maltego/transforms/attributetoevent.py

@@ -82,19 +82,24 @@ class SearchInMISP(Transform):
 
         # for all other normal entities
         conn = MISPConnection(config, request.parameters)
-        events_json = conn.misp.search(controller='events', value=request.entity.value, with_attachments=False)
+
         # we need to do really rebuild the Entity from scratch as request.entity is of type Unknown
-        for e in events_json:
-            # find the value as attribute
-            attr = get_attribute_in_event(e, request.entity.value, substring=True)
-            if attr:
-                for item in attribute_to_entity(attr, only_self=True):
-                    response += item
-            # find the value as object, and return the object
-            if 'Object' in e['Event']:
-                for o in e['Event']['Object']:
-                    if get_attribute_in_object(o, attribute_value=request.entity.value, substring=True).get('value'):
-                        response += conn.object_to_entity(o, link_label=link_label)
+        # TODO First try to build the object, then only attributes (for those that are not in object, or for all?)
+        # TODO check for the right version of MISP before, it needs to be 2.4.127 or higher.
+        # obj_json = conn.misp.search(controller='objects', value=request.entity.value, with_attachments=False)
+        # for o in obj_json:
+        #         for item in attribute_to_entity(attr, only_self=True, link_label=link_label):
+        #             response += item
+        #     # find the value as object, and return the object
+        #     if 'Object' in e['Event']:
+        #         for o in e['Event']['Object']:
+        #             if get_attribute_in_object(o, attribute_value=request.entity.value, substring=True).get('value'):
+        #                 response += conn.object_to_entity(o, link_label=link_label)
+
+        attr_json = conn.misp.search(controller='attributes', value=request.entity.value, with_attachments=False)
+        for a in attr_json['Attribute']:
+            for item in attribute_to_entity(a, only_self=True, link_label=link_label):
+                response += item
 
         return response
 
@@ -123,7 +128,7 @@ class SearchInMISP(Transform):
 
 class AttributeToEvent(Transform):
     input_type = Unknown
-    display_name = 'to MISP Events'
+    display_name = 'To MISP Events'
     remote = True
 
     def do_transform(self, request, response, config):

src/MISP_maltego/transforms/common/util.py

@@ -12,7 +12,7 @@ import requests
 import tempfile
 import time
 
-__version__ = '1.4.5'  # also update version in setup.py
+__version__ = '1.4.6'  # also update version in setup.py
 
 tag_note_prefixes = ['tlp:', 'PAP:', 'de-vs:', 'euci:', 'fr-classif:', 'nato:']
 
@@ -85,7 +85,7 @@ class MISPConnection():
                     misp_key = parameters['mispkey'].value
                 except AttributeError:
                     raise MaltegoException("ERROR: mispurl and mispkey need to be set to something valid")
-            self.misp = PyMISP(misp_url, misp_key, misp_verify, 'json', misp_debug, tool='misp_maltego')
+            self.misp = PyMISP(url=misp_url, key=misp_key, ssl=misp_verify, debug=misp_debug, tool='misp_maltego', timeout=(2, 60))
         except Exception:
             if is_local_exec_mode():
                 raise MaltegoException("ERROR: Cannot connect to MISP server. Please verify your MISP_Maltego.conf settings.")
@@ -219,6 +219,7 @@ def attribute_to_entity(a, link_label=None, event_tags=[], only_self=False):
     if a['type'] in ('url', 'uri'):
         yield(URL(url=a['value'], short_title=a['value'], link_label=link_label, notes=notes, bookmark=Bookmark.Green))
         return
+    # FIXME implement attachment screenshot type
 
     # attribute is from an object, and a relation gives better understanding of the type of attribute
     if a.get('object_relation') and mapping_misp_to_maltego.get(a['object_relation']):
@@ -444,9 +445,9 @@ def galaxycluster_to_entity(c, link_label=None, link_direction=LinkDirection.Inp
 
 # LATER this uses the galaxies from github as the MISP web UI does not fully support the Galaxies in the webui.
 # See https://github.com/MISP/MISP/issues/3801
-galaxy_archive_url = 'https://github.com/MISP/misp-galaxy/archive/master.zip'
+galaxy_archive_url = 'https://github.com/MISP/misp-galaxy/archive/main.zip'
 local_path_uuid_mapping = os.path.join(local_path_root, 'MISP_maltego_galaxy_mapping.json')
-local_path_clusters = os.path.join(local_path_root, 'misp-galaxy-master', 'clusters')
+local_path_clusters = os.path.join(local_path_root, 'misp-galaxy-main', 'clusters')
 galaxy_cluster_uuids = None
 
 
@@ -479,6 +480,8 @@ def galaxy_update_local_copy(force=False):
             zf.extractall(local_path_root)
             zf.close()
         except Exception:
+            # remove the lock
+            os.remove(lockfile)
             raise(MaltegoException("ERROR: Could not download Galaxy data from htts://github.com/MISP/MISP-galaxy/. Please check internet connectivity."))
 
         # generate the uuid mapping and save it to a file

src/MISP_maltego/transforms/eventtoattributes.py

@@ -85,6 +85,7 @@ class EventToTransform(Transform):
 
 class EventToAll(EventToTransform):
     input_type = MISPEvent
+    display_name = 'To All'
     description = 'Expands an Event to Attributes, Objects, Tags, Galaxies'
     remote = True
 
@@ -100,7 +101,7 @@ class EventToAll(EventToTransform):
 
 class EventToAttributes(EventToTransform):
     input_type = MISPEvent
-    description = 'Expands an Event to Attributes'
+    display_name = 'To Attributes/Objects'
     remote = True
 
     def do_transform(self, request, response, config):
@@ -113,6 +114,7 @@ class EventToAttributes(EventToTransform):
 
 class EventToTags(EventToTransform):
     input_type = MISPEvent
+    display_name = 'To Tags'
     description = 'Expands an Event to Tags and Galaxies'
     remote = True
 
@@ -126,7 +128,7 @@ class EventToTags(EventToTransform):
 
 class EventToGalaxies(EventToTransform):
     input_type = MISPEvent
-    description = 'Expands an Event to Galaxies'
+    display_name = 'To Galaxies / ATT&CK'
     remote = True
 
     def do_transform(self, request, response, config):
@@ -138,7 +140,7 @@ class EventToGalaxies(EventToTransform):
 
 class EventToObjects(EventToTransform):
     input_type = MISPEvent
-    description = 'Expands an Event to Objects'
+    display_name = 'To Objects'
     remote = True
 
     def do_transform(self, request, response, config):
@@ -150,7 +152,7 @@ class EventToObjects(EventToTransform):
 
 class EventToRelations(EventToTransform):
     input_type = MISPEvent
-    description = 'Expands an Event to related Events'
+    display_name = 'To Related Events'
     remote = True
 
     def do_transform(self, request, response, config):
@@ -161,9 +163,8 @@ class EventToRelations(EventToTransform):
 
 
 class ObjectToAttributes(Transform):
-    """"Expands an object to its attributes"""
     input_type = MISPObject
-    description = 'Expands an Object to Attributes'
+    display_name = 'To Attributes'
     remote = True
 
     def do_transform(self, request, response, config):
@@ -184,9 +185,8 @@ class ObjectToAttributes(Transform):
 
 
 class ObjectToRelations(Transform):
-    """Expands an object to the relations of the object"""
     input_type = MISPObject
-    description = 'Expands an Object to Relations'
+    display_name = 'To Related Objects'
     remote = True
 
     def do_transform(self, request, response, config):

src/MISP_maltego/transforms/galaxytoevent.py

@@ -15,26 +15,6 @@ __email__ = 'christophe@vandeplas.com'
 __status__ = 'Development'
 
 
-class GalaxyToEvents(Transform):
-    """Expands a Galaxy to multiple MISP Events."""
-
-    # The transform input entity type.
-    input_type = MISPGalaxy
-    remote = True
-
-    def do_transform(self, request, response, config):
-        response += check_update(config)
-        conn = MISPConnection(config, request.parameters)
-        if request.entity.tag_name:
-            tag_name = request.entity.tag_name
-        else:
-            tag_name = request.entity.value
-        events_json = conn.misp.search(controller='events', tags=tag_name, with_attachments=False)
-        for e in events_json:
-            response += MISPEvent(e['Event']['id'], uuid=e['Event']['uuid'], info=e['Event']['info'], link_direction=LinkDirection.OutputToInput)
-        return response
-
-
 class GalaxyToTransform(Transform):
     input_type = None
 
@@ -106,8 +86,8 @@ class GalaxyToTransform(Transform):
 
 
 class GalaxyToRelations(GalaxyToTransform):
-    """Expands a Galaxy to related Galaxies and Clusters"""
     input_type = MISPGalaxy
+    display_name = 'To Related Galaxies'
     remote = True
 
     def do_transform(self, request, response, config, type_filter=MISPGalaxy):
@@ -115,8 +95,8 @@ class GalaxyToRelations(GalaxyToTransform):
 
 
 class GalaxyToSoftware(GalaxyToTransform):
-    """Expands a Galaxy to related Software/Tool Galaxies"""
     input_type = MISPGalaxy
+    display_name = 'To Malware/Software/Tools'
     remote = True
 
     def do_transform(self, request, response, config, type_filter=Software):
@@ -124,8 +104,8 @@ class GalaxyToSoftware(GalaxyToTransform):
 
 
 class GalaxyToThreatActor(GalaxyToTransform):
-    """Expands a Galaxy to related ThreatActor Galaxies"""
     input_type = MISPGalaxy
+    display_name = 'To Threat Actors'
     remote = True
 
     def do_transform(self, request, response, config, type_filter=ThreatActor):
@@ -133,8 +113,8 @@ class GalaxyToThreatActor(GalaxyToTransform):
 
 
 class GalaxyToAttackTechnique(GalaxyToTransform):
-    """Expands a Galaxy to related Attack Techniques Galaxies"""
     input_type = MISPGalaxy
+    display_name = 'To Attack Techniques'
     remote = True
 
     def do_transform(self, request, response, config, type_filter=AttackTechnique):