Update TRANSFORM_HUB_DISCLAIMER.md

Fix timeout

README.md

@@ -1,9 +1,13 @@
+
+
 ![logo](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/logo.png)
-# Quick start guide
+
 This is a [Maltego](https://www.paterva.com/web7/) [MISP](https://www.misp-project.org) integration tool allowing you to view (read-only) data from a MISP instance. 
+It also allows browsing through the [MITRE ATT&CK](https://attack.mitre.org/) entities. (no MISP connection needed)
 
-It also allows browsing through the [MITRE ATT&CK](https://attack.mitre.org/) entities.
+This user guide should help you through the [installation](#installation) of **MISP-Maltego**, and should guide you how to use it through a few [use-cases](#use-cases). As this is a collaborative project, do not hesitate to propose changes, write other use-cases or raise [feature requests](https://github.com/MISP/MISP-maltego/issues) for missing features.
 
+## Quick start
 Currently supported MISP elements are : Event, Attribute, Object (incl relations), Tag, Taxonomy, Galaxy (incl relations).
 
 Once installed you can start by creating a `MISPEvent` entity, then load the Machine `EventToAll` or the transform `EventToAttributes`.
@@ -11,17 +15,112 @@ Once installed you can start by creating a `MISPEvent` entity, then load the Mac
 Alternatively initiate a transform on an existing Maltego entity.
 The currently supported entities are: `AS`, `DNSName`, `Domain`, `EmailAddress`, `File`, `Hash`, `IPv4Address`, `NSRecord`, `Person`, `PhoneNumber`, `URL`, `Website`
 
+For MITRE ATT&CK pivoting, feel free to start with an `Attack Technique`, `Software`, `Threat Actor`, or `MISPGalaxy`. Create your entity, enter a keyword such as `%gama%` and use the `Search in MISP` transform to get started. 
 
-## Installation and User Guide:
-Installation is fairly easy by using `pip`, just read the steps in the [documentation](https://github.com/MISP/MISP-maltego/blob/master/doc/README.md).
+## Installation
+### Transform Hub
+Open the Transform Hub, locate **ATT&CK - MISP** and press the **Install** button. 
+
+Your transforms will go through Paterva's servers and ours. See the [Transform Hub Disclaimer](https://github.com/MISP/MISP-maltego/blob/master/TRANSFORM_HUB_DISCLAIMER.md) for more information.
+
+- ATT&CK transforms do not require a MISP server or API key to be configured.
+- MISP transforms requires your MISP server to be reachable from the internet! To enter your MISP server URL and key click **Details** on the Transform Hub item and then **Settings** at the bottom right. 
+
+### Local Transform Installation
+If you trust nobody, or just want to connect to your local MISP server you can install everything as local transforms.
+
+These instructions have been tested on Ubuntu 18.04 LTS, but should be similar on other systems.
+1. Download and install [Maltego](https://www.paterva.com/web7/downloads.php)
+2. Install using pip: `sudo pip3 install MISP-maltego`
+3. Generate the Maltego bundle: `canari create-profile MISP_maltego`
+4. Import this bundle in Maltego. 
+   1. Open Maltego
+   2. Click on the home button (Maltego icon, top-left corner).
+   3. Click on 'Import'
+   4. Click on 'Import Configuration'.
+   5. Load the `MISP_maltego.mtz` file and follow the prompts.
+5. Edit `$HOME/.canari/MISP_maltego.conf` and enter your `misp_url` and `misp_key`
+
+## Custom Entities
+MISP-Maltego tries to use as much as possible the default Paterva entities, or the most popular from the community. It however comes with a few custom entities: 
+* **MISPEvent**: A representation of an *Event* on MISP, containing *Attributes* (MISP) / *Entities* (Maltego)
+* **MISPObject**: A way to group associated attributes in a structured way.
+* **MISPGalaxy**: A *Tag* containing much more metadata. Please refer to the [MISP Galaxy](https://github.com/MISP/misp-galaxy) for more information. **MITRE ATT&CK** is for example completely available through MISPGalaxy entities (see use-cases for an example)
+* **Attack Technique**: Attack patterns or techniques, see [MITRE ATT&CK](https://attack.mitre.org/techniques/enterprise/) for more information.
+* **Threat Actor**: Threat actor or intrusion sets.
+* **Software**: Software is a generic term for custom or commercial code, operating system utilities, open-source software, or other tools used to conduct behavior modeled in ATT&CK. 
+
+# Use Cases
+## Transform on existing data
+In this use case we will be using already existing entities and will initiate a transform using MISP. The currently supported entities are: `AS`, `DNSName`, `Domain`, `EmailAddress`, `File`, `Hash`, `IPv4Address`, `NSRecord`, `Person`, `PhoneNumber`, `URL`, `Website`.
+
+Example:
+* create an entity `domain` with the value `1dnscontrol.com`.
+* right click and choose *Local Transforms*  > *MISP_maltego* > *Domain To Event*  
+![animated screenshot](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase1-transform.gif)
+* continue loading transforms on the *MISP Event*
+
+## Transform from MISP Event ID
+While MISP already has a graphing capability we would like to use the power of Maltego to look at the data and expand the work.
+* Create a *MISP Event* and give it an `event id`, or `UUID`
+* One **manual** way is to right click and choose *Local Transforms* > *MISP_maltego* > *Event To Attributes* 
+  * Notice the event is transformed to *Attributes*, *Objects*, *Tags*, *Galaxies* and related *MISP Events*
+  * You can now further transform on an *Object* > *Object To Attributes* and see the content of the object
+![machine transforms](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase2-manual.gif)
+* Alternatively you can also use the **Maltego Machine** to speed up things. 
+   * Click on the *MISP Event* and in the left menu choose *Event to All* in the *Machines* section. 
+![machine transforms](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase2-machine-menu.png)
+   * Notice that the whole event, objects and such will get expanded with data from your MISP instance.
+![animated screenshot](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase2-machine.gif)
+* You can now further transform on any data.
+
+## Which data is already in MISP?
+If you use MISP as central database it can be quite convenient to know which data is present in MISP, and which data is not; especially after using a number of other transforms.
+To permit this MISP-Maltego will always add a green bookmark to all the data that is present in MISP.
+![green bookmark](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase3-bookmark.png)
+
+
+## Searching in MISP using keywords
+As with the MISP attribute search through the MISP Web UI you can use `%` wildcards at the front and end to specify the substring. You might be tempted to always use `%keyword%`, but bare in mind how databases indexes work; a search for `keyword%` will always be much faster than `%keyword`.
+![Search in MISP](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/search_in_misp.gif)
+
+
+## Transform from Galaxy
+Galaxies are actually tags with much more contextual data. Examples are threat actors, malware families, but also the whole MITRE ATT&CK data is available as Galaxy. All this data comes from the [MISP Galaxy](https://github.com/MISP/misp-galaxy) repository. Today the integration is not done using a MISP server because of limitations in MISP.
+You might encounter Galaxies when transforming from MISP Events or Attributes. An alternative use-case is by starting immediately from a Galaxy.
+There are 3 ways to manually create a good Galaxy Entity.
+1. Using a find capability (see below)
+2. Create the Galaxy and set the UUID. You can find the UUIDs in the [MISP Galaxy](https://github.com/MISP/misp-galaxy) repository.
+3. Create the Galaxy with the right tag name; for example: `misp-galaxy:`
+
+To use the magical search feature:
+* Create a *MISP Galaxy* and type the keyword as value.
+* Run the *Galaxy To Relation* transform, notice the search results will appear as connected entities
+* Remove the non-relevant entities, including the your search-keyword
+![animated galaxy search](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase4-galaxy-search.gif)
+
+## Visualize MITRE ATT&CK
+Apply the same steps for MITRE ATT&CK browsing:
+
+![animated ATTACK](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase5-attack.gif)
+
+You might end up with such a graph:
+
+![ATTACK](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase5-attack.png)
+
+## Visualise common ATT&CK patterns
+Having access to a large amount of Threat information through MISP Threat Sharing communities gives you outstanding opportunities to aggregate this information and take the process of trying to understand how all this data fits together telling a broader story to the next level. We are transforming technical data or indicators of compromise (IOCs) into cyber threat intelligence. This is where the analytical challenge begins. [[read more](https://www.misp-project.org/2019/10/27/visualising_common_patterns_attack.html)]
+
+
+## Massively large MISP event? Think before you transform.
+In some communities such as the [COVID-19 MISP](https://www.misp-project.org/covid-19-misp/) some events contain tens of thousands attributes. Loading all the attributes from these events might not be a good idea if you do not have Maltego XL.
+You can see the amount of attributes and objects in the Event properties, so you can think before you click:
+
+![object count](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/event_count_attr1.png)![attribute count](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/event_count_attr2.png)
 
-The [User Guide](https://github.com/MISP/MISP-maltego/blob/master/doc/README.md#use-cases) gives some example use-cases.
 
 
-## Screenshot
-![Screenshot](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/screenshot.png)
 
-![ATT&CK](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/attack.png)
 
 
 ## License

SECURITY.md

@@ -0,0 +1 @@
+For security issues, please refer to the instructions in the main MISP repository: https://github.com/MISP/MISP/

TRANSFORM_HUB_DISCLAIMER.md

@@ -3,13 +3,13 @@
 
 The public Transform Distribution Server (TDS) is located on the Internet and is free for all to use. It’s a convenient way to immediately start writing remote transforms. Since this server is located on Paterva’s infrastructure data (entity, and settings) will be flowing from the Maltego GUI to this server. Paterva states they DO NOT store the details of your transforms (entities, MISP URL, API KEY).
 
-Finally it will flow further to a server managed by the MISP-maltego developer(s), where the transform code runs. We also DO NOT store or look at the details of your transforms (entities, MISP URL, API KEY). As you can see in the code (open source), this data is only used live in memory to provide the transform functionality. The only reasons why we would be seeing this data is by accident; while troubleshooting or by unintentional mis-configuration.
+Finally it will flow further to a server managed by the MISP-maltego developer(s), where the transform code runs and for MISP transforms connects to YOUR server. We DO NOT store or look at the details of your transforms (entities, MISP URL, API KEY). As you can see in the code (open source), this data is only used live in memory to provide the transform functionality. The only reasons why we would be seeing this data is by accident; while troubleshooting or by unintentional mis-configuration.
 
 We do keep standard HTTP logs for troubleshooting and anonymous statistics, although these contain the IP addresses of Paterva's TDS server, and not yours.
 
 **DO NOT use these Transform Hub transforms if you do not agree or if this is in violation with your MISP community.**
 
-**If so, feel free to use the MISP-Maltego transforms locally, where all the code runs on your own system. Installation instructions can be found [here](https://github.com/MISP/MISP-maltego/blob/master/doc/README.md#installation).**
+**If so, feel free to use the MISP-Maltego transforms locally, where all the code runs on your own system. Installation instructions can be found [here](https://github.com/MISP/MISP-maltego/blob/master/README.md#installation).**
 
 You can also run this on your own iTDS server if you have the license. Have a look at the [Dockerfile](https://github.com/MISP/MISP-maltego/blob/master/Dockerfile) for more info.
 

ansible/nginx.conf

@@ -21,6 +21,12 @@ server {
 	root /var/www/html;
 	server_name _;
 
+	location /munin/ {
+		alias /var/cache/munin/www/;
+		index index.html;
+		allow 127.0.0.1;
+		deny all;
+	}
 
 	location / {
 		proxy_set_header X-Real-IP $remote_addr;
@@ -32,4 +38,4 @@ server {
 	location @redirect {
 		return 302 https://github.com/MISP/MISP-maltego;
 	}
-}
+}

ansible/plume.yaml

@@ -168,6 +168,29 @@
       name: plume
       state: started
 
+  # MONITORING
+  #############
+  - name: install munin
+    package:
+      name: ['munin', 'munin-node', 'munin-plugins-extra']
+
+  - name: munin - enabling plugins
+    file:
+      state: link
+      src: '/usr/share/munin/plugins/{{item}}'
+      dest: '/etc/munin/plugins/{{item}}'
+    loop:
+      - nginx_request
+      - nginx_status
+    notify: restart munin-node
+
+  - name: munin - service active and running
+    service:
+      name: munin-node
+      state: started
+      enabled: yes
+
+
   # FIREWALLING
   #############
   - name: firewall logging
@@ -225,3 +248,8 @@
       service:
         name: nginx
         state: restarted
+
+    - name: restart munin-node
+      service:
+        name: munin-node
+        state: restarted

ansible/redeploy.yaml

@@ -0,0 +1,59 @@
+---
+# Install MISP-maltego remote transform using ansible.
+
+- hosts: all
+  become: yes
+  vars:
+    misp_maltego_version: 1.4.5  # TODO change this !!!
+    host_locale: en_US.UTF-8
+    host_locale_dict: {
+      LANG: "{{ host_locale }}",
+      LC_COLLATE: "{{ host_locale }}",
+      LC_CTYPE: "{{ host_locale }}",
+      LC_MESSAGES: "{{ host_locale }}",
+      LC_MONETARY: "{{ host_locale }}",
+      LC_NUMERIC: "{{ host_locale }}",
+      LC_TIME: "{{ host_locale }}",
+      LC_ALL: "{{ host_locale }}",
+    }
+
+  tasks:
+  # use the public pip package
+  - name: install MISP-maltego
+    pip:
+      executable: pip3
+      name: ['MISP-maltego']
+      state: latest
+    notify: restart plume
+
+  # use local git repo instead, useful for development
+  # - name: bundle MISP-maltego
+  #   delegate_to: 127.0.0.1
+  #   command: 
+  #     cmd: python3 setup.py sdist
+  #     chdir: ../
+  #   become: no
+  # - name: copy MISP-maltego
+  #   copy:
+  #     src: ../dist/MISP_maltego-{{misp_maltego_version}}.tar.gz
+  #     dest: /usr/local/src/
+  # - name: install MISP-maltego
+  #   pip:
+  #     executable: /usr/bin/pip3
+  #     name: file:///usr/local/src/MISP_maltego-{{misp_maltego_version}}.tar.gz
+  #     state: forcereinstall
+  #   environment: "{{host_locale_dict}}"
+  #   notify: restart plume
+
+  # - name: remove local MISP-maltego bundle
+  #   delegate_to: 127.0.0.1
+  #   file:
+  #     path: ../dist/MISP_maltego-{{misp_maltego_version}}.tar.gz
+  #     state: absent
+  #   become: no
+
+  handlers:
+    - name: restart plume
+      service:
+        name: plume
+        state: restarted

doc/README.md

@@ -1,90 +0,0 @@
-
-# MISP-Maltego User Guide
-
-This user guide should help you through the installation of **MISP-Maltego**, and should guide you how to use it through a few use-cases. As this is a collaborative project, do not hesitate to propose changes, write other use-cases or raise [feature requests](https://github.com/MISP/MISP-maltego/issues) for missing features.
-
-## Remote Transform Installation
-This is coming soon. An entry will appear in the Transform Hub of Maltego, on which you can simply press the "install" button, and no local code needs to be installed. Your transforms will however go throught Paterva's servers and ours. See the [Transform Hub Disclaimer](https://github.com/MISP/MISP-maltego/blob/master/TRANSFORM_HUB_DISCLAIMER.md) for more information.
-
-**For MISP specific transforms this requires your MISP server to be reachable from the internet!
-ATT&CK transforms do not require a MISP server or API key to be configured.**
-
-## Local Transform Installation
-If you trust nobody, or just want to connect to your local MISP server you can install everything as local transforms.
-
-These instructions have been tested on Ubuntu 18.04 LTS, but should be similar on other systems.
-1. Download and install [Maltego](https://www.paterva.com/web7/downloads.php)
-2. Install using pip: `sudo pip3 install MISP-maltego`
-3. Generate the Maltego bundle: `canari create-profile MISP_maltego`
-4. Import this bundle in Maltego. 
-   1. Open Maltego
-   2. Click on the home button (Maltego icon, top-left corner).
-   3. Click on 'Import'
-   4. Click on 'Import Configuration'.
-   5. Load the `MISP_maltego.mtz` file and follow the prompts.
-5. Edit `$HOME/.canari/MISP_maltego.conf` and enter your `misp_url` and `misp_key`
-
-## Custom Entities
-MISP-Maltego tries to use as much as possible the default Paterva entities, or the most popular from the community. It however comes with a few custom entities: 
-* **MISPEvent**: A representation of an *Event* on MISP, containing *Attributes* (MISP) / *Entities* (Maltego)
-* **MISPObject**: A way to group associated attributes in a structured way.
-* **MISPGalaxy**: A *Tag* containing much more metadata. Please refer to the [MISP Galaxy](https://github.com/MISP/misp-galaxy) for more information. **MITRE ATT&CK** is for example completely available through MISPGalaxy entities (see use-cases for an example)
-* **Attack Technique**: Attack patterns or techniques, see [MITRE ATT&CK](https://attack.mitre.org/techniques/enterprise/) for more information.
-* **Threat Actor**: Threat actor or intrusion sets.
-* **Software**: Software is a generic term for custom or commercial code, operating system utilities, open-source software, or other tools used to conduct behavior modeled in ATT&CK. 
-
-# Use Cases
-## Transform on existing data
-In this use case we will be using already existing entities and will initiate a transform using MISP. The currently supported entities are: `AS`, `DNSName`, `Domain`, `EmailAddress`, `File`, `Hash`, `IPv4Address`, `NSRecord`, `Person`, `PhoneNumber`, `URL`, `Website`.
-
-Example:
-* create an entity `domain` with the value `1dnscontrol.com`.
-* right click and choose *Local Transforms*  > *MISP_maltego* > *Domain To Event*  
-![animated screenshot](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase1-transform.gif)
-* continue loading transforms on the *MISP Event*
-
-## Transform from MISP Event ID
-While MISP already has a graphing capability we would like to use the power of Maltego to look at the data and expand the work.
-* Create a *MISP Event* and give it an `event id`, or `UUID`
-* One **manual** way is to right click and choose *Local Transforms* > *MISP_maltego* > *Event To Attributes* 
-  * Notice the event is transformed to *Attributes*, *Objects*, *Tags*, *Galaxies* and related *MISP Events*
-  * You can now further transform on an *Object* > *Object To Attributes* and see the content of the object
-![machine transforms](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase2-manual.gif)
-* Alternatively you can also use the **Maltego Machine** to speed up things. 
-   * Click on the *MISP Event* and in the left menu choose *Event to All* in the *Machines* section. 
-![machine transforms](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase2-machine-menu.png)
-   * Notice that the whole event, objects and such will get expanded with data from your MISP instance.
-![animated screenshot](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase2-machine.gif)
-* You can now further transform on any data.
-
-## Which data is already in MISP?
-If you use MISP as central database it can be quite convenient to know which data is present in MISP, and which data is not; especially after using a number of other transforms.
-To permit this MISP-Maltego will always add a green bookmark to all the data that is present in MISP.
-![green bookmark](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase3-bookmark.png)
-
-
-## Transform from Galaxy
-Galaxies are actually tags with much more contextual data. Examples are threat actors, malware families, but also the whole MITRE ATT&CK data is available as Galaxy. All this data comes from the [MISP Galaxy](https://github.com/MISP/misp-galaxy) repository. Today the integration is not done using a MISP server because of limitations in MISP.
-You might encounter Galaxies when transforming from MISP Events or Attributes. An alternative use-case is by starting immediately from a Galaxy.
-There are 3 ways to manually create a good Galaxy Entity.
-1. Using a find capability (see below)
-2. Create the Galaxy and set the UUID. You can find the UUIDs in the [MISP Galaxy](https://github.com/MISP/misp-galaxy) repository.
-3. Create the Galaxy with the right tag name; for example: `misp-galaxy:`
-
-To use the magical search feature:
-* Create a *MISP Galaxy* and type the keyword as value.
-* Run the *Galaxy To Relation* transform, notice the search results will appear as connected entities
-* Remove the non-relevant entities, including the your search-keyword
-![animated galaxy search](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase4-galaxy-search.gif)
-
-## Visualize MITRE ATT&CK
-Apply the same steps for MITRE ATT&CK browsing:
-
-![animated ATTACK](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase5-attack.gif)
-
-You might end up with such a graph:
-
-![ATTACK](https://raw.githubusercontent.com/MISP/MISP-maltego/master/doc/img/usecase5-attack.png)
-
-## Visualise common ATT&CK patterns
-Having access to a large amount of Threat information through MISP Threat Sharing communities gives you outstanding opportunities to aggregate this information and take the process of trying to understand how all this data fits together telling a broader story to the next level. We are transforming technical data or indicators of compromise (IOCs) into cyber threat intelligence. This is where the analytical challenge begins. [[read more](https://www.misp-project.org/2019/10/27/visualising_common_patterns_attack.html)]

k8s.yaml

@@ -1,43 +0,0 @@
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: misp-maltego
-spec:
-  selector:
-    matchLabels:
-      app: misp-maltego
-  replicas: 1
-  template:
-    metadata:
-      labels:
-        app: misp-maltego
-    spec:
-      containers:
-        - name: misp-maltego
-          image: cvandeplas/misp-maltego
-          imagePullPolicy: Always
-          ports:
-            - name: listen-port
-              containerPort: 8080
-          resources:
-            requests:
-              cpu: 10m
-              memory: 4Gi
-            limits:
-              cpu: 1
-              memory: 4Gi
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: misp-maltego
-  annotations:
-    dev.okteto.com/auto-ingress: "true"
-spec:
-  type: ClusterIP  
-  ports:
-  - name: "misp-maltego"
-    port: 8080
-  selector:
-    app: misp-maltego

setup.py

@@ -9,7 +9,7 @@ setup(
     name='MISP_maltego',
     author='Christophe Vandeplas',
     # also update version in util.py
-    version='1.4.4',
+    version='1.4.6',
     author_email='christophe@vandeplas.com',
     maintainer='Christophe Vandeplas',
     url='https://github.com/MISP/MISP-maltego',
@@ -36,7 +36,7 @@ setup(
     python_requires='>=3.5',
     install_requires=[
         'canari>=3.3.10,<4',
-        'PyMISP>=2.4.114'
+        'PyMISP>=2.4.127'
     ],
     dependency_links=[
         # custom links for the install_requires

src/MISP_maltego/resources/maltego/misp_MISPEventToAll.machine

@@ -3,16 +3,22 @@ machine("misp.MISPEventToAll",
         author:"Christophe Vandeplas",
         description: "Automatically expands MISP Objects to their attributes") {
     start {
-        run("MISP_maltego.EventToAttributes")
-        run("MISP_maltego.AttributeToEvent")
         paths {
             path {
-                run("MISP_maltego.ObjectToAttributes")
-                // run("MISP_maltego.AttributeToEvent")
+                run("MISP_maltego.EventToAttributes")
+                paths {
+                    path {
+                        run("MISP_maltego.AttributeToEvent")                
+                    }
+                    path {
+                        run("MISP_maltego.ObjectToAttributes")
+                    }
+                }
             }
             path {
+                run("MISP_maltego.EventToTags")
                 run("MISP_maltego.GalaxyToRelations")
-            }
+            }  
         }
         
     }

src/MISP_maltego/resources/maltego/misp_MISPEventToAttributes.machine

@@ -4,11 +4,14 @@ machine("misp.MISPEventToAttributes",
         description: "Also automatically expands MISP Objects to their attributes") {
     start {
         paths {
-            run("MISP_maltego.EventToAttributes")
             path {
                 run("MISP_maltego.EventToObjects")
                 run("MISP_maltego.ObjectToAttributes")
             }
+            path {
+                run("MISP_maltego.EventToAttributes")
+            
+            }
         }
     }
 }

src/MISP_maltego/transforms/attributetoevent.py

@@ -1,8 +1,8 @@
-from canari.maltego.entities import Unknown
+from canari.maltego.entities import Unknown, Hashtag
 from canari.maltego.transform import Transform
-# from canari.framework import EnableDebugWindow
-from MISP_maltego.transforms.common.util import check_update, get_misp_connection, event_to_entity, object_to_entity, get_attribute_in_event, get_attribute_in_object, attribute_to_entity, get_entity_property
-from canari.maltego.message import LinkDirection
+from MISP_maltego.transforms.common.entities import MISPGalaxy
+from MISP_maltego.transforms.common.util import check_update, MISPConnection, event_to_entity, get_attribute_in_event, get_attribute_in_object, attribute_to_entity, get_entity_property, search_galaxy_cluster, galaxycluster_to_entity
+from canari.maltego.message import LinkDirection, Bookmark
 
 __author__ = 'Christophe Vandeplas'
 __copyright__ = 'Copyright 2018, MISP_maltego Project'
@@ -15,33 +15,93 @@ __email__ = 'christophe@vandeplas.com'
 __status__ = 'Development'
 
 
-# @EnableDebugWindow
-class AttributeInMISP(Transform):
-    """Green bookmark if known in MISP"""
+class SearchInMISP(Transform):
+    """Use % at the front/end for wildcard search"""
     input_type = Unknown
-    display_name = 'in MISP?'
+    display_name = 'Search in MISP'
+    description = "Use % at the front/end for wildcard search"
     remote = True
 
     def do_transform(self, request, response, config):
         response += check_update(config)
-        maltego_misp_attribute = request.entity
-        # skip MISP Events (value = int)
-        try:
-            int(maltego_misp_attribute.value)
+        link_label = 'Search result'
+
+        if 'properties.mispevent' in request.entity.fields:
+            conn = MISPConnection(config, request.parameters)
+            # if event_id
+            try:
+                if request.entity.value == '0':
+                    return response
+                eventid = int(request.entity.value)
+                events_json = conn.misp.search(controller='events', eventid=eventid, with_attachments=False)
+                for e in events_json:
+                    response += event_to_entity(e, link_label=link_label, link_direction=LinkDirection.OutputToInput)
+                return response
+            except ValueError:
+                pass
+            # if event_info string as value
+            events_json = conn.misp.search(controller='events', eventinfo=request.entity.value, with_attachments=False)
+            for e in events_json:
+                response += event_to_entity(e, link_label=link_label, link_direction=LinkDirection.OutputToInput)
             return response
-        except Exception:
-            pass
 
-        misp = get_misp_connection(config, request.parameters)
-        events_json = misp.search(controller='events', value=maltego_misp_attribute.value, with_attachments=False)
+        # From galaxy or Hashtag
+        if 'properties.mispgalaxy' in request.entity.fields or 'properties.temp' in request.entity.fields or 'twitter.hashtag' in request.entity.fields:
+            if request.entity.value == '-':
+                return response
+            # First search in galaxies
+            keyword = get_entity_property(request.entity, 'Temp')
+            if not keyword:
+                keyword = request.entity.value
+            # assume the user is searching for a cluster based on a substring.
+            # Search in the list for those that match and return galaxy entities'
+            potential_clusters = search_galaxy_cluster(keyword)
+            # LATER check if duplicates are possible
+            if potential_clusters:
+                for potential_cluster in potential_clusters:
+                    new_entity = galaxycluster_to_entity(potential_cluster, link_label=link_label)
+                    # LATER support the type_filter - unfortunately this is not possible, we need Canari to tell us the original entity type
+                    if isinstance(new_entity, MISPGalaxy):
+                        response += new_entity
+
+            # from Hashtag search also in tags
+            if 'properties.temp' in request.entity.fields or 'twitter.hashtag' in request.entity.fields:
+                keyword = get_entity_property(request.entity, 'Temp')
+                if not keyword:
+                    keyword = request.entity.value
+                conn = MISPConnection(config, request.parameters)
+                result = conn.misp.direct_call('tags/search', {'name': keyword})
+                for t in result:
+                    # skip misp-galaxies as we have processed them earlier on
+                    if t['Tag']['name'].startswith('misp-galaxy'):
+                        continue
+                    # In this case we do not filter away those we add as notes, as people might want to pivot on it explicitly.
+                    response += Hashtag(t['Tag']['name'], link_label=link_label, bookmark=Bookmark.Green)
+
+            return response
+
+        # for all other normal entities
+        conn = MISPConnection(config, request.parameters)
+
         # we need to do really rebuild the Entity from scratch as request.entity is of type Unknown
-        for e in events_json:
-            attr = get_attribute_in_event(e, maltego_misp_attribute.value)
-            if attr:
-                for item in attribute_to_entity(attr, only_self=True):
-                    response += item
-        return response
+        # TODO First try to build the object, then only attributes (for those that are not in object, or for all?)
+        # TODO check for the right version of MISP before, it needs to be 2.4.127 or higher.
+        # obj_json = conn.misp.search(controller='objects', value=request.entity.value, with_attachments=False)
+        # for o in obj_json:
+        #         for item in attribute_to_entity(attr, only_self=True, link_label=link_label):
+        #             response += item
+        #     # find the value as object, and return the object
+        #     if 'Object' in e['Event']:
+        #         for o in e['Event']['Object']:
+        #             if get_attribute_in_object(o, attribute_value=request.entity.value, substring=True).get('value'):
+        #                 response += conn.object_to_entity(o, link_label=link_label)
 
+        attr_json = conn.misp.search(controller='attributes', value=request.entity.value, with_attachments=False)
+        for a in attr_json['Attribute']:
+            for item in attribute_to_entity(a, only_self=True, link_label=link_label):
+                response += item
+
+        return response
 
 # placeholder for https://github.com/MISP/MISP-maltego/issues/11
 # waiting for support of CIDR search through the REST API
@@ -56,7 +116,7 @@ class AttributeInMISP(Transform):
 #         misp = get_misp_connection(config, request.parameters)
 #         import ipaddress
 #         ip_start, ip_end = maltego_misp_attribute.value.split('-')
-#         # FIXME make this work with IPv4 and IPv6
+#         # LATER make this work with IPv4 and IPv6
 #         # automagically detect the different CIDRs
 #         cidrs = ipaddress.summarize_address_range(ipaddress.IPv4Address(ip_start), ipaddress.IPv4Address(ip_end))
 #         for cidr in cidrs:
@@ -66,10 +126,9 @@ class AttributeInMISP(Transform):
 #         return response
 
 
-# @EnableDebugWindow
 class AttributeToEvent(Transform):
     input_type = Unknown
-    display_name = 'to MISP Event'
+    display_name = 'To MISP Events'
     remote = True
 
     def do_transform(self, request, response, config):
@@ -84,37 +143,37 @@ class AttributeToEvent(Transform):
             # placeholder for https://github.com/MISP/MISP-maltego/issues/11
             pass
 
-        misp = get_misp_connection(config, request.parameters)
+        conn = MISPConnection(config, request.parameters)
         # from Galaxy
         if 'properties.mispgalaxy' in request.entity.fields:
             tag_name = get_entity_property(request.entity, 'tag_name')
             if not tag_name:
                 tag_name = request.entity.value
-            events_json = misp.search(controller='events', tags=tag_name, with_attachments=False)
+            events_json = conn.misp.search(controller='events', tags=tag_name, with_attachments=False)
             for e in events_json:
                 response += event_to_entity(e, link_direction=LinkDirection.OutputToInput)
             return response
         # from Object
         elif 'properties.mispobject' in request.entity.fields:
             if request.entity.fields.get('event_id'):
-                events_json = misp.search(controller='events', eventid=request.entity.fields.get('event_id').value, with_attachments=False)
+                events_json = conn.misp.search(controller='events', eventid=request.entity.fields.get('event_id').value, with_attachments=False)
                 for e in events_json:
                     response += event_to_entity(e, link_direction=LinkDirection.OutputToInput)
                 return response
             else:
                 return response
         # from Hashtag
-        elif 'properties.temp' in request.entity.fields:
+        elif 'properties.temp' in request.entity.fields or 'twitter.hashtag' in request.entity.fields:
             tag_name = get_entity_property(request.entity, 'Temp')
             if not tag_name:
                 tag_name = request.entity.value
-            events_json = misp.search(controller='events', tags=tag_name, with_attachments=False)
+            events_json = conn.misp.search_index(tags=tag_name)
             for e in events_json:
-                response += event_to_entity(e, link_direction=LinkDirection.OutputToInput)
+                response += event_to_entity({'Event': e}, link_direction=LinkDirection.OutputToInput)
             return response
         # standard Entities (normal attributes)
         else:
-            events_json = misp.search(controller='events', value=request.entity.value, with_attachments=False)
+            events_json = conn.misp.search(controller='events', value=request.entity.value, with_attachments=False)
 
         # return the MISPEvent or MISPObject of the attribute
         for e in events_json:
@@ -126,6 +185,6 @@ class AttributeToEvent(Transform):
             if 'Object' in e['Event']:
                 for o in e['Event']['Object']:
                     if get_attribute_in_object(o, attribute_value=request.entity.value).get('value'):
-                        response += object_to_entity(o, link_direction=LinkDirection.OutputToInput)
+                        response += conn.object_to_entity(o, link_direction=LinkDirection.OutputToInput)
 
         return response

src/MISP_maltego/transforms/common/entities.py

@@ -32,6 +32,8 @@ class MISPEvent(Entity):
     # threat_level = EnumEntityField('type.enum', choices=['Undefined', 'Low', 'Medium', 'High'], display_name='Threat Level')
     # analysis = EnumEntityField('type.enum', choices=['Initial', 'Ongoing', 'Completed'])
     # org = StringEntityField('type.str', display_name='Organisation')
+    count_attributes = IntegerEntityField('count_attributes', display_name="# attributes", matching_rule=MatchingRule.Loose)
+    count_objects = IntegerEntityField('count_objects', display_name="# objects", matching_rule=MatchingRule.Loose)
 
 
 class MISPObject(Entity):

src/MISP_maltego/transforms/common/util.py

@@ -12,13 +12,10 @@ import requests
 import tempfile
 import time
 
-# FIXME from galaxy 'to MISP Event' is confusing
-
-__version__ = '1.4.4'  # also update version in setup.py
+__version__ = '1.4.6'  # also update version in setup.py
 
 tag_note_prefixes = ['tlp:', 'PAP:', 'de-vs:', 'euci:', 'fr-classif:', 'nato:']
 
-misp_connection = None
 update_url = 'https://raw.githubusercontent.com/MISP/MISP-maltego/master/setup.py'
 local_path_root = os.path.join(tempfile.gettempdir(), 'MISP-maltego')
 local_path_version = os.path.join(local_path_root, 'versioncheck')
@@ -64,37 +61,116 @@ def check_update(config):
     return None
 
 
-def get_misp_connection(config=None, parameters=None):
-    global misp_connection
-    if misp_connection:
-        return misp_connection
-    if not config:
-        raise MaltegoException("ERROR: MISP connection not yet established, and config not provided as parameter.")
-    misp_verify = True
-    misp_debug = False
-    misp_url = None
-    misp_key = None
-    try:
-        if is_local_exec_mode():
-            misp_url = config['MISP_maltego.local.misp_url']
-            misp_key = config['MISP_maltego.local.misp_key']
-            if config['MISP_maltego.local.misp_verify'] in ['False', 'false', 0, 'no', 'No']:
-                misp_verify = False
-            if config['MISP_maltego.local.misp_debug'] in ['True', 'true', 1, 'yes', 'Yes']:
-                misp_debug = True
-        if is_remote_exec_mode():
+class MISPConnection():
+    def __init__(self, config=None, parameters=None):
+        self.misp = None
+
+        if not config:
+            raise MaltegoException("ERROR: MISP connection not yet established, and config not provided as parameter.")
+        misp_verify = True
+        misp_debug = False
+        misp_url = None
+        misp_key = None
+        try:
+            if is_local_exec_mode():
+                misp_url = config['MISP_maltego.local.misp_url']
+                misp_key = config['MISP_maltego.local.misp_key']
+                if config['MISP_maltego.local.misp_verify'] in ['False', 'false', 0, 'no', 'No']:
+                    misp_verify = False
+                if config['MISP_maltego.local.misp_debug'] in ['True', 'true', 1, 'yes', 'Yes']:
+                    misp_debug = True
+            else:
+                try:
+                    misp_url = parameters['mispurl'].value
+                    misp_key = parameters['mispkey'].value
+                except AttributeError:
+                    raise MaltegoException("ERROR: mispurl and mispkey need to be set to something valid")
+            self.misp = PyMISP(url=misp_url, key=misp_key, ssl=misp_verify, debug=misp_debug, tool='misp_maltego', timeout=(2, 60))
+        except Exception:
+            if is_local_exec_mode():
+                raise MaltegoException("ERROR: Cannot connect to MISP server. Please verify your MISP_Maltego.conf settings.")
+            else:
+                raise MaltegoException("ERROR: Cannot connect to MISP server. Please verify your settings (MISP URL and API key), and ensure the MISP server is reachable from the internet.")
+
+    def object_to_entity(self, o, link_label=None, link_direction=LinkDirection.InputToOutput):
+        # find a nice icon for it
+        try:
+            icon_url = mapping_object_icon[o['name']]
+        except KeyError:
+            # it's not in our mapping, just ignore and leave the default icon
+            icon_url = None
+        # Generate a human readable display-name:
+        # - find the first RequiredOneOf that exists
+        # - if none, use the first RequiredField
+        # LATER further finetune the human readable version of this object
+        o_template = self.misp.get_object_template(o['template_uuid'])
+        human_readable = None
+        try:
+            found = False
+            while not found:  # the while loop is broken once something is found, or the requiredOneOf has no elements left
+                required_ote_type = o_template['ObjectTemplate']['requirements']['requiredOneOf'].pop(0)
+                for ote in o_template['ObjectTemplateElement']:
+                    if ote['object_relation'] == required_ote_type:
+                        required_a_type = ote['type']
+                        break
+                for a in o['Attribute']:
+                    if a['type'] == required_a_type:
+                        human_readable = '{}:\n{}'.format(o['name'], a['value'])
+                        found = True
+                        break
+        except Exception:
+            pass
+        if not human_readable:
             try:
-                misp_url = parameters['mispurl'].value
-                misp_key = parameters['mispkey'].value
-            except AttributeError:
-                raise MaltegoException("ERROR: mispurl and mispkey need to be set to something valid")
-        misp_connection = PyMISP(misp_url, misp_key, misp_verify, 'json', misp_debug, tool='misp_maltego')
-    except Exception:
-        if is_local_exec_mode():
-            raise MaltegoException("ERROR: Cannot connect to MISP server. Please verify your MISP_Maltego.conf settings.")
-        if is_remote_exec_mode():
-            raise MaltegoException("ERROR: Cannot connect to MISP server. Please verify your settings (MISP URL and API key), and ensure the MISP server is reachable from the internet.")
-    return misp_connection
+                found = False
+                parts = []
+                for required_ote_type in o_template['ObjectTemplate']['requirements']['required']:
+                    for ote in o_template['ObjectTemplateElement']:
+                        if ote['object_relation'] == required_ote_type:
+                            required_a_type = ote['type']
+                            break
+                    for a in o['Attribute']:
+                        if a['type'] == required_a_type:
+                            parts.append(a['value'])
+                            break
+                human_readable = '{}:\n{}'.format(o['name'], '|'.join(parts))
+            except Exception:
+                human_readable = o['name']
+        return MISPObject(
+            human_readable,
+            uuid=o['uuid'],
+            event_id=int(o['event_id']),
+            meta_category=o.get('meta_category'),
+            description=o.get('description'),
+            comment=o.get('comment'),
+            icon_url=icon_url,
+            link_label=link_label,
+            link_direction=link_direction,
+            bookmark=Bookmark.Green
+        )
+
+    def object_to_relations(self, o, e):
+        # process forward and reverse references, so just loop over all the objects of the event
+        if 'Object' in e['Event']:
+            for eo in e['Event']['Object']:
+                if 'ObjectReference' in eo:
+                    for ref in eo['ObjectReference']:
+                        # we have found original object. Expand to the related object and attributes
+                        if eo['uuid'] == o['uuid']:
+                            # the reference is an Object
+                            if ref.get('Object'):
+                                # get the full object in the event, as our objectReference included does not contain everything we need
+                                sub_object = get_object_in_event(ref['Object']['uuid'], e)
+                                yield self.object_to_entity(sub_object, link_label=ref['relationship_type'])
+                            # the reference is an Attribute
+                            if ref.get('Attribute'):
+                                ref['Attribute']['event_id'] = ref['event_id']   # LATER remove this ugly workaround - object can't be requested directly from MISP using the uuid, and to find a full object we need the event_id
+                                for item in attribute_to_entity(ref['Attribute'], link_label=ref['relationship_type']):
+                                    yield item
+
+                        # reverse-lookup - this is another objects relating the original object
+                        if ref['referenced_uuid'] == o['uuid']:
+                            yield self.object_to_entity(eo, link_label=ref['relationship_type'], link_direction=LinkDirection.OutputToInput)
 
 
 def entity_obj_to_entity(entity_obj, v, t, **kwargs):
@@ -127,15 +203,15 @@ def attribute_to_entity(a, link_label=None, event_tags=[], only_self=False):
 
     # complement the event tags with the attribute tags.
     if 'Tag' in a and not only_self:
-            for t in a['Tag']:
-                combined_tags.append(t['name'])
-                # ignore all misp-galaxies
-                if t['name'].startswith('misp-galaxy'):
-                    continue
-                # ignore all those we add as notes
-                if tag_matches_note_prefix(t['name']):
-                    continue
-                yield Hashtag(t['name'], bookmark=Bookmark.Green)
+        for t in a['Tag']:
+            combined_tags.append(t['name'])
+            # ignore all misp-galaxies
+            if t['name'].startswith('misp-galaxy'):
+                continue
+            # ignore all those we add as notes
+            if tag_matches_note_prefix(t['name']):
+                continue
+            yield Hashtag(t['name'], bookmark=Bookmark.Green)
 
     notes = convert_tags_to_note(combined_tags)
 
@@ -143,6 +219,7 @@ def attribute_to_entity(a, link_label=None, event_tags=[], only_self=False):
     if a['type'] in ('url', 'uri'):
         yield(URL(url=a['value'], short_title=a['value'], link_label=link_label, notes=notes, bookmark=Bookmark.Green))
         return
+    # FIXME implement attachment screenshot type
 
     # attribute is from an object, and a relation gives better understanding of the type of attribute
     if a.get('object_relation') and mapping_misp_to_maltego.get(a['object_relation']):
@@ -173,66 +250,7 @@ def attribute_to_entity(a, link_label=None, event_tags=[], only_self=False):
 
     # not supported in our maltego mapping are not handled
 
-    # LATER : relationships from attributes - not yet supported by MISP yet, but there are references in the datamodel
-
-
-def object_to_entity(o, link_label=None, link_direction=LinkDirection.InputToOutput):
-    misp = get_misp_connection()
-    # find a nice icon for it
-    try:
-        icon_url = mapping_object_icon[o['name']]
-    except KeyError:
-        # it's not in our mapping, just ignore and leave the default icon
-        icon_url = None
-    # Generate a human readable display-name:
-    # - find the first RequiredOneOf that exists
-    # - if none, use the first RequiredField
-    # LATER further finetune the human readable version of this object
-    o_template = misp.get_object_template(o['template_uuid'])
-    human_readable = None
-    try:
-        found = False
-        while not found:  # the while loop is broken once something is found, or the requiredOneOf has no elements left
-            required_ote_type = o_template['ObjectTemplate']['requirements']['requiredOneOf'].pop(0)
-            for ote in o_template['ObjectTemplateElement']:
-                if ote['object_relation'] == required_ote_type:
-                    required_a_type = ote['type']
-                    break
-            for a in o['Attribute']:
-                if a['type'] == required_a_type:
-                    human_readable = '{}:\n{}'.format(o['name'], a['value'])
-                    found = True
-                    break
-    except Exception:
-        pass
-    if not human_readable:
-        try:
-            found = False
-            parts = []
-            for required_ote_type in o_template['ObjectTemplate']['requirements']['required']:
-                for ote in o_template['ObjectTemplateElement']:
-                    if ote['object_relation'] == required_ote_type:
-                        required_a_type = ote['type']
-                        break
-                for a in o['Attribute']:
-                    if a['type'] == required_a_type:
-                        parts.append(a['value'])
-                        break
-            human_readable = '{}:\n{}'.format(o['name'], '|'.join(parts))
-        except Exception:
-            human_readable = o['name']
-    return MISPObject(
-        human_readable,
-        uuid=o['uuid'],
-        event_id=int(o['event_id']),
-        meta_category=o.get('meta_category'),
-        description=o.get('description'),
-        comment=o.get('comment'),
-        icon_url=icon_url,
-        link_label=link_label,
-        link_direction=link_direction,
-        bookmark=Bookmark.Green
-    )
+    # LATER relationships from attributes - not yet supported by MISP yet, but there are references in the datamodel
 
 
 def object_to_attributes(o, e):
@@ -248,37 +266,13 @@ def object_to_attributes(o, e):
             yield item
 
 
-def object_to_relations(o, e):
-    # process forward and reverse references, so just loop over all the objects of the event
-    if 'Object' in e['Event']:
-        for eo in e['Event']['Object']:
-            if 'ObjectReference' in eo:
-                for ref in eo['ObjectReference']:
-                    # we have found original object. Expand to the related object and attributes
-                    if eo['uuid'] == o['uuid']:
-                        # the reference is an Object
-                        if ref.get('Object'):
-                            # get the full object in the event, as our objectReference included does not contain everything we need
-                            sub_object = get_object_in_event(ref['Object']['uuid'], e)
-                            yield object_to_entity(sub_object, link_label=ref['relationship_type'])
-                        # the reference is an Attribute
-                        if ref.get('Attribute'):
-                            ref['Attribute']['event_id'] = ref['event_id']   # LATER remove this ugly workaround - object can't be requested directly from MISP using the uuid, and to find a full object we need the event_id
-                            for item in attribute_to_entity(ref['Attribute'], link_label=ref['relationship_type']):
-                                yield item
-
-                    # reverse-lookup - this is another objects relating the original object
-                    if ref['referenced_uuid'] == o['uuid']:
-                        yield object_to_entity(eo, link_label=ref['relationship_type'], link_direction=LinkDirection.OutputToInput)
-
-
 def get_object_in_event(uuid, e):
     for o in e['Event']['Object']:
         if o['uuid'] == uuid:
             return o
 
 
-def get_attribute_in_object(o, attribute_type=False, attribute_value=False, drop=False):
+def get_attribute_in_object(o, attribute_type=False, attribute_value=False, drop=False, substring=False):
     '''Gets the first attribute of a specific type within an object'''
     found_attribute = {'value': ''}
     for i, a in enumerate(o['Attribute']):
@@ -291,22 +285,87 @@ def get_attribute_in_object(o, attribute_type=False, attribute_value=False, drop
             found_attribute = a.copy()
             if drop:    # drop the attribute from the object
                 o['Attribute'].pop(i)
+            break
         if '|' in a['type'] or a['type'] == 'malware-sample':
             if attribute_value in a['value'].split('|'):
                 found_attribute = a.copy()
                 if drop:    # drop the attribute from the object
                     o['Attribute'].pop(i)
+                break
+        # substring matching
+        if substring:
+            keyword = attribute_value.strip('%')
+            if attribute_value.startswith('%') and attribute_value.endswith('%'):
+                if attribute_value in a['value']:
+                    found_attribute = a.copy()
+                    if drop:    # drop the attribute from the object
+                        o['Attribute'].pop(i)
+                    break
+                if '|' in a['type'] or a['type'] == 'malware-sample':
+                    val1, val2 = a['value'].split('|')
+                    if attribute_value in val1 or attribute_value in val2:
+                        found_attribute = a.copy()
+                        if drop:    # drop the attribute from the object
+                            o['Attribute'].pop(i)
+                        break
+            elif attribute_value.startswith('%'):
+                if a['value'].endswith(keyword):
+                    found_attribute = a.copy()
+                    if drop:    # drop the attribute from the object
+                        o['Attribute'].pop(i)
+                    break
+                if '|' in a['type'] or a['type'] == 'malware-sample':
+                    val1, val2 = a['value'].split('|')
+                    if val1.endswith(keyword) or val2.endswith(keyword):
+                        found_attribute = a.copy()
+                        if drop:    # drop the attribute from the object
+                            o['Attribute'].pop(i)
+                        break
 
+            elif attribute_value.endswith('%'):
+                if a['value'].startswith(keyword):
+                    return a
+                if '|' in a['type'] or a['type'] == 'malware-sample':
+                    val1, val2 = a['value'].split('|')
+                    if val1.startswith(keyword) or val2.startswith(keyword):
+                        found_attribute = a.copy()
+                        if drop:    # drop the attribute from the object
+                            o['Attribute'].pop(i)
+                        break
     return found_attribute
 
 
-def get_attribute_in_event(e, attribute_value):
+def get_attribute_in_event(e, attribute_value, substring=False):
     for a in e['Event']["Attribute"]:
         if a['value'] == attribute_value:
             return a
         if '|' in a['type'] or a['type'] == 'malware-sample':
             if attribute_value in a['value'].split('|'):
                 return a
+        if substring:
+            keyword = attribute_value.strip('%')
+            if attribute_value.startswith('%') and attribute_value.endswith('%'):
+                if attribute_value in a['value']:
+                    return a
+                if '|' in a['type'] or a['type'] == 'malware-sample':
+                    val1, val2 = a['value'].split('|')
+                    if attribute_value in val1 or attribute_value in val2:
+                        return a
+            elif attribute_value.startswith('%'):
+                if a['value'].endswith(keyword):
+                    return a
+                if '|' in a['type'] or a['type'] == 'malware-sample':
+                    val1, val2 = a['value'].split('|')
+                    if val1.endswith(keyword) or val2.endswith(keyword):
+                        return a
+
+            elif attribute_value.endswith('%'):
+                if a['value'].startswith(keyword):
+                    return a
+                if '|' in a['type'] or a['type'] == 'malware-sample':
+                    val1, val2 = a['value'].split('|')
+                    if val1.startswith(keyword) or val2.startswith(keyword):
+                        return a
 
     return None
 
@@ -329,7 +388,7 @@ def tag_matches_note_prefix(tag):
     return False
 
 
-def event_to_entity(e, link_style=LinkStyle.Normal, link_direction=LinkDirection.InputToOutput):
+def event_to_entity(e, link_style=LinkStyle.Normal, link_label=None, link_direction=LinkDirection.InputToOutput):
     tags = []
     if 'Tag' in e['Event']:
         for t in e['Event']['Tag']:
@@ -340,7 +399,10 @@ def event_to_entity(e, link_style=LinkStyle.Normal, link_direction=LinkDirection
         uuid=e['Event']['uuid'],
         info=e['Event']['info'],
         link_style=link_style,
+        link_label=link_label,
         link_direction=link_direction,
+        count_attributes=len(e['Event'].get('Attribute') or ""),
+        count_objects=len(e['Event'].get('Object') or ""),
         notes=notes,
         bookmark=Bookmark.Green)
 
@@ -354,7 +416,7 @@ def galaxycluster_to_entity(c, link_label=None, link_direction=LinkDirection.Inp
     else:
         synonyms = ''
 
-    galaxy_cluster = get_galaxy_cluster(c['uuid'])
+    galaxy_cluster = get_galaxy_cluster(uuid=c['uuid'])
     # map the 'icon' name from the cluster to the icon filename of the intelligence-icons repository
     try:
         icon_url = mapping_galaxy_icon[galaxy_cluster['icon']]
@@ -383,9 +445,9 @@ def galaxycluster_to_entity(c, link_label=None, link_direction=LinkDirection.Inp
 
 # LATER this uses the galaxies from github as the MISP web UI does not fully support the Galaxies in the webui.
 # See https://github.com/MISP/MISP/issues/3801
-galaxy_archive_url = 'https://github.com/MISP/misp-galaxy/archive/master.zip'
+galaxy_archive_url = 'https://github.com/MISP/misp-galaxy/archive/main.zip'
 local_path_uuid_mapping = os.path.join(local_path_root, 'MISP_maltego_galaxy_mapping.json')
-local_path_clusters = os.path.join(local_path_root, 'misp-galaxy-master', 'clusters')
+local_path_clusters = os.path.join(local_path_root, 'misp-galaxy-main', 'clusters')
 galaxy_cluster_uuids = None
 
 
@@ -418,6 +480,8 @@ def galaxy_update_local_copy(force=False):
             zf.extractall(local_path_root)
             zf.close()
         except Exception:
+            # remove the lock
+            os.remove(lockfile)
             raise(MaltegoException("ERROR: Could not download Galaxy data from htts://github.com/MISP/MISP-galaxy/. Please check internet connectivity."))
 
         # generate the uuid mapping and save it to a file
@@ -464,7 +528,7 @@ def galaxy_load_cluster_mapping():
     return cluster_uuids
 
 
-def get_galaxy_cluster(uuid=None, tag=None):
+def get_galaxy_cluster(uuid=None, tag=None, request_entity=None):
     global galaxy_cluster_uuids
     if not galaxy_cluster_uuids:
         galaxy_cluster_uuids = galaxy_load_cluster_mapping()
@@ -475,6 +539,13 @@ def get_galaxy_cluster(uuid=None, tag=None):
         for item in galaxy_cluster_uuids.values():
             if item['tag_name'] == tag:
                 return item
+    if request_entity:
+        if request_entity.uuid:
+            return get_galaxy_cluster(uuid=request_entity.uuid)
+        elif request_entity.tag_name:
+            return get_galaxy_cluster(tag=request_entity.tag_name)
+        elif request_entity.name:
+            return get_galaxy_cluster(tag=request_entity.name)
 
 
 def search_galaxy_cluster(keyword):
@@ -482,14 +553,42 @@ def search_galaxy_cluster(keyword):
     global galaxy_cluster_uuids
     if not galaxy_cluster_uuids:
         galaxy_cluster_uuids = galaxy_load_cluster_mapping()
-    for item in galaxy_cluster_uuids.values():
-        if keyword in item['tag_name'].lower():
-            yield item
-        else:
-            if 'meta' in item and 'synonyms' in item['meta']:
-                for synonym in item['meta']['synonyms']:
-                    if keyword in synonym.lower():
-                        yield item
+
+    # % only at start
+    if keyword.startswith('%') and not keyword.endswith('%'):
+        keyword = keyword.strip('%')
+        for item in galaxy_cluster_uuids.values():
+            if item['value'].lower().endswith(keyword):
+                yield item
+            else:
+                if 'meta' in item and 'synonyms' in item['meta']:
+                    for synonym in item['meta']['synonyms']:
+                        if synonym.lower().endswith(keyword):
+                            yield item
+
+    # % only at end
+    elif keyword.endswith('%') and not keyword.startswith('%'):
+        keyword = keyword.strip('%')
+        for item in galaxy_cluster_uuids.values():
+            if item['value'].lower().startswith(keyword):
+                yield item
+            else:
+                if 'meta' in item and 'synonyms' in item['meta']:
+                    for synonym in item['meta']['synonyms']:
+                        if synonym.lower().startswith(keyword):
+                            yield item
+
+    # search substring assuming % at start and end
+    else:
+        keyword = keyword.strip('%')
+        for item in galaxy_cluster_uuids.values():
+            if keyword in item['value'].lower():
+                yield item
+            else:
+                if 'meta' in item and 'synonyms' in item['meta']:
+                    for synonym in item['meta']['synonyms']:
+                        if keyword in synonym.lower():
+                            yield item
 
 
 def get_galaxies_relating(uuid):

src/MISP_maltego/transforms/eventtoattributes.py

@@ -1,8 +1,7 @@
 from canari.maltego.entities import Hashtag
 from canari.maltego.transform import Transform
-# from canari.framework import EnableDebugWindow
 from MISP_maltego.transforms.common.entities import MISPEvent, MISPObject
-from MISP_maltego.transforms.common.util import check_update, get_misp_connection, attribute_to_entity, event_to_entity, galaxycluster_to_entity, object_to_entity, object_to_attributes, object_to_relations, tag_matches_note_prefix
+from MISP_maltego.transforms.common.util import check_update, MISPConnection, attribute_to_entity, event_to_entity, galaxycluster_to_entity, object_to_attributes, tag_matches_note_prefix
 from canari.maltego.message import LinkStyle
 
 
@@ -16,7 +15,7 @@ __maintainer__ = 'Christophe Vandeplas'
 __email__ = 'christophe@vandeplas.com'
 __status__ = 'Development'
 
-# FIXME have a more human readable version of the MISP event value in the graph. change entity + event_to_entity + do_transform
+# TODO have a more human readable version of the MISP event value in the graph. change entity + event_to_entity + do_transform
 
 
 class EventToTransform(Transform):
@@ -27,7 +26,7 @@ class EventToTransform(Transform):
         self.request = None
         self.response = None
         self.config = None
-        self.misp = None
+        self.conn = None
         self.event_json = None
         self.event_tags = None
 
@@ -37,9 +36,9 @@ class EventToTransform(Transform):
         self.config = config
         self.response += check_update(config)
         maltego_misp_event = request.entity
-        self.misp = get_misp_connection(config, request.parameters)
+        self.conn = MISPConnection(config, request.parameters)
         event_id = maltego_misp_event.id
-        search_result = self.misp.search(controller='events', eventid=event_id, with_attachments=False)
+        search_result = self.conn.misp.search(controller='events', eventid=event_id, with_attachments=False)
         if search_result:
             self.event_json = search_result.pop()
         else:
@@ -77,16 +76,16 @@ class EventToTransform(Transform):
 
     def gen_response_objects(self):
         for o in self.event_json['Event']['Object']:
-            self.response += object_to_entity(o)
+            self.response += self.conn.object_to_entity(o)
 
     def gen_response_relations(self):
         for e in self.event_json['Event']['RelatedEvent']:
             self.response += event_to_entity(e, link_style=LinkStyle.DashDot)
 
 
-# @EnableDebugWindow
 class EventToAll(EventToTransform):
     input_type = MISPEvent
+    display_name = 'To All'
     description = 'Expands an Event to Attributes, Objects, Tags, Galaxies'
     remote = True
 
@@ -100,22 +99,22 @@ class EventToAll(EventToTransform):
         return self.response
 
 
-# @EnableDebugWindow
 class EventToAttributes(EventToTransform):
     input_type = MISPEvent
-    description = 'Expands an Event to Attributes'
+    display_name = 'To Attributes/Objects'
     remote = True
 
     def do_transform(self, request, response, config):
         if super().do_transform(request, response, config):
             self.gen_response_attributes()
+            self.gen_response_objects()
 
         return self.response
 
 
-# @EnableDebugWindow
 class EventToTags(EventToTransform):
     input_type = MISPEvent
+    display_name = 'To Tags'
     description = 'Expands an Event to Tags and Galaxies'
     remote = True
 
@@ -127,10 +126,9 @@ class EventToTags(EventToTransform):
         return self.response
 
 
-# @EnableDebugWindow
 class EventToGalaxies(EventToTransform):
     input_type = MISPEvent
-    description = 'Expands an Event to Galaxies'
+    display_name = 'To Galaxies / ATT&CK'
     remote = True
 
     def do_transform(self, request, response, config):
@@ -140,10 +138,9 @@ class EventToGalaxies(EventToTransform):
         return self.response
 
 
-# @EnableDebugWindow
 class EventToObjects(EventToTransform):
     input_type = MISPEvent
-    description = 'Expands an Event to Objects'
+    display_name = 'To Objects'
     remote = True
 
     def do_transform(self, request, response, config):
@@ -153,10 +150,9 @@ class EventToObjects(EventToTransform):
         return self.response
 
 
-# @EnableDebugWindow
 class EventToRelations(EventToTransform):
     input_type = MISPEvent
-    description = 'Expands an Event to related Events'
+    display_name = 'To Related Events'
     remote = True
 
     def do_transform(self, request, response, config):
@@ -166,45 +162,41 @@ class EventToRelations(EventToTransform):
         return self.response
 
 
-# @EnableDebugWindow
 class ObjectToAttributes(Transform):
-    """"Expands an object to its attributes"""
     input_type = MISPObject
-    description = 'Expands an Object to Attributes'
+    display_name = 'To Attributes'
     remote = True
 
     def do_transform(self, request, response, config):
         response += check_update(config)
         maltego_object = request.entity
-        misp = get_misp_connection(config, request.parameters)
-        event_json = misp.get_event(maltego_object.event_id)
+        conn = MISPConnection(config, request.parameters)
+        event_json = conn.misp.get_event(maltego_object.event_id)
         for o in event_json['Event']['Object']:
             if o['uuid'] == maltego_object.uuid:
                 for entity in object_to_attributes(o, event_json):
                     if entity:
                         response += entity
-                for entity in object_to_relations(o, event_json):
+                for entity in conn.object_to_relations(o, event_json):
                     if entity:
                         response += entity
 
         return response
 
 
-# @EnableDebugWindow
 class ObjectToRelations(Transform):
-    """Expands an object to the relations of the object"""
     input_type = MISPObject
-    description = 'Expands an Object to Relations'
+    display_name = 'To Related Objects'
     remote = True
 
     def do_transform(self, request, response, config):
         response += check_update(config)
         maltego_object = request.entity
-        misp = get_misp_connection(config, request.parameters)
-        event_json = misp.get_event(maltego_object.event_id)
+        conn = MISPConnection(config, request.parameters)
+        event_json = conn.misp.get_event(maltego_object.event_id)
         for o in event_json['Event']['Object']:
             if o['uuid'] == maltego_object.uuid:
-                for entity in object_to_relations(o, event_json):
+                for entity in conn.object_to_relations(o, event_json):
                     if entity:
                         response += entity
 

src/MISP_maltego/transforms/galaxytoevent.py

@@ -1,7 +1,6 @@
 from canari.maltego.transform import Transform
-# from canari.framework import EnableDebugWindow
 from MISP_maltego.transforms.common.entities import MISPEvent, MISPGalaxy, ThreatActor, Software, AttackTechnique
-from MISP_maltego.transforms.common.util import check_update, get_misp_connection, galaxycluster_to_entity, get_galaxy_cluster, get_galaxies_relating, search_galaxy_cluster, mapping_galaxy_icon
+from MISP_maltego.transforms.common.util import check_update, MISPConnection, galaxycluster_to_entity, get_galaxy_cluster, get_galaxies_relating, search_galaxy_cluster, mapping_galaxy_icon
 from canari.maltego.message import UIMessageType, UIMessage, LinkDirection
 
 
@@ -16,55 +15,26 @@ __email__ = 'christophe@vandeplas.com'
 __status__ = 'Development'
 
 
-# @EnableDebugWindow
-class GalaxyToEvents(Transform):
-    """Expands a Galaxy to multiple MISP Events."""
-
-    # The transform input entity type.
-    input_type = MISPGalaxy
-    remote = True
-
-    def do_transform(self, request, response, config):
-        response += check_update(config)
-        maltego_misp_galaxy = request.entity
-        misp = get_misp_connection(config, request.parameters)
-        if maltego_misp_galaxy.tag_name:
-            tag_name = maltego_misp_galaxy.tag_name
-        else:
-            tag_name = maltego_misp_galaxy.value
-        events_json = misp.search(controller='events', tags=tag_name, with_attachments=False)
-        for e in events_json:
-            response += MISPEvent(e['Event']['id'], uuid=e['Event']['uuid'], info=e['Event']['info'], link_direction=LinkDirection.OutputToInput)
-        return response
-
-
-# @EnableDebugWindow
 class GalaxyToTransform(Transform):
     input_type = None
 
     def do_transform(self, request, response, config, type_filter=MISPGalaxy):
         response += check_update(config)
-        maltego_misp_galaxy = request.entity
 
-        current_cluster = None
-        if maltego_misp_galaxy.uuid:
-            current_cluster = get_galaxy_cluster(uuid=maltego_misp_galaxy.uuid)
-        elif maltego_misp_galaxy.tag_name:
-            current_cluster = get_galaxy_cluster(tag=maltego_misp_galaxy.tag_name)
-        elif maltego_misp_galaxy.name:
-            current_cluster = get_galaxy_cluster(tag=maltego_misp_galaxy.name)
+        current_cluster = get_galaxy_cluster(request_entity=request.entity)
 
-        if not current_cluster and maltego_misp_galaxy.name != '-':
+        # legacy - replaced by Search in MISP
+        if not current_cluster and request.entity.name != '-':
             # maybe the user is searching for a cluster based on a substring.
             # Search in the list for those that match and return galaxy entities
-            potential_clusters = search_galaxy_cluster(maltego_misp_galaxy.name)
-            # TODO check if duplicates are possible
+            potential_clusters = search_galaxy_cluster(request.entity.name)
             if potential_clusters:
                 for potential_cluster in potential_clusters:
                     new_entity = galaxycluster_to_entity(potential_cluster, link_label='Search result')
                     if isinstance(new_entity, type_filter):
                         response += new_entity
                 return response
+        # end of legacy
 
         if not current_cluster:
             response += UIMessage("Galaxy Cluster UUID not in local mapping. Please update local cache; non-public UUID are not supported yet.", type=UIMessageType.Inform)
@@ -72,7 +42,7 @@ class GalaxyToTransform(Transform):
         c = current_cluster
 
         # update existing object
-        galaxy_cluster = get_galaxy_cluster(c['uuid'])
+        galaxy_cluster = get_galaxy_cluster(uuid=c['uuid'])
         icon_url = None
         if 'icon' in galaxy_cluster:  # map the 'icon' name from the cluster to the icon filename of the intelligence-icons repository
             try:
@@ -97,7 +67,7 @@ class GalaxyToTransform(Transform):
         # find related objects
         if 'related' in current_cluster:
             for related in current_cluster['related']:
-                related_cluster = get_galaxy_cluster(related['dest-uuid'])
+                related_cluster = get_galaxy_cluster(uuid=related['dest-uuid'])
                 if related_cluster:
                     new_entity = galaxycluster_to_entity(related_cluster, link_label=related['type'])
                     if isinstance(new_entity, type_filter):
@@ -116,8 +86,8 @@ class GalaxyToTransform(Transform):
 
 
 class GalaxyToRelations(GalaxyToTransform):
-    """Expands a Galaxy to related Galaxies and Clusters"""
     input_type = MISPGalaxy
+    display_name = 'To Related Galaxies'
     remote = True
 
     def do_transform(self, request, response, config, type_filter=MISPGalaxy):
@@ -125,8 +95,8 @@ class GalaxyToRelations(GalaxyToTransform):
 
 
 class GalaxyToSoftware(GalaxyToTransform):
-    """Expands a Galaxy to related Software/Tool Galaxies"""
     input_type = MISPGalaxy
+    display_name = 'To Malware/Software/Tools'
     remote = True
 
     def do_transform(self, request, response, config, type_filter=Software):
@@ -134,8 +104,8 @@ class GalaxyToSoftware(GalaxyToTransform):
 
 
 class GalaxyToThreatActor(GalaxyToTransform):
-    """Expands a Galaxy to related ThreatActor Galaxies"""
     input_type = MISPGalaxy
+    display_name = 'To Threat Actors'
     remote = True
 
     def do_transform(self, request, response, config, type_filter=ThreatActor):
@@ -143,8 +113,8 @@ class GalaxyToThreatActor(GalaxyToTransform):
 
 
 class GalaxyToAttackTechnique(GalaxyToTransform):
-    """Expands a Galaxy to related Attack Techniques Galaxies"""
     input_type = MISPGalaxy
+    display_name = 'To Attack Techniques'
     remote = True
 
     def do_transform(self, request, response, config, type_filter=AttackTechnique):