chg: [changelog] updated

add: misp elastic docker lab blogpost

content/blog/elastic-misp-docker.md

@@ -0,0 +1,264 @@
+---
+title: MISP - Elastic Stack - Docker Lab 
+date: 2024-04-05
+layout: post
+banner: /img/blog/elastic-lab/elastic-misp-architecture.png
+tags: ["MISP", "Threat Intelligence", "Elastic", "Elasticsearch", "Filebeat", "Alerts", "Integration", "Docker"]
+categories: []
+authors:
+- Luciano Righetti
+---
+
+# MISP - Elastic Stack - Docker
+
+This lab explains how to connect MISP to the Elastic Stack in order to leverage IOCs from MISP and trigger alerts based on user defined rules.
+
+![Elastic-MISP Overview Diagram](/img/blog/elastic-lab/architecture.png "{class='img-responsive'}")
+
+
+MISP is used to gather IOCs from different sources such as open source Threat Intelligence feeds. 
+
+Filebeat pulls IOCs from MISP and pushed them to the Elasticsearch instance via the Threat Intel module [^1].
+
+To simplify this guide Elastic Agent [^2] in installed standalone mode is installed on the hosts we want to monitor, but Fleet [^3] could be used instead. 
+
+Kibana [^4] is used for exploring the IOCs, creating rules and visualizing the alerts.  
+
+Everything in this lab is run on Docker [^5].
+
+[^1]: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-threatintel.html
+[^2]: https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html
+[^3]: https://www.elastic.co/guide/en/fleet/current/fleet-overview.html
+[^4]: https://www.elastic.co/kibana
+[^5]: https://github.com/righel/elastic-misp-docker-lab
+
+## Installation
+1. Clone the lab repository.
+    ```bash
+    $ git clone https://github.com/righel/elastic-misp-docker-lab.git & cd elastic-misp-docker-lab
+    ```
+
+### MISP
+
+2. Create the .env file:
+    ```
+    $ cp template.env .env
+    ```
+
+3. Start the MISP containers.
+    ```
+    $ docker compose up -d
+    ```
+
+4. When MISP containers finish starting, create a sync user for Elastic on MISP.
+
+
+    Using MISP CLI:
+    ```
+    $ docker-compose exec misp-core app/Console/cake User create elastic@admin.test 5 1
+    $ docker-compose exec misp-core app/Console/cake User change_authkey elastic2@admin.test
+    Old authentication keys disabled and new key created: 06sDmKQK3E6MSJwsOhYT3N4NzfTpe53ruV0Bydf0
+    ```
+    
+    Using MISP UI:
+    
+    Default MISP credentials
+
+    > User: admin@admin.test
+    > 
+    > Password: admin
+    
+
+### Elastic Stack
+
+> The yaml configuration files for Elasticsearch, Kibana and Filebeat are located in `elastic/config/` directory. You can review and change these settings before deploying Elastic Stack.
+
+For adjusting the Filebeat MISP Threat Intel module, check `elastic/config/filebeat.yml`:
+```yaml
+filebeat.modules:
+  - module: threatintel
+    misp:
+      enabled: true
+      var.input: httpjson
+      var.url: "https://${MISP_HOST}/events/restSearch"
+      var.api_token: "${MISP_ELASTIC_API_KEY}"
+      var.first_interval: 24h
+      var.interval: 10m
+      var.ssl.verification_mode: none
+      var.filters:
+        type: ["md5", "sha256", "sha512", "url", "uri", "ip-src", "ip-dst", "hostname", "domain"]
+        tags: ['workflow:state="complete"']
+```
+
+For more details refer to the official docs: 
+* https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-threatintel.html#misp
+
+
+5. Modify some environment variables before firing up Elastic Stack.
+
+    * `KIBANA_ENCRYPTION_KEY`: Has to be 32 chars string to set up `xpack.encryptedSavedObjects.encryptionKey` on Kibana. 
+    * `MISP_ELASTIC_API_KEY`: The MISP API key generated in step 4 for the sync user `elastic@admin.test`.
+
+6. Start the Elastic Stack containers.
+    ```
+    $ docker-compose -f docker-compose.elastic.yml -d
+    ```
+    
+Now you can use MISP and the Elastic Stack.
+
+
+## Creating Kibana detection rules
+
+1. Go to _Kibana_ > _Security_ > _Rules_ and click on _Detection rules (SIEM)_
+
+    ![Kibana Security Rules page](/img/blog/elastic-lab/kibana_detection_rules.png "{class='img-responsive'}")
+
+2. Click on ![Kibana Add Elastic Rules](/img/blog/elastic-lab/kibana_add_rules_button.png "{class='img-responsive'}") and search for _Threat Intel_, and select the rules you are interested on.
+
+    ![Kibana Install Threat Intel rules](/img/blog/elastic-lab/kibana_add_elastic_rules.png "{class='img-responsive'}")
+
+3. Click on ![Kibana Install rules](/img/blog/elastic-lab/kibana_install_rules_button.png "{class='img-responsive'}")
+
+4. Go back to _Kibana_ > _Security_ > _Rules_ and click on _Detection rules (SIEM)_, click on the _Disabled rules_ filter to show the recently installed rules (by default they are disabled), and enable them.
+
+    ![Kibana Enabled rules](/img/blog/elastic-lab/kibana_installed_rules.png "{class='img-responsive'}")
+
+Now Elastic will generate alerts if it detects any hash, url or domain matching with MISP IOCs.
+
+## Demo
+
+### Install Elastic Agent (standalone)
+
+1. Go to _Kibana_ > _Management_ > _Fleet_ and switch to the _Agent policies_ tab and click on the _Create agent policy_ button.
+
+![Kibana Fleet Create Agent Policy](/img/blog/elastic-lab/kibana_create_agent_policy.png "{class='img-responsive'}")
+
+2. Click on the policy to access it and click on _Add integration_ and search for _Network Packet Capture_, click on it and click on _Add Network Packet Capture_
+
+    ![Add Network Packet Capture](/img/blog/elastic-lab/kibana_network_packet_capture_integration.png "{class='img-responsive'}")
+
+3. Configure the Network Packet Capture integration if needed.
+    ![Network Packet Capture configuration](/img/blog/elastic-lab/kibana_add_network_packet_capture_integration.png "{class='img-responsive'}")
+
+    Click on _Save and continue_.
+
+    > If prompted to add a Elastic Agent, click on _Add Elastic Agent later_.
+
+    ![Kibana Elastic Agent Policy](/img/blog/elastic-lab/kibana_agent_policy_view.png "{class='img-responsive'}")
+
+
+4. Click on _Actions_ > _Add agent_, switch to the _Run standalone_ tab.
+
+    ![Configure standanlone Elastic Agent](/img/blog/elastic-lab/kibana_add_elastic_agent.png "{class='img-responsive'}")
+    
+    Click on _Download Policy_.
+    
+5. Create an API key for the Elastic Agent communication to Elasticsearch [^6]. Navigate to _Kibana_ > _Stack Management_ > _API keys_ and click Create API key.
+
+
+    ![Kibana Create Elastic Agent API key](/img/blog/elastic-lab/kibana_add_api_key.png "{class='img-responsive'}")
+
+    1. Enable the _Restrict privileges_ toggle and copy the following configuration.
+    
+        ```json
+        {
+          "standalone_agent": {
+            "cluster": [
+              "monitor"
+            ],
+            "indices": [
+              {
+                "names": [
+                  "logs-*-*", "metrics-*-*", "traces-*-*", "synthetics-*-*" 
+                ],
+                "privileges": [
+                  "auto_configure", "create_doc"
+                ]
+              }
+            ]
+          }
+        }
+
+        ```
+
+    2. Click on _Create API key_.
+
+    3. Select _Beats_ and copy the API key.
+
+        ![Elastic Agent API key](/img/blog/elastic-lab/kibana_api_key_view.png "{class='img-responsive'}")
+
+    
+7. Open the `elastic-agent.yml` configuration downloaded on step 5 and modify the `outputs` section it to use API authentication and change the Elasticsearch host from `localhost` to `elasticsearch`.
+
+    ```yaml
+    outputs:
+      default:
+        type: elasticsearch
+        hosts:
+          - 'http://elasticsearch:9200'
+        api_key: 'RZAxqY4BZZJcsxX19zAw:d9AabeoqRkKwm27K_KEgKA'
+        preset: balanced
+    ``` 
+
+8. Install Elastic Agent in the host we want to monitor. Start an Ubuntu container and plug it to the MISP-Elastic Stack lab network.
+    ```
+    $ docker run -it --network=elastic-misp-docker-lab_default --name=my_monitored_host ubuntu
+    ```
+
+
+9. Follow the steps to install the standalone Elastic Agent [^7].
+    ```bash
+    $ curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.13.1-linux-x86_64.tar.gz
+    root@be44a9a86e24:/# apt update
+    ...
+    root@be44a9a86e24:/# apt install curl -y
+    ...
+    root@be44a9a86e24:/# cd /tmp
+    root@be44a9a86e24:/# curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.13.1-linux-x86_64.tar.gz
+    root@be44a9a86e24:/# tar xzvf elastic-agent-8.13.1-linux-x86_64.tar.gz
+    ...
+    
+    # from a different terminal, copy the elastic-agent.yml config file into the container
+    docker cp elastic-agent.yml my_monitored_host:/tmp/elastic-agent-8.13.1-linux-x86_64/elastic-agent.yml
+    
+    # back on the docker terminal, install the agent
+    root@be44a9a86e24:/# cd elastic-agent-8.13.1-linux-x86_64/
+    root@be44a9a86e24:/# ./elastic-agent install
+        Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y
+        Do you want to enroll this Agent into Fleet? [Y/n]:n
+        [=   ] Service Started  [32s] Elastic Agent successfully installed, starting enrollment.
+        [=   ] Done  [32s]                       
+        Elastic Agent has been successfully installed.
+    ```
+    
+    Done. Now the docker container is being monitored by the Elastic Agent and the metrics are being pushed to Elasticsearch.
+    
+### Testing
+
+1. Go to MISP, create a new Event an add a _ip-dst_ type attribute.
+    2. ![Add MISP Attribute](/img/blog/elastic-lab/misp_add_attribute.png "{class='img-responsive'}")
+
+2. Add the _workflow:state="complete"_ tag to the event to it is picked up by Elasticsearch.
+
+    ![MISP Add Tag](/img/blog/elastic-lab/misp_add_workflow_tag.png "{class='img-responsive'}")
+
+3. Publish the Event.
+
+> Depending on the `vars.interval` set on the Threat Intel module in the `filebeat.yml` configuration file, it may take some time for the IOC to get into Elasticsearch.
+
+4. From your monitored docker host, generate traffic to one of the MISP IOCs.
+    ```
+    root@be44a9a86e24:/# curl -I https://circl.lu
+    ...
+    ```
+
+5. Navigate to _Kibana_ > _Security_ > _Alerts_, you should now see some alerts triggered by the _Threat Intel IP Address Indicator Match_.
+    ![Kibana Security Alerts](/img/blog/elastic-lab/kibana_security_alerts.png "{class='img-responsive'}")
+
+> By default Elastic runs these detection rules every 4 hours, you can configure the internal reduce 
+
+
+Done! Now you will get alerts on Kibana when an event in one of your monitored hosts matches a MISP IOC. 
+
+[^6]: https://www.elastic.co/guide/en/fleet/current/grant-access-to-elasticsearch.html
+[^7]: https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html

static/Changelog-PyMISP.txt

@@ -2,6 +2,15 @@ Changelog
 =========
 
 
+%%version%% (unreleased)
+------------------------
+
+Changes
+~~~~~~~
+- [data] describeTypes file updated. [Alexandre Dulaunoy]
+- Bump deps. [Raphaël Vinot]
+
+
 v2.4.188 (2024-03-22)
 ---------------------
 

static/Changelog-misp-galaxy.txt

@@ -1,10 +1,30 @@
 # Changelog
 
 
-## %%version%% (unreleased)
+## v2.4.189 (2024-04-05)
 
 ### Other
 
+* Merge pull request #957 from gregWDumont/NACE_CODE_valid. [Alexandre Dulaunoy]
+
+  [Sector] Add NACE_CODE to cluster
+
+* [Sector] Add NACE_CODE to cluster. [gregWDumont]
+
+* Merge pull request #953 from Mathieu4141/threat-actors/d9ef3240-7f58-453c-926b-7757caf17f1a. [Alexandre Dulaunoy]
+
+  [threat actors] 4 new actors and 1 alias
+
+* [threat-actors] Add UNC5174. [Mathieu4141]
+
+* [threat-actors] Add Saad Tycoon. [Mathieu4141]
+
+* [threat-actors] Add Edalat-e Ali. [Mathieu4141]
+
+* [threat-actors] Add SilitNetwork. [Mathieu4141]
+
+* [threat-actors] Add Lazarus Group aliases. [Mathieu4141]
+
 * Merge pull request #952 from Delta-Sierra/main. [Alexandre Dulaunoy]
 
   add Germany as target for several Threat actors

static/Changelog-misp-objects.txt

@@ -1,6 +1,57 @@
 # Changelog
 
 
+## v2.4.189 (2024-04-05)
+
+### New
+
+* [cert-pl-phishing] first draft of a template for the CERT.PL phishing system. [Alexandre Dulaunoy]
+
+### Changes
+
+* [cert-pl-phishing] fixed. [Alexandre Dulaunoy]
+
+* [cert-pl-phishing] fixed. [Alexandre Dulaunoy]
+
+* [schema] add the new 'integer' attribute type in the JSON schema. [Alexandre Dulaunoy]
+
+* [pe] Using the new `integer` attribute type. [Christian Studer]
+
+* [pe] Added `characteristics` & `machine-type` enumerations. [Christian Studer]
+
+  - Characteristics are usually in a list, so we
+    have now both the list of characteristics with
+    their name, and the hex value of the addition
+    of all the characteristics numeric values
+  - We represent the machine type with its name
+
+* [ddos] object type alone authorized if the source/target cannot be disclosed. [Alexandre Dulaunoy]
+
+### Fix
+
+* [pe] Removing the `disable_correlation` flag for a `size-in-bytes` attribute type. [Christian Studer]
+
+* [pe] typo fixed. [Alexandre Dulaunoy]
+
+* [pe] Sizes in the PE format should be in bytes. [Christian Studer]
+
+* [pe] `counter` makes more sense here. [Christian Studer]
+
+### Other
+
+* Add: [readme] Added `pe-optional-header` description. [Christian Studer]
+
+* Add: [pe-optional-header] New object template for PE optional headers. [Christian Studer]
+
+* Merge branch 'main' of github.com:MISP/misp-objects. [Christian Studer]
+
+* Merge branch 'main' of github.com:MISP/misp-objects. [Christian Studer]
+
+* Add: [pe] Added some PE fields as available with `lief` API. [Christian Studer]
+
+* Merge branch 'main' of github.com:MISP/misp-objects. [Christian Studer]
+
+
 ## v2.4.188 (2024-03-20)
 
 ### Changes

static/Changelog-misp-taxonomies.txt

@@ -1,14 +1,20 @@
 # Changelog
 
 
-## %%version%% (unreleased)
+## v2.4.189 (2024-04-05)
 
 ### Changes
 
+* [priority-taxonomy] update reference url. [Jeroen Pinoy]
+
 * [malware_classification] add `Stalkerware` from #275. [Alexandre Dulaunoy]
 
 ### Other
 
+* Merge pull request #278 from Wachizungu/update-priority-taxonomy-reference-url. [Alexandre Dulaunoy]
+
+  chg: [priority-taxonomy] update reference url
+
 * Merge pull request #277 from cudeso/main. [Alexandre Dulaunoy]
 
   Add review-relevance and review-completeness to workflow taxonomy

static/Changelog-misp-warninglists.txt

@@ -1,6 +1,17 @@
 # Changelog
 
 
+## v2.4.189 (2024-04-05)
+
+### New
+
+* [warning-lists] add the check-host.net source of IP addresses. [Alexandre Dulaunoy]
+
+### Changes
+
+* [warning-lists] updated. [Alexandre Dulaunoy]
+
+
 ## v2.4.188 (2024-03-20)
 
 ### Changes

static/Changelog.txt

@@ -2,6 +2,169 @@ Changelog
 =========
 
 
+v2.4.189 (2024-04-05)
+---------------------
+
+New
+~~~
+- [sighting sync] blocklisting added. [iglocska]
+
+  - block organisations' sightings from being created / pulled
+  - Added a new option to the restsearch of sightings too which this feature uses if available
+    - if it isn't, the system will block the insertion on the beforeValidate() level
+
+  - Outcome of the JTAN hackathon on 04.04.2024 in Luxembourg
+- [attribute] new attribute type added `integer` [Alexandre Dulaunoy]
+
+  Initially, we utilised a counter type across numerous objects.
+
+  However, the semantic significance of this type became unclear when establishing relationships with integers in various objects.
+- [analyst-data] Added Inbound Relationship to all views. [Sami
+  Mokaddem]
+
+Changes
+~~~~~~~
+- [version] bump. [iglocska]
+- [GeoOpen] updated to the latest version. [Alexandre Dulaunoy]
+- [misp-objects] updated to the latest version. [Alexandre Dulaunoy]
+- [taxonomies] updated. [Alexandre Dulaunoy]
+- [warninglists] updated. [Alexandre Dulaunoy]
+- [misp-galaxy] updated. [Alexandre Dulaunoy]
+- [schema] updated. [iglocska]
+- [ACL] entries added. [iglocska]
+- [setting] added for the sighting blocklisting. [iglocska]
+- [sighting restsearch] added org negations. [iglocska]
+
+  - the org_id filter now allows for the use of a prepended '!' character for negations
+- [test] Check if MISP and STIX2 are valid in build-test.sh. [Jakub
+  Onderka]
+- [internal] Log exception when importing stix file. [Jakub Onderka]
+- [internal] Update misp-stix. [Jakub Onderka]
+- [PyMISP] updated. [Alexandre Dulaunoy]
+- [internal] Speedup sighting rest search. [Jakub Onderka]
+- [UI] event view now only load analyst data for objects/attributes
+  actually shown via pagination. [iglocska]
+- [curl client] added option for timeout. [iglocska]
+- [internal] Remove possible empty lines from output. [Jakub Onderka]
+- [docs:new-background-workers] add rhel specific steps to migration
+  guide. [Jeroen Pinoy]
+- [test] Check attribute search. [Jakub Onderka]
+- [internal] Better error handling when fetching sightings. [Jakub
+  Onderka]
+
+Fix
+~~~
+- [junk] removed. [iglocska]
+- [internal] Try to fix STIX import. [Jakub Onderka]
+- [sync] Avoid problem with duplicate sightings UUID. [Jakub Onderka]
+- [analyst-data:attachData] Make sure to also load child notes and
+  opinions. [Sami Mokaddem]
+
+  Changed the old behavior: Before we were loading 3 children. Now, we only load 1 by default.
+- [analyst-data:UI] Added missing entries for view elements. [Sami
+  Mokaddem]
+- [analystdata] added to events as the previous commits purged it.
+  [iglocska]
+- [analyst data chunk size] increased. [iglocska]
+- [internal] Attribute.php code style fix. [Jakub Onderka]
+- [sync] Drop support for zstd from CurlClient. [Jakub Onderka]
+- [oidc] Use the same handling of org also for Oidc::isUserValid. [Jakub
+  Onderka]
+- [search] Attribute search error 500 because of force index search.
+  [Jakub Onderka]
+- [UI] Showing event logo in correlation graph. [Jakub Onderka]
+- [internal] Check if values is not empty for MysqlExtended. [Jakub
+  Onderka]
+- [internal] Undefined index in error message during sync. [Jakub
+  Onderka]
+- [doc:rhel-installer] Correct conditional addition of httpd Listen 443
+  line. [Jeroen Pinoy]
+- [API] Cleanup compression marks added by Apache from Etag. [Jakub
+  Onderka]
+
+Other
+~~~~~
+- Merge branch 'develop' into 2.4. [iglocska]
+- Merge branch '2.4' into develop. [iglocska]
+- Merge pull request #9553 from jloehel/fix-9552. [Andras Iklody]
+
+  fix [INSTALL/MySQL]: Create table `user_login_profiles` only if it not exists
+- Fix [INSTALL/MySQL]: Create table `user_login_profiles` only if it not
+  exists. [Jürgen Löhel]
+
+  fixes: #9552
+- Merge pull request #9662 from JakubOnderka/build-test-json-valid.
+  [Jakub Onderka]
+
+  chg: [test] Check if MISP and STIX2 are valid in build-test.sh
+- Merge branch 'develop' of github.com:MISP/MISP into develop. [Sami
+  Mokaddem]
+- Merge pull request #9658 from JakubOnderka/stix-exception-logging.
+  [Jakub Onderka]
+
+  chg: [internal] Log exception when importing stix file
+- Merge pull request #9660 from JakubOnderka/duplicate-sighting-uuid.
+  [Jakub Onderka]
+
+  fix: [sync] Avoid problem with duplicate sightings UUID
+- Merge pull request #9661 from JakubOnderka/misp-stix-update. [Jakub
+  Onderka]
+
+  chg: [internal] Update misp-stix
+- Merge branch 'develop' of github.com:MISP/MISP into develop. [Sami
+  Mokaddem]
+- Merge branch 'develop' of github.com:MISP/MISP into develop.
+  [iglocska]
+- Merge branch 'develop' of github.com:MISP/MISP into develop. [Sami
+  Mokaddem]
+- Merge pull request #8760 from JakubOnderka/sightings-conditions-
+  simplify. [Jakub Onderka]
+
+  chg: [internal] Speedup sighting rest search
+- Merge branch 'develop' of github.com:MISP/MISP into develop.
+  [Alexandre Dulaunoy]
+- Merge pull request #9657 from JakubOnderka/remove-php-ends. [Jakub
+  Onderka]
+
+  chg: [internal] Remove possible empty lines from output
+- Merge pull request #9652 from JakubOnderka/curl-zstd-drop. [Jakub
+  Onderka]
+
+  fix: [sync] Drop support for zstd from CurlClient
+- Merge pull request #9649 from JakubOnderka/oidc-is-user-valid-fix.
+  [Jakub Onderka]
+
+  fix: [oidc] Use the same handling of org also for Oidc::isUserValid
+- Merge pull request #9641 from Wachizungu/chg-background-jobs-
+  migration-guide-add-rhel. [Alexandre Dulaunoy]
+
+  chg: [docs:new-background-workers] add rhel specific steps to migrati…
+- Merge pull request #9642 from JakubOnderka/attibute-search-500. [Jakub
+  Onderka]
+
+  chg: [test] Check attribute search
+- Merge pull request #9640 from JakubOnderka/event-log-correlation-
+  graph. [Jakub Onderka]
+
+  fix: [UI] Showing event logo in correlation graph
+- Merge pull request #9637 from JakubOnderka/undefined-index-fixes.
+  [Jakub Onderka]
+
+  Undefined index fixes
+- Merge pull request #9636 from Wachizungu/fix-rhel-httpd-listen-config.
+  [Alexandre Dulaunoy]
+
+  fix: [doc:rhel-installer] Correct conditional addition of httpd Liste…
+- Merge pull request #9635 from JakubOnderka/error-handling-sighting.
+  [Jakub Onderka]
+
+  chg: [internal] Better error handling when fetching sightings
+- Merge pull request #9634 from JakubOnderka/response-etag. [Jakub
+  Onderka]
+
+  fix: [API] Cleanup compression marks added by Apache from Etag
+
+
 v2.4.188 (2024-03-22)
 ---------------------