Merge pull request #133 from SteveClement/master

chg: [doc] Added requirements, faq and more content to the appendix

SUMMARY.md

@@ -2,6 +2,7 @@
 
 * [Book Convention](book-convention/README.md)
 * [Quick Start](quick-start/README.md)
+* [Requirements](requirements/README.md)
 * [Get Your Instance](get-your-instance/README.md)
 * [General Layout](general-layout/README.md)
 * [General Concepts](general-concepts/README.md)
@@ -24,4 +25,5 @@
 * [Synchronisation/Sharing](sharing/README.md)
 * [ZeroMQ - MISP publish-subscribe](misp-zmq/README.md)
 * [Translations - i18n & l10n](translation/README.md)
+* [FAQ](faq/README.md)
 * [Appendices](appendices/README.md)

appendices/README.md

@@ -1,3 +1,7 @@
+# Summary
+
+<!-- toc -->
+
 # Appendix A: External Authentication
 
 #### The external authentication mechanism described
@@ -220,3 +224,85 @@ https://<misp url>/servers/queryACL/findMissingFunctionNames
 
 Functions that have not been tied into the new ACL yet show up here. These functions will (until added to the ACL) only be accessible to site admins.
 
+# Appendix C: Official MISP developments
+
+This section lists the projects that can be found on the main [MISP GitHub](https://github.com/MISP/repositories) page
+ e know of but not officially support and rely on their respective maintainers to keep up to date to the MISP 2.4 developments.
+
+
+| Project | Description | Status |
+| -- | -- | -- |
+| [misp-objects](https://github.com/MISP/misp-objects) | Definition, description and relationship types of MISP objects | Core to MISP, frequently updated and tested |
+
+<!--
+| []() |  | Core to MISP, frequently updated and tested |
+| []() |  | Core to MISP, frequently updated and tested |
+| []() |  | Core to MISP, frequently updated and tested |
+-->
+
+# Appendix D: Third-party development
+
+This section lists some projects we know of but not officially support and rely on their respective maintainers to keep up to date to the MISP 2.4 developments.
+
+| Project | Description | Status |
+| -- | -- | -- |
+| [MISP-STIX-ESM](https://github.com/mohlcyber/MISP-STIX-ESM) | Exports MISP events to STIX and ingest into McAfee ESM | Not tested by MISP core team |
+| [Docker MISP](https://github.com/harvard-itsecurity/docker-misp) | Automated Docker MISP container | Not tested by MISP core team |
+| [misp42splunk](https://github.com/remg427/misp42splunk) | A Splunk app to use MISP in background and combine with TheHive | Not tested by MISP core team |
+| [getmispioc](https://github.com/xme/splunk/tree/master/getmispioc) | getiocmisp is a Splunk custom search command that helps to extract IOCs from a MISP instance. | Not tested by MISP core team |
+| [OTX MISP](https://github.com/gcrahay/otx_misp) | Imports Alienvault OTX pulses to a MISP instance | Not tested by MISP core team |
+| [BTG](https://github.com/conix-security/BTG) | BTG's purpose is to make fast and efficient search on IOC | Not tested by MISP core team |
+| [MISP OSINT Collection](https://github.com/adulau/misp-osint-collection) | Collection of best practices to add OSINT into MISP and/or MISP communities | Not tested by MISP core team |
+| [Ansible MISP](https://github.com/StamusNetworks/ansible-misp) | Ansible playbook to install Malware Information Sharing Platform (MISP) | Not tested by MISP core team |
+| [IBM XFE module](https://github.com/johestephan/XFE) | Various IBM X-Force Exchange modules | Not tested by MISP core team |
+| [MISP dockerized](https://github.com/DCSO/MISP-dockerized-misp-modules) | MISP dockerized is a project designed to provide an easy-to-use and easy-to-install'out of the box' MISP instance that includes everything you need to run MISP with minimal host-side requirements. | Not tested by MISP core team |
+| [MISP dockerized modules](https://github.com/DCSO/MISP-dockerized-misp-modules) | MISP-modules for MISP dockerized | Not tested by MISP core team |
+| [FireMISP](https://github.com/deralexxx/FireMISP) | FireEye Alert json files to MISP Malware information sharing plattform (Alpha) | Not tested by MISP core team |
+| [MISP Chrome Plugin](https://github.com/deralexxx/misp-chrome-plugin) | MISP Chrome plugin for adding and looking up indicators | Not tested by MISP core team |
+| [PySight2MISP](https://github.com/deralexxx/PySight2MISP) | PySight2MISP is a project that can be run to be used as glue between iSight intel API and MISP API | Not tested by MISP core team |
+| [tie2misp](https://github.com/DCSO/tie2misp) | Import DCSO TIE IOCs as MISP events | Not tested by MISP core team |
+| [security onion MISP](https://github.com/weslambert/securityonion-misp) | Grab NIDS rules and Bro Intel generated from a MISP instance and use them in Security Onion | Not tested by MISP core team |
+| [virustream](https://github.com/ntddk/virustream) | A script to track malware IOCs with OSINT on Twitter. | Not tested by MISP core team |
+| [LAC CSV Import](https://github.com/LAC-Japan/MISP-CSVImport) | Register MISP events based on information described in files such as CSV and TSV. | Not tested by MISP core team |
+| [The Hive](https://github.com/TheHive-Project/TheHive) | TheHive: a Scalable, Open Source and Free Security Incident Response Platform | Strong links between core team members, tested and known working |
+| [puppet-misp](https://github.com/voxpupuli/puppet-misp) | This module installs and configures MISP - [puppet forge site](https://forge.puppet.com/puppet/misp) | Not tested by MISP core team |
+| [ansible MISP](https://github.com/juju4/ansible-MISP) | ansible role to setup MISP | Not tested by MISP core team |
+| [OpenDXL ATD MISP](https://github.com/mohlcyber/OpenDXL-ATD-MISP) | Automated threat intelligence collection with McAfee ATD, OpenDXL and MISP | Not tested by MISP core team |
+| [IMAP Proxy](https://github.com/CIRCL/IMAP-Proxy) | Modular IMAP proxy (including PyCIRCLeanMail and MISP forward modules) | Not tested by MISP core team |
+| [AutoMISP](https://github.com/da667/AutoMISP) | automate your MISP installs - This shell script is designed to automatically install [MISP](https://github.com/MISP/MISP) and the [misp-modules](https://github.com/MISP/misp-modules) extension on either Ubuntu 16.04, or 18.04. | Not tested by MISP core team |
+| [Palo Alto Networks report_to_misp](https://github.com/PaloAltoNetworks/report_to_misp) | Parse a report and import the events into MISP | Not tested by MISP core team |
+| [Palo Alto Networks minemeld-misp](https://github.com/PaloAltoNetworks/minemeld-misp) | MineMeld nodes for MISP | Not tested by MISP core team |
+| [golang-misp](https://github.com/0xrawsec/golang-misp) | Golang Library to interact with your MISP instance | Not tested by MISP core team |
+| [go-misp](https://github.com/Zenithar/go-misp) | Golang MISP [API Client](http://zenithar.org/go/misp) | Not tested by MISP core team |
+| [MISP MAR](https://github.com/mohlcyber/MISP-MAR) | Integration between MISP platform and McAfee Active Response | Not tested by MISP core team |
+| [MISP IoC Validator](https://github.com/tom8941/MISP-IOC-Validator) | Validate IOC from MISP ; Export results and iocs to SIEM and sensors using syslog and CEF format | Not tested by MISP core team |
+| [vt2misp](https://github.com/eCrimeLabs/vt2misp) | Script to fetch data from virustotal and add it to a specific event as an object | Not tested by MISP core team |
+| [Threat Pinch Lookup](https://github.com/cloudtracer/ThreatPinchLookup) | Documentation and Sharing Repository for ThreatPinch Lookup Chrome & Firefox [Extension](https://chrome.google.com/webstore/detail/threatpinch-lookup/ljdgplocfnmnofbhpkjclbefmjoikgke) | Not tested by MISP core team |
+| [dovehawk](https://github.com/tylabs/dovehawk) | Dovehawk is a Bro module that automatically imports MISP indicators and reports Sightings | Not tested by MISP core team |
+| [yara-exporter](https://github.com/CERT-Bund/yara-exporter) | Exporting MISP event attributes to yara rules usable with Thor apt scanner | Not tested by MISP core team |
+| [volatility-misp](https://github.com/CIRCL/volatility-misp) | Volatility plugin to interface with MISP | Not tested by MISP core team |
+| [misp2bro](https://github.com/thnyheim/misp2bro) | Python script that gets IOC from MISP and converts it into BRO intel files. | Not tested by MISP core team |
+| [TA-misp](https://github.com/stricaud/TA-misp) | Splunk integration with MISP | Not tested by MISP core team |
+| [MISP QRadar](https://github.com/karthikkbala/MISP-QRadar-Integration) | The Project can used to integrate QRadar with MISP Threat Sharing Platform | Not tested by MISP core team |
+| [pymisp-suricata_search](https://github.com/raw-data/pymisp-suricata_search) | Multi-threaded suricata search module for MISP | Not tested by MISP core team |
+| [MISP-ThreatExchange](https://github.com/EC-DIGIT-CSIRC/MISP-ThreatExchange) | Script to interface MISP with Facebook ThreatExchange | Not tested by MISP core team |
+| [aptc](https://github.com/jymcheong/aptc) | [Automated Payload Test Controller](https://jymcheong.github.io/aptc/) | Not tested by MISP core team |
+| [aptmap](https://github.com/3c7/aptmap) | A [map](https://aptmap.netlify.com) displaying threat actors from the [misp-galaxy](https://github.com/MISP/misp-galaxy) | Not tested by MISP core team |
+| [mispy](https://github.com/nbareil/mispy) | Another MISP module for Python | Not tested by MISP core team |
+| [MispSharp](https://github.com/DBHeise/MispSharp) | C# Library for MISP | Not tested by MISP core team |
+| [Privacy Aware Sharing of IoCs in MISP](https://github.com/charly077/MISP-privacy-aware-sharing-master-thesis) | [Master Thesis](https://github.com/charly077/MISP-privacy-aware-sharing-master-thesis/blob/master/report/report.pdf) including MISP data. | Master thesis |
+
+<!--
+| []() |  | Not tested by MISP core team |
+| []() |  | Not tested by MISP core team |
+| []() |  | Not tested by MISP core team |
+-->
+
+# Appendix E: Other Threat Intel Ressources
+
+A brief list of online ressources that around #ThreatIntel
+
+* [Curated list of awesome cybersecurity companies and solutions.](https://github.com/Annsec/awesome-cybersecurity/blob/master/README.md) (Updated April 2017)
+* [A curated list of awesome malware analysis tools and resources](https://github.com/rshipp/awesome-malware-analysis/blob/master/README.md). Inspired by [awesome-python](https://github.com/vinta/awesome-python) and [awesome-php](https://github.com/ziadoz/awesome-php).
+* [An authoritative list of awesome devsecops tools with the help from community experiments and contributions](https://github.com/devsecops/awesome-devsecops/blob/master/README.md).[DEV.SEC.OPS](http://devsecops.org)
+* [Advance Python IoC extractor](https://github.com/InQuest/python-iocextract)

faq/README.md

@@ -0,0 +1,95 @@
+<!-- toc -->
+
+# Frequently Asked Questions
+
+The following page hosts most frequently asked questions as seen on our [issues](https://github.com/MISP/issues) and [gitter](https://gitter.im/MISP/MISP).
+
+## Permission issues
+
+If you have any permission issues, please [set the permissions](https://misp.github.io/MISP/INSTALL.ubuntu1804/#5-set-the-permissions) to something sane first.
+
+## When to update MISP?
+
+One question might be how often to update MISP.
+You can update MISP as ofte as you like. If you see the follwing:
+
+![MISP Update](./figures/misp-diag-update.png)
+
+This means that the main repository has an update available.
+
+If you want to play it safer or want to integrate it in your Weekly/Bi-Monthly update routine you can track our [Changelog](https://www.misp-project.org/Changelog.txt) a more up to date version is available [here](https://misp.github.io/MISP/Changelog/)
+
+## Update MISP fails
+
+If your MISP instance is outdated, meaning ONLY the core, not the modules or dashboard or python modules, you well see the following.
+
+![MISP outdated](./figures/misp-outdated.png)
+
+Once you click on update MISP you will be asked confirmation.
+
+![MISP Update Yes/No](./figures/update-misp-YN.png)
+
+If you are not on a branch, the UI will tell you this, the update will fail.
+
+![not on branch](./figures/misp-not-on-branch.png)
+
+If you cannot write the **.git** files and directory as the user running the web server (and thus PHP), the update will fail.
+The following diagnostic check will let you know if you can update or not.
+
+![.git not writeable](./figures/misp-diag-not-writeable-files-git.png)
+
+In case you get a file not found on **.git/ORIG_HEAD**, this means that you have never updated your MISP OR you have installed git from an archive file (like .zip/.tar.gz or similar)
+Try to click update MISP and see what happens.
+
+![ORIG_HEAD file not found](./figures/misp-diag-writeable-files-not_found-git.png)
+
+### What can go wrong if I update MISP?
+
+In theory nothing. We put great effort into protecting the integrity of the data stored in your MISP instance.
+DB upgrades happen upon login or on reload once you have update the repository.
+You cannot "break" anything by clicking **Update MISP** worse case it will complain about something and you will certainly find the answer on this page.
+
+IF not, please open an [issue](https://github.com/MISP/MISP/issues) on GitHub or come to our [gitter](https://gitter.im/MISP/MISP) chat to see if the community can help.
+
+### error: pathspec 'app/composer.json' did not match any file(s) known to git
+
+This is **not** an error and can be ignore. Nothing will be impacted by this.
+
+![pathspec](./figures/misp-pathspec.png)
+
+### MISP modules "Connection refused"
+
+![MISP Modules ](./figures/misp-module-system-diag.png)
+
+If you get have a **Connection refused state** on your modules one of the following might be true.
+
+- You have no [misp-modules](https://github.com/MISP/misp-modules) not installed
+- They are instaled but not running
+- Something completly different
+
+If they are not installed, check out this section of the [INSTALL guide](https://github.com/MISP/misp-modules/#how-to-install-and-start-misp-modules-in-a-python-virtualenv) of [misp-modules](https://github.com/MISP/misp-modules).
+
+In case they are not running, try this on the console:
+
+```
+sudo -u www-data /var/www/MISP/venv/bin/misp-modules -l 127.0.0.1 -s &
+```
+
+OR if you were foolish enough to not install in a Python virtualenv:
+
+```
+sudo -u www-data misp-modules -l 127.0.0.1 -s &
+```
+
+:warning: Running misp-modules like this will certainly kill it once you quit the session. Make sure it is in your **/etc/rc.local** or some ther init script that gets run on boot.
+
+## Uninstalling MISP
+
+There is no official procedure to uninstalling a MISP instance.
+
+If you want to re-use a machine where MISP was installed, wipe the machine and do a fresh install.
+Consider the data in your MISP instance as potentially confidential and if you synchronized with other instances, be respectful and wipe it clean.
+
+  <!-- 
+  Comment Place Holder
+  -->

publish.sh

@@ -3,8 +3,8 @@ gitbook build
 gitbook pdf
 gitbook epub
 gitbook mobi
-cp book.pdf _book
-cp book.epub _book
-cp book.mobi _book
+mv book.pdf _book
+mv book.epub _book
+mv book.mobi _book
 cd _book
-rsync -av . circl@cpab.circl.lu:/var/www/nwww.circl.lu/doc/misp/
+rsync -azv . circl@cpab:/var/www/nwww.circl.lu/doc/misp/ && rm -rf _book

requirements/README.md

@@ -0,0 +1,28 @@
+# MISP Instance requirements
+
+<!-- toc -->
+
+## Intro
+
+There are various ways you can run a MISP instance.
+
+- Virtualized with docker/ansible/packer etc
+- VMware/Virtualbox/Xen etc
+- Dedicated hardware
+- Road warrior setups
+- Air-gapped setups
+
+Whilst there is never an ultimate answer to what specifications a system needs, we try to give an approximate answer depending on your use case.
+
+## The biggie
+
+Having millions of events with millions of attributes (indicators) will eventually result in sub-par performance.
+Ideally you have millions of attributes and thousands of events. But this also depends on how you ingest the data.
+With millions of attributes a bottleneck could be the correlation engine.
+Especially if you have many duplicates in your events. (Use the feed matrix to see if feeds are massively overlapping)
+
+### Tool assisted sizing
+
+During a hackathon [misp-sizer](https://www.misp-project.org/MISP-sizer/) was conceived. ([code](https://github.com/MISP/MISP-sizer))
+This can give you a very rough estimate and needs some more [improvements](https://github.com/MISP/MISP-sizer/issues).
+