MISP
/
MISP
mirror of https://github.com/MISP/MISP
2
2
Fork 0
Code Issues Releases Wiki Activity

Merge branch 'develop' of github.com:MISP/MISP into develop

pull/7358/head
mokaddem 2021-04-20 15:30:47 +02:00
parent 514b09b4d6 c25dd7082a
commit df9fa87503
No known key found for this signature in database
GPG Key ID: 164C473F627A06FA
22 changed files with 588 additions and 90 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.gitmodules vendored
View File

@ -1,3 +1,7 @@
[submodule "app/Lib/cakephp"]
path = app/Lib/cakephp
url = https://github.com/cakephp/cakephp.git
branch = 2.x
[submodule "PyMISP"]
path = PyMISP
url = https://github.com/MISP/PyMISP.git
@ -7,6 +11,10 @@
[submodule "app/files/warninglists"]
path = app/files/warninglists
url = https://github.com/MISP/misp-warninglists.git
[submodule "app/Lib/random_compat"]
path = app/Lib/random_compat
url = https://github.com/paragonie/random_compat
branch = master
[submodule "app/files/misp-galaxy"]
path = app/files/misp-galaxy
url = https://github.com/MISP/misp-galaxy

54
CODINGSTYLE.md Normal file
View File

@ -0,0 +1,54 @@
# Coding guidelines for MISP developers
Maintaining proper coding style is very important for any large software project, such as MISP. Here’s why:
- It eases maintenance tasks, such as adding new functionality or generalizing code later
- It allows others (as well as the future you!) to easily understand fragments of code and what they were supposed to do, and thus makes it easier to later extend them with newer functionality or bug fixes
- It allows others to easily review the code and catch bugs
- It provides for an aesthetically pleasing experience when one reads the code
## General typographic conventions
- Maintain a maximum line length of 80 characters. Even though today’s monitors often are very wide and it’s often not a problem to have 120 characters displayed in an editor, maintaining shorter line lengths improves readability. It also allows others to have two parallel windows open, side by side, each with different parts of the source code.
- Naming conventions:
- `ClassName`,
- `someVariable`, `someFunction`, `someArgument`
- Maintain a decent amount of horizontal spacing, e.g. add a space after `if` or before `{` in PHP, Python, JavaScript, and similar in other languages. Whether and where to also use spaces within expressions, such as `(y*4+8)` vs. `(y * 4 + 8)` is left to the developer’s judgment. Do not put spaces immediately after or before the brackets in expressions, so avoid constructs like this: `if ( condition )` and use ones like this: `if (condition)` instead.
- Use descriptive names for variables and functions. At a time when most editors have auto-completion features, there is no excuse for using short variable names.
- Comments should be indented together with the code, e.g. like this:
```
class HttpClientJsonException extends Exception
{
/** @var HttpSocketResponse */
private $response;
}
```
## File naming conventions
- Never use spaces within file names
- **PHP:** Write file names in title case ,e.g. `AttachmentTool.php`
- **Python:** Write file names with small letters, use a dash to separate words, rather than underscores, e.g. `load_warninglists.py`
- **JavaScript:** Write file names with small letters, use dashes to separate words, rather than underscores, e.g. `bootstrap-colorpicker.js`
## General programming style guidelines
- Always prefer readability over trickiness! No need to use tricks to save a few lines of code.
- Make sure your code compiles and builds without warnings
- Always think first about interfaces (e.g. function arguments, or class methods) and data structures before you start writing the actual code.
- Use comments to explain non-trivial code fragments, or expected behavior of more complex functions, if it is not clear from their name.
- Do not use comments for code fragments where it is immediately clear what the code does. E.g. avoid constructs like this:
function ret(tp, style, cont) {
type = tp; content = cont;
// Return style
return style;
}
- In production code, there should be little to no commented or disabled code fragments. Do not use comments to disable code fragments, unless you need to. But generally, there is little excuse to keep old, unused code fragments in the code. Instead, use the functionality provided by the source code management system, such as git. For example, create a special branch for storing the old, unused code – this way you will always be able to merge this code into upstream in the future.
- Try not to hardcode values in the code.
## Commit message guidelines
Please attempt to follow our [commit messages best practices](https://github.com/MISP/MISP/wiki/CommitMessageBestPractices) when writing your git commit messages.
Also, when committing changes that will resolve an issue once merged, include #NNNN in the commit message, NNNN being the issue number.
Then, GitHub will automatically reference this commit on the corresponding issue, once the branch is pushed to our Git repository.
For example:
```
chg: [doc] Fix spelling errors (#3120)
```

161
CONTRIBUTING.md
View File

@ -1,66 +1,171 @@
# Contributing to MISP Project
MISP project is a large free software project composed of multiple sub-projects which are contributed by different contributors who are generally active users of the MISP project. MISP project fully supports the [Contributor Covenant Code of Conduct](https://github.com/MISP/MISP/blob/2.4/code_of_conduct.md) to foster an open and dynamic environment for contributing and the exchange in the threat intelligence and information exchange field.
The MISP roadmap is mostly based on the user communities (e.g. private communities, CSIRTs communities, security researchers, ISACs - Information Sharing and Analysis Center, security providers, governmental or military organisations) relying on MISP to perform their duties of information sharing and analysis. If you see an existing issue which covers your needs, don't hesitate to comment on it.
The [MISP roadmap](/roadmap.md) is mostly based on the user communities (e.g. private communities, CSIRTs communities, security researchers, ISACs - Information Sharing and Analysis Center, security providers, governmental or military organisations) relying on MISP to perform their duties of information sharing and analysis.
Participating in the MISP project is easy and everyone can contribute following their own ability:
Participating in the MISP project is easy and everyone can contribute following their ability.
Get familiar with [how we use GitHub at MISP Project](/GITWORKFLOW.md), then read on for details on some ways you can contribute:
## Reporting a bug or an issue, suggesting features
## Reporting bugs, suggesting features
Reporting a bug or an issue, suggesting a new feature in the [MISP core](https://github.com/MISP/MISP/issues), [misp-modules](https://github.com/MISP/misp-modules/issues), [misp-book](https://github.com/MISP/misp-book/issues), [misp-taxonomies](https://github.com/MISP/misp-taxonomies/issues), [misp-galaxy](https://github.com/MISP/misp-galaxy/issues) or any of the projects within the [MISP project org](https://github.com/MISP/). Don't hesitate to add as much information as you can, including the version of MISP which you are running, screen-shots with annotation, suggested features and steps on how to reproduce an issue. Don't hesitate to search the existing open issues and comment on existing ones. This is an indicator for us regarding the priority of certain features
and how important these are to the users.
The most common way to contribute to the MISP project is to report bugs, issues or suggest features.
Each project ([MISP core](https://github.com/MISP/MISP/issues), [misp-modules](https://github.com/MISP/misp-modules/issues), [misp-book](https://github.com/MISP/misp-book/issues), [misp-taxonomies](https://github.com/MISP/misp-taxonomies/issues), [misp-galaxy](https://github.com/MISP/misp-galaxy/issues) or any of the other projects within the [MISP project organanisation](https://github.com/MISP/)) had their own issue management.
Don’t forget that you can cross-reference issues from other sub-projects.
### Issue tracker guidelines
- **Use the provided issue template.** When reporting an issue on GitHub, please use one of the [issue templates](https://github.com/MISP/MISP/issues/new/choose). Do not delete it or remove parts of it. The issue template is carefully designed to elicit important information. Without this information, the issue is likely to be incomplete. It is also important to note the placement and content of the HTML comments in the issue template. These help us to have issues with a consistent format.
- **New issues should include all relevant information.** Add as much information as you can, including the version of MISP which you are running, screenshots with annotation, suggested features, and steps on how to reproduce an issue. You can also comment on existing issues; this is an indicator for us regarding the priority of certain features and how important these are to the users. If you know an answer or could help on a specific issue, we welcome all contributions including useful comments to reach a resolution.
- **Security policy.** To disclose a security issue confidentially, please see the [Reporting Security Vulnerabilities](#reporting-security-vulnerabilities) section.
- **New issues should not be duplicates of existing issues.** Before opening a new issue, search both open and closed issues to avoid duplicate issues. For example, you may be experiencing a bug that was just fixed, in which case the report for that bug is probably closed. Here, it would be useful to view all bug reports, both open and closed, with the most recently updated sorted to the top. If you find an issue that seems to be similar to yours, read through it. If you find an issue that is the same as or subsumes yours, leave a comment on the existing issue rather than filing a new one, even if the existing issue is closed. The MISP team will see your comment and reopen the issue, if appropriate. For example, you can leave a comment with additional information to help the maintainer debug it. Adding a comment will subscribe you to email notifications, which can be helpful in getting important updates regarding the issue. If you don’t have anything to add but still want to receive email updates, you can click the “Subscribe” button at the side or bottom of the comments. Commenting on existing issues is an indicator for us regarding the priority of certain features and how important these are to the users.
- **There are no guarantees that your issue will be addressed.** Creating a new issue is simply a way for you to submit an item for the MISP team’s consideration. It is up to the MISP team to decide whether or how to address your issue, which may include closing the issue without taking any action on it. Even if your issue is kept open, however, you should not expect it to be addressed within any particular time frame, or at all. At the time of this writing, there are well over 1.7 thousand open issues in the main MISP repo alone, not considering the other related repositories. The MISP team has its own [roadmap and priorities](/ROADMAP.md), which will govern the manner and order in which open issues are addressed.
### Following up afterward
If the MISP developers make a code change that resolves your issue, then your GitHub issue will typically be closed from the relevant patch message.
There is one main MISP core branch, `2.4` (current stable version), that we consider as stable with frequent updates as hotfixes.
Features are developed in separated branches and then regularly merged into the stable branch.
If you so choose, you can test the fix while it’s in the feature branch, or you can wait for it to land in the stable repository.
If, after testing the fix, you find that it does not fix your bug, please leave a comment on your issue explaining the situation.
When you do, we will receive a notification and respond to your issue or reopen it (or both).
Please do not create a duplicate issue.
In other cases, your issue may be closed with a specific resolution, such as `R: invalid`, `R: duplicate`, or `R: wontfix`.
Each of these labels has a description that explains the label.
We’ll also leave a comment explaining why we’re closing the issue with one of these specific resolutions.
If the issue is closed without one of these specific resolutions or a comment, then it means, by default, that your reported bug was fixed or your requested enhancement was implemented.
## Reporting security vulnerabilities
Reporting security vulnerabilities is of great importance for us, as MISP is used in multiple critical infrastructures. In the case of a security vulnerability report, we ask the reporter to directly report to [CIRCL](https://www.circl.lu/contact/), encrypting the report with the GnuPG key: CA57 2205 C002 4E06 BA70 BE89 EAAD CFFC 22BD 4CD5. We usually fix reported and confirmed security vulnerabilities in less than 48 hours, followed by a software release containing the fixes within the following days. If you report security vulnerabilities, don't forget to tell us if and how you want to be acknowledged and if you already requested CVE(s). If not, we will request the CVE directly.
View our [Security Policy](https://github.com/MISP/MISP/security/policy).
As one of the critical user-bases of MISP consists of the CSIRT community, it is our duty to clearly state which bug could be potentially abused and could have a security impact on a deployed MISP instance. CVE assignment is performed even for minor bugs having some possible security impact. This allows users using MISP instances in their environment to understand which bugs could have an impact on their security. We firmly believe that, even though unfortunately it is often not regarded as common practice in our industry, being as transparent as possible about vulnerabilities, no matter how minor, is of absolute crucial importance. At MISP-project, we care about the security of our users and prefer to have a high number of published CVEs than to a few swept under the rug.
## MISP core
## Contributing to MISP core
Before you get started, read our [coding guidelines](/CODINGSTYLE.md).
If you want to contribute to the [MISP core](https://github.com/MISP/MISP) project:
- First fork the [MISP core project](https://github.com/MISP/MISP)
- Branch off from 2.4 (2.4 branch is the main branch of development in MISP) `git checkout 2.4`
- Then create a branch for your own contribution (bug fixes, enhancement, new features) by typing `git checkout -b fix-glossy-user-interface`
- Work on your fix or feature (only work on that, avoid committing any debug functionalities, testing or unused code)
- Commit your fix or feature (and sign it with GnuPG - if you have a GnuPG key) with a meaningful commit message as recommended in [A Note About Git Commit Messages](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
- MISP uses [gitchangelog](https://github.com/vaab/gitchangelog/blob/master/src/gitchangelog/gitchangelog.rc.reference) to generate changelog, so it's recommended to use `new:` for new features, `fix:` when it's a bug-fix or `chg` when it's re-factoring or clean-up..
- Push and then open a pull-request via the GitHub interface.
- Work on your fix or feature (only work on that, avoid committing any debug functionalities, testing, or unused code)
- Commit your fix or feature (and sign it with GnuPG -- if you have a GnuPG key) with a meaningful commit message as recommended in our [Commit Messages Best Practices](https://github.com/MISP/MISP/wiki/CommitMessageBestPractices). MISP uses [gitchangelog](https://github.com/vaab/gitchangelog/blob/master/src/gitchangelog/gitchangelog.rc.reference) to generate changelog, so it's recommended that when writing commit messages, use `new:` for new features, `fix:` when it's a bug-fix or `chg` when it's re-factoring or clean-up.
- Push and then [open a pull-request](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request) via the GitHub interface.
If you never did a pull-request, there is [The beginner's guide to contributing to a GitHub project](https://akrabat.com/the-beginners-guide-to-contributing-to-a-github-project/)
For changes in categories and types see the [Categories & Types Changes CheckList](https://github.com/MISP/MISP/wiki/Categories-&-Types-changes-CheckList).
If you have never done a pull-request, there is [The beginner's guide to contributing to a GitHub project](https://akrabat.com/the-beginners-guide-to-contributing-to-a-github-project/)
Some recommendations to get your PR merged quickly:
- Small gradual changes are preferred over big and complex changes.
- If the changes require some documentation changes, a pull-request on [misp-book](https://github.com/MISP/misp-book) is strongly recommended.
- If the commit message contain all the information regarding the changes, it's easier for the maintainer to do the review.
- Avoid committing sensitive information, debugging code or unrelated code in the PR.
- If the commit message contains all the information regarding the changes, it's easier for the maintainer to do the review.
- Avoid committing sensitive information, debugging code, or unrelated code in the PR.
## MISP taxonomies
## Contributing to a JSON library (objects, taxonomies, galaxy or warning-list)
If you cannot find an existing taxonomy fitting your needs, you can extend an existing one (especially the ones originated from the MISP project) or create a new one.
All of MISP's JSON format (galaxy, taxonomies, objects, or warning-lists) are described in a JSON Schema, named using the convention `schema_name.json`.
The TravisCI tests include JSON validation (via `jq`) and are validated with the associated JSON Schema.
When you update a JSON library, validate the associated JSON schema by running `jq_all_the_things.sh`.
This prompts the JSON validation tests (via `jq`) using [TravisCI](https://travis-ci.org/MISP). It should be fast and easy.
If the checks fail, review your JSON.
Once everything works, commit your code and make a pull request against the specific library.
Create a JSON file describing your taxonomy as triple tags (e.g. check an existing one such as the [Admiralty Scale](https://github.com/MISP/misp-taxonomies/tree/master/admiralty-scale)) taxonomy, create a directory matching your name space, put your machinetag file in the directory and create a pull request. That's it. Everyone can benefit from your taxonomy and it can be automatically enabled in information sharing tools such as [MISP](https://www.github.com/MISP/MISP).
Documentation (in PDF and HTML format) for the librairies are automatically generated from the JSON via [asciidoctor](https://asciidoctor.org/). Look at [this example](https://github.com/MISP/misp-galaxy/blob/main/tools/adoc_galaxy.py).
We strongly recommend to validate the JSON file using [jq](https://github.com/MISP/misp-taxonomies/blob/master/jq_all_the_things.sh) before doing a pull-request. We also strongly recommend to run [the validator](https://github.com/MISP/misp-taxonomies/blob/master/validate_all.sh) to check if the JSON validates the schema.
## Contributing to MISP taxonomies
If you cannot find an existing taxonomy fitting your needs, you can extend an existing one (especially the ones that originated from the MISP project) or create a new one. To do this:
1. Create a JSON file describing your taxonomy as triple tags (e.g. check an existing one such as the [Admiralty Scale](https://github.com/MISP/misp-taxonomies/tree/master/admiralty-scale)) taxonomy
2. Create a directory matching your namespace
3. Put your machinetag file in the directory
4. (Optional, but recommended) Validate the JSON file using [jq](https://github.com/MISP/misp-taxonomies/blob/master/jq_all_the_things.sh) and run [the validator](https://github.com/MISP/misp-taxonomies/blob/master/validate_all.sh) to check if the JSON validates the schema.
5. Commit your change and create a pull request
Everyone can benefit from your taxonomy and it can be automatically enabled in information-sharing tools such as [MISP](https://www.github.com/MISP/MISP).
For more information, see the presentation slides on "[Information Sharing and Taxonomies Practical Classification of Threat Indicators using MISP](https://www.circl.lu/assets/files/misp-training/3.2-MISP-Taxonomy-Tagging.pdf)" given at the last MISP training in Luxembourg.
## MISP galaxy
## Contributing to MISP galaxy
In the world of threat intelligence, there are many different models or approaches to order, classify or describe threat actors, threats or activity groups. We welcome new ways of describing threat intelligence as the galaxy model allows to reuse the ones you use or trust for your organization or community.
In the world of threat intelligence, there are many different models or approaches to order, classify or describe threat actors, threats, or activity groups.
We welcome new ways of describing threat intelligence as the galaxy model allows you to reuse the ones you use or trust for your organization or community.
Fork the project, update or create a galaxy or clusters and make a pull-request.
To contribute, fork the [project](https://github.com/MISP/misp-galaxy), update or create a galaxy or clusters and make a pull-request.
We strongly recommend to validate the JSON file using [jq](https://github.com/MISP/misp-galaxy/blob/master/jq_all_the_things.sh) before doing a pull-request. We also strongly recommend to run [the validator](https://github.com/MISP/misp-galaxy/blob/master/validate_all.sh) to check if the JSON validates the schema.
Before making a pull request, we strongly recommend validating the JSON file using [jq](https://github.com/MISP/misp-galaxy/blob/master/jq_all_the_things.sh) and run [the validator](https://github.com/MISP/misp-galaxy/blob/master/validate_all.sh) to check if the JSON validates the schema.
## Building software compatible with MISP formats and improving MISP formats
[MISP formats](https://github.com/MISP/misp-rfc) are open and free standards. We invite software developers to use the native MISP format for exchanging threat intelligence and support information sharing/analysis. The MISP formats are simple JSON formats implemented in various software including the MISP core application along with various libraries such as [PyMISP](https://github.com/MISP/PyMISP).
[MISP formats](https://github.com/MISP/misp-rfc) are open and free standards, which are actively based on the MISP core implementation.
MISP formats are simple JSON formats implemented in various software including the MISP core application along with various libraries such as [PyMISP](https://github.com/MISP/PyMISP).
We invite software developers to use the native MISP format for exchanging threat intelligence and support information sharing/analysis.
If you want to contribute to the MISP formats (which are actively based on the MISP core implementation), feel free to open an issue at [misp-rfc](https://github.com/MISP/misp-rfc).
If you want to contribute to our IETF Internet-Draft for the MISP standard, [misp-rfc](https://github.com/MISP/misp-rfc) is the repository to propose changes.
Each format folder has several files of different extensions, including XML and Markdown (MD).
You should update only the Markdown file; the XML and ASCII for the IETF I-D are automatically generated.
If a major release or updates happen in the format, we will [publish the I-D to the IETF](https://datatracker.ietf.org/doc/search/?name=misp&
activedrafts=on&rfcs=on).
## Writing documentation
We will appreciate new documentation or improvements on our documentation, particularly [MISP Book](https://github.com/MISP/misp-book).
See our [specific guidelines on contributing to MISP Book](https://github.com/MISP/misp-book/blob/main/CONTRIBUTING.md).
## Automatic integration and testing
The majority of the repositories within the MISP GitHub organisation include automatic integration with TravisCI.
Feedback and patches are welcome.
For example, you can propose new tests that we could run on Travis CI, or suggest additional automatic tests including unit testing for the MISP core software Please explain the expected benefit of your work on MISP developers and users -- this will help us prioritize our work.
## Testing new releases and updates
Testing new MISP releases and updates is one of the ways that you can contribute to the MISP Project.
However, you should only attempt to do this if you know what you’re doing. Never rely on code that is in testing for critical work!
After your testing, we would be grateful for your feedback via GitHub issues.
If you would like to test MISP and don’t want to do an installation, you can use automatically-generated VM images. See more on our [download page](https://www.misp-project.org/download/#virtual-images).
## Translating MISP
Thank you for your interest in making MISP easier to use in your language!
We accept translations on Crowdin: https://crowdin.com/project/misp.
Crowdin is a localization management platform that helps companies to translate their software.
Note that CrowdIn is independent of MISP and they have their own [privacy policy](https://support.crowdin.com/privacy-policy/).
You can help correct, improve, or complete the translations of MISP programs into your native language.
MISP can be translated into more than 140 languages this way.
Most of MISP can be translated directly online, through a simple web interface, after logging in with CrowdIn.
To get started with using CrowdIn, you can [read their introductory article](https://support.crowdin.com/for-volunteer-translators/).
Note that only reviewed translations are included in MISP.
If you want to go further, reach out to us if you want to help translate the MISP website or MISP Book.
## For native English speakers
Most MISP developers are not native English speakers so you are more than welcome to correct or improve our English.
For this, you can either submit a pull request or another way that suits you better to share your improvements with us.
## Improving the MISP experience
As a MISP user, you can contribute to our UX efforts by filling the [MISP User Experience Survey](https://misp-project.org/ux-survey).
The purpose of this survey is to assess the user experience of MISP and learn more about the needs of its users.
If you're a UX researcher or designer and want to help with UX issues, look at [issues with the `UX` label](https://github.com/MISP/MISP/labels/UX) on our GitHub issue tracker, corresponding to things that we would like to see improved in the MISP user experience. If you have other ideas to improve MISP, let us know! Even if you don't implement your suggestions yourself, create issues with your ideas so that others can benefit from your insight.
You can also help us refine or enhance our [user personas](https://www.circl.lu/doc/misp/user-personas/) and [user stories](https://www.circl.lu/doc/misp/user-stories/).
For any questions or comments related to UX, please get in touch with us at <ux@misp-project.org>
## Improving the website
We have identified some things that we would like to see improved on our website: see the [website-related issues on GitHub](https://github.com/MISP/misp-website/issues). We are also open to new suggestions on what should be improved.
The website https://misp-project.org/ is built using Jekyll.
You can [build a local copy of the website](https://github.com/MISP/misp-website/issues) on your computer.
Building the website produces a set of HTML pages stored on your system that you can open in your usual web browser even while working offline.
Doing so is useful for writers and designers to see how their changes will apply to the website.

157
GITWORKFLOW.md Normal file
View File

@ -0,0 +1,157 @@
# How we use GitHub
MISP issues and code are managed on [GitHub](https://github.com/MISP).
This page focuses on aspects of GitHub usage that are specific to MISP.
This workflow is not set in stone: it will evolve based on what we learn while using GitHub.
For general GitHub usage information, see the [GitHub user documentation](https://docs.github.com/en/github).
## Get started
To create your GitHub account, visit the [registration page](https://github.com/join) in a web browser.
Then you will be allowed to open new issues, create merge requests, and fork your copy of our repositories.
## Organisation
All of our repositories are available under the [MISP GitHub account](https://github.com/MISP).
The main Git repository and most issues live in the [MISP/MISP repository](https://github.com/MISP/MISP).
We use git to maintain all code for MISP Project. We have divided the project into several components, each of which has its own separate repository. For example:
- `MISP.git` – The core MISP code.
- `misp-taxonomies.git` – Taxonomies used in MISP taxonomy system and can be used by other information-sharing tools.
- `misp-galaxy.git` – Clusters and elements to attach to MISP events or attributes (like threat actors).
- `misp-warninglists.git` – Warning lists to inform users of MISP about potential false positives or other information in indicators.
MISP code is divided into many git repositories for the following reasons:
- This creates natural boundaries between different code blocks, enforcing proper interfaces, and easing independent development to be conducted on various code parts at the same time, without the fear of running into conflicts.
- By maintaining relatively small git repositories, it is easy for new developers to understand the code and contribute new patches, without the need to understand all the other code.
- Code repositories represent also licensing boundaries. So, e.g. because misp-objects and PYMISP are maintained in two different repositories, it is possible to have the latter under a CC0 1.0 Universal license, while keeping the former under a simplified 2-BSD license.
### Git branches organisation
MISP uses several branches:
- Version branch: This is the default branch, and it is named to reflect the latest MISP release. At the time of writing, the default branch is `2.4`
- Topic branches: We use topic branches, sometimes called `fix-*` and `feature-*`, respectively aimed at fixing a single bug and implementing a single new feature. Once ready, a topic branch is merged into the appropriate branch (generally the default branch). Until it has been merged, a topic branch's history may be rewritten.
When the MISP developers make a code change that resolves an issue, the GitHub issue will typically be closed from the relevant patch message.
The main MISP core branch, `2.4` (current stable version), that we consider as stable with frequent updates as hotfixes.
Features are developed in separated branches and then regularly merged into the `2.4` stable branch.
## How we use GitHub metadata
On GitHub, issues and merge requests have metadata.
Being consistent in the use of GitHub metadata makes collective work smoother.
### Title and description
The title should be a short but clear description of what this is about. Some people are case sensitive, please try to consider that.
### Status and resolution
Open issues may have a status label, starting with `S:`. Each label has a description of what it represents.
See the [list of status labels](https://github.com/MISP/MISP/labels?q=S%3A+), along with their descriptions.
Closed issue may have up to one resolution label, starting with `R:`. See the [list of resolution labels](https://github.com/MISP/MISP/labels?q=R%3A+) and their descriptions. When an issue does not fit into MISP's mission, we reject it by creating a comment explaining why it's being rejected, closing the issue, and the `R: wontfix` label. When an issue is closed as a duplicate, we create a comment that mentions the other, duplicated issue and add the `R: duplicate` label. If the issue is closed without one of the specific resolutions or a comment, then it means, by default, that the issue in question was fixed or the requested enhancement was implemented.
The main advantage of using these labels is to organize and visualize issues on Issue Boards.
Using these labels accurately improves the team's workflow.
### Assignee
We use the Assignee field in a way that helps us organize our work as a team, focus on what matters most currently, and avoid individual over-commitment and feelings of failure.
To this aim, most tasks should be up for grabs for anyone who has spare capacity and the required skills.
So in general, issues and merge requests should not be assigned to anyone.
This being said, we do assign them if at least one of these conditions is met:
- Someone is actively working on it.
- The task is important and urgent.
- The milestone is set to the next MISP release.
- Only one of us can complete the task. This helps identify bottlenecks and over-commitment.
### Milestone
Sometimes, [Milestones](https://github.com/MISP/MISP/milestones) are treated as a commitment that other MISP contributors should be able to rely on.
Other times, it is used it as a pool of tasks they want to have on their short-term radar.
### Type of work
To indicate the type of work that's needed to complete the next step on an issue, we use labels that start with `T:`.
See the [list of the 'type of work' labels](https://github.com/MISP/MISP/labels?q=t%3A) and their descriptions.
### Other labels
See our [full list of labels](https://github.com/MISP/MISP/labels) for other uncategorized labels we use.
## Relationships between issues
GitHub is a bit limited when it comes to expressing semantic relationships between issues. Here is how we can overcome these limitations.
- Parent/subtask: Issues with the `T: meta` label indicate parent issues that have subtasks under it. In the child issues, a comment will be added mentioning the parent issue.
- Related issues: Related issues can be listed either in the description or in a comment. Either way, this adds a message in the activity stream of the referenced issue, with a link to the referencing issue.
## How to document progress
### Create and update issues
For details about labels, see [metadata](#how-we-use-github-metadata). If you are very certain to work on the issue, leave a comment expressing your interest or ask to be assigned to it to avoid duplicate work.
All the knowledge useful to the others should be kept on that issue, or at least linked from there.
When committing changes that will resolve an issue once merged, please include #NNNN in the commit message, NNNN being the issue number. Then, GitHub will automatically reference this commit on the corresponding issue, once the branch is pushed to our Git repository. For example:
```
chg: [doc] Fix spelling errors (#3120)
```
### Report progress or failure
It is important to:
- Keep the team informed of how you feel committed to issues assigned to you, and about the timeline you're envisioning.
- Avoid individual over-commitment & feelings of failure.
If you don't think you will manage to work on an issue any time soon, it's often best to make this clear on the issue or to de-assign yourself.
### Propose changes
We use Pull Requests (aka. PRs) to propose, review, and merge changes.
You can comment on issues and pull requests.
[Our code of conduct](https://github.com/MISP/MISP/blob/2.4/code_of_conduct.md) applies.
To submit your work:
- Fork us on our GitHub
- Push your work to a dedicated git topic branch
- Once you would like your branch to be reviewed, and possibly merged, submit it by creating a pull request (PR). In this new PR, use the description field to summarize what problem this PR will fix, in terms of impact on users, and reference the issues this PR will solve, e.g "Closes #xxx, #yyyy".
Follow these conventions when submitting changes to any MISP Project repository:
- Submit one pull request (PR) per fix/feature/change. Don't split one feature into multiple PRs. Similarly, do not join several fixes and features into one pull request.
- Keep the number of commits per PR as small as possible. If for any reason, you need to fix your commit after the pull request, please squash the changes in one single commit (or tell us why not).
- Always ensure your PR is mergeable in the default branch.
- Make sure Travis CI works on the PR, or update the test cases if needed.
- Any major changes adding a functionality should be disabled by default in the config.
### Request input from someone else
If you need input from someone else on an issue or pull request, ask your question in a comment there, mentioning them with their GitHub login name: @nick. If you want to raise the attention of every single member of a team, mention it with the name of the corresponding group: @xyz-team. GitHub will send them an email notification about it.
### Act upon input requests
It's important to provide the requested information as quickly as you can, to make the MISP contribution process more efficient and enjoyable.
When input is requested from you on an issue or pull request with @nick:
- GitHub may send you an email notification
- Please ensure your GitHub email notification settings and your email setup allow you to receive and notice these messages.
When you receive such a request, if you cannot provide the requested input immediately, you're responsible for keeping track of this task. For example, by creating a new issue assigned to yourself, or using whatever personal organization tools work for you.
## Automatic integration and testing
MISP core uses CodeQL and LGTM for some code analysis and security checks. When you submit a PR, consider the results of checks. Also, ensure that your code builds without errors.
The majority of the repositories within the MISP GitHub organisation include automatic integration with [TravisCI](https://travis-ci.org/MISP).
If you contribute and make a pull request, verify if your changes affect the result of the tests.
Automatic integration is not perfect including Travis but it’s a quick way to catch new bugs or major issues in contribution.
When you make a pull request, TravisCI is automatically called. If there are failing checks, no worries, review the output at Travis (it’s not
always you).
## Access control
If you need to do something in GitHub and you appear to lack the needed credentials, please ask the MISP team to grant you more power.
For example, you will need "Triage" access to add labels or assign issues. See [GitHub's access permissions documentation](https://docs.github.com/en/github/getting-started-with-github/access-permissions-on-github).

2
PyMISP

@ -1 +1 @@
Subproject commit 95e31bd2e3b4f14b55001a40cd51e44ec5cf7644
Subproject commit 67457bec5380139af3aa8dc93c021457659c5697

43
app/Console/cake.php
View File

@ -3,45 +3,30 @@
/**
* Command-line code generation utility to automate programmer chores.
*
* CakePHP(tm) : Rapid Development Framework (https://cakephp.org)
* Copyright (c) Cake Software Foundation, Inc. (https://cakefoundation.org)
* PHP 5
*
* CakePHP(tm) : Rapid Development Framework (http://cakephp.org)
* Copyright 2005-2012, Cake Software Foundation, Inc.
*
* Licensed under The MIT License
* For full copyright and license information, please see the LICENSE.txt
* Redistributions of files must retain the above copyright notice.
*
* @copyright Copyright (c) Cake Software Foundation, Inc. (https://cakefoundation.org)
* @link https://cakephp.org CakePHP(tm) Project
* @copyright Copyright 2005-2012, Cake Software Foundation, Inc. (http://cakefoundation.org)
* @link http://cakephp.org CakePHP(tm) Project
* @package app.Console
* @since CakePHP(tm) v 2.0
* @license https://opensource.org/licenses/mit-license.php MIT License
* @license MIT License (http://www.opensource.org/licenses/mit-license.php)
*/
if (!defined('DS')) {
define('DS', DIRECTORY_SEPARATOR);
}
$dispatcher = 'Cake' . DS . 'Console' . DS . 'ShellDispatcher.php';
$ds = DIRECTORY_SEPARATOR;
$dispatcher = dirname(__DIR__) . $ds . 'Lib' . $ds . 'cakephp' . $ds . 'lib' . $ds . 'Cake' . $ds . 'Console' . $ds . 'ShellDispatcher.php';
if (function_exists('ini_set')) {
$root = dirname(dirname(dirname(__FILE__)));
$appDir = basename(dirname(dirname(__FILE__)));
$install = $root . DS . 'lib';
$composerInstall = $root . DS . $appDir . DS . 'Vendor' . DS . 'cakephp' . DS . 'cakephp' . DS . 'lib';
// the following lines differ from its sibling
// /lib/Cake/Console/Templates/skel/Console/cake.php
if (file_exists($composerInstall . DS . $dispatcher)) {
$install = $composerInstall;
}
ini_set('include_path', $install . PATH_SEPARATOR . ini_get('include_path'));
unset($root, $appDir, $install, $composerInstall);
$root = dirname(dirname(dirname(__FILE__)));
ini_set('include_path', $root . $ds . 'lib' . PATH_SEPARATOR . ini_get('include_path'));
}
if (!include $dispatcher) {
trigger_error('Could not locate CakePHP core files.', E_USER_ERROR);
if (!include ($dispatcher)) {
trigger_error('Could not locate CakePHP core files.', E_USER_ERROR);
}
unset($dispatcher);
unset($paths, $path, $dispatcher, $root, $ds);
return ShellDispatcher::run($argv);

4
app/Controller/TagsController.php
View File

@ -885,7 +885,9 @@ class TagsController extends AppController
$failedMessage = __('Failed to attach %s tags. Reasons: %s', count($fails), json_encode($fails, JSON_FORCE_OBJECT));
}
if ($successes > 0) {
$message = __('Successfully attached %s tags to %s (%s)', $successes, $objectType, $object[$objectType]['id']);
if ($successes > 1) {
$message = __('Successfully attached %s tags to %s (%s)', $successes, $objectType, $object[$objectType]['id']);
}
$message .= !empty($fails) ? PHP_EOL . $failedMessage : '';
return $this->RestResponse->saveSuccessResponse('Tags', 'attachTagToObject', false, $this->response->type(), $message);
} else {

96
app/Lib/Dashboard/OrgsEvolutionWidget.php Normal file
View File

@ -0,0 +1,96 @@
<?php
class OrgsEvolutionWidget
{
public $title = 'Evolution of orgs count';
public $render = 'MultiLineChart';
public $width = 7;
public $height = 6;
public $description = 'A graph to show the evolution of total users over time. The distinction between remote and local org for all datapoints is based on the current state as there is no historical data for it.';
public $cacheLifetime = 10;
public $autoRefreshDelay = false;
public $params = array(
'days' => 'Number of days to consider for the graph, takes priority over months and weeks. There will be a data entry for each day. Value between 1 and 180.',
'weeks' => 'Number of days to consider for the graph, takes priority over months. There will be a data entry for each week. Value between 1 and 180.',
'months' => 'Number of days to consider for the graph. There will be a data entry for each month. Value between 1 and 180.',
);
public $placeholder =
'{
"days": "30",
"widget_config": {
"enable_total": "1"
}
}';
public function handler($user, $options = array())
{
$this->Organisation = ClassRegistry::init('Organisation');
$currentTime = strtotime("now");
$endOfDay = strtotime("tomorrow", $currentTime) - 1;
if (!empty($options['days'])) {
$limit = (int)($options['days']);
$delta = 'day';
} else if (!empty($options['weeks'])) {
$limit = (int)($options['weeks']);
$delta = 'week';
} else if (!empty($options['months'])) {
$limit = (int)($options['months']);
$delta = 'month';
} else {
$limit = 30;
$delta = 'day';
}
if ($limit <= 0 || $limit > 180) {
throw new InvalidArgumentException("Number of days, weeks or months must be a number between 1 and 180.");
}
$data = array();
$data['data'] = array();
// Add total users data for all timestamps
for ($i = 0; $i < $limit; $i++) {
$itemTime = strtotime('- ' . $i . $delta, $endOfDay);
//Separate time for db query as date_created is stored in DateTime string
$itemTimeDateTime = date('Y-m-d H:i:s', $itemTime);
$item = array();
$item['date'] = strftime('%Y-%m-%d', $itemTime);
$item['local orgs'] = $this->localOrgsAtTime($itemTimeDateTime);
$item['remote orgs'] = $this->remoteOrgsAtTime($itemTimeDateTime);
$data['data'][] = $item;
}
return $data;
}
private function localOrgsAtTime($time)
{
return $this->Organisation->find('count', array(
'recursive' => -1,
'conditions' => array(
'date_created <=' => $time,
'local' => 1
)
));
}
private function remoteOrgsAtTime($time)
{
return $this->Organisation->find('count', array(
'recursive' => -1,
'conditions' => array(
'date_created <=' => $time,
'local' => 0
)
));
}
public function checkPermissions($user)
{
if (empty($user['Role']['perm_site_admin'])) {
return false;
}
return true;
}
}

82
app/Lib/Dashboard/UsersEvolutionWidget.php Normal file
View File

@ -0,0 +1,82 @@
<?php
class UsersEvolutionWidget
{
public $title = 'Evolution of user count';
public $render = 'MultiLineChart';
public $width = 7;
public $height = 6;
public $description = 'A graph to show the evolution of total users over time';
public $cacheLifetime = 10;
public $autoRefreshDelay = false;
public $params = array(
'days' => 'Number of days to consider for the graph, takes priority over months and weeks. There will be a data entry for each day. Value between 1 and 180.',
'weeks' => 'Number of days to consider for the graph, takes priority over months. There will be a data entry for each week. Value between 1 and 180.',
'months' => 'Number of days to consider for the graph. There will be a data entry for each month. Value between 1 and 180.',
);
public $placeholder =
'{
"days": "30"
}';
public function handler($user, $options = array())
{
$this->User = ClassRegistry::init('User');
$currentTime = strtotime("now");
$endOfDay = strtotime("tomorrow", $currentTime) - 1;
if (!empty($options['days'])) {
$limit = (int)($options['days']);
$delta = 'day';
} else if (!empty($options['weeks'])) {
$limit = (int)($options['weeks']);
$delta = 'week';
} else if (!empty($options['months'])) {
$limit = (int)($options['months']);
$delta = 'month';
} else {
$limit = 30;
$delta = 'day';
}
if ($limit <= 0 || $limit > 180) {
throw new InvalidArgumentException("Number of days, weeks or months must be a number between 1 and 180.");
}
$data = array();
$data['data'] = array();
// Add total users data for all timestamps
for ($i = 0; $i < $limit; $i++) {
$itemTime = strtotime('- ' . $i . $delta, $endOfDay);
$item = array();
$item['date'] = strftime('%Y-%m-%d', $itemTime);
$item['users'] = $this->usersAtTime($itemTime);
$data['data'][] = $item;
}
return $data;
}
private function usersAtTime($time)
{
return $this->User->find('count', array(
'recursive' => -1,
'conditions' => array(
'OR' => array(
array('date_created' => null),
array('date_created <=' => $time),
)
)
));
}
public function checkPermissions($user)
{
if (empty($user['Role']['perm_site_admin'])) {
return false;
}
return true;
}
}

10
app/Lib/Tools/RandomTool.php
View File

@ -2,6 +2,16 @@
class RandomTool
{
public function __construct()
{
// import compatibility library for PHP < 7.0
if (!function_exists('random_int')) {
if (file_exists(APP . 'Lib' . DS . 'random_compat' . DS . 'lib' . DS . 'random.php')) {
require_once(APP . 'Lib' . DS . 'random_compat' . DS . 'lib' . DS . 'random.php');
}
}
}
/**
* Generate a random string
*

1
app/Lib/cakephp Submodule

@ -0,0 +1 @@
Subproject commit cf14e6546ec44e3369e3531add11fdb946656280

1
app/Lib/random_compat Submodule

@ -0,0 +1 @@
Subproject commit 088c04e2f261c33bed6ca5245491cfca69195ccf

6
app/Model/MispObject.php
View File

@ -317,7 +317,7 @@ class MispObject extends AppModel
}
$newObjectAttributes[] = hash(
'sha256',
$attribute['object_relation'] . $attribute['category'] . $attribute['type'] . $this->data['Attribute']['value'] = $this->Attribute->modifyBeforeValidation($attribute['type'], $attribute['value'])
$attribute['object_relation'] . $attribute['category'] . $attribute['type'] . $this->Attribute->modifyBeforeValidation($attribute['type'], $attribute['value'])
);
}
$newObjectAttributeCount = count($newObjectAttributes);
@ -1056,8 +1056,8 @@ class MispObject extends AppModel
'email' => $user['email'],
'action' => 'edit',
'user_id' => $user['id'],
'title' => 'Attribute dropped due to validation for Event ' . $eventId . ' failed: ' . $object['name'],
'change' => 'Validation errors: ' . json_encode($this->validationErrors) . ' Full Object: ' . json_encode($attribute),
'title' => 'Object dropped due to validation for Event ' . $eventId . ' failed: ' . $object['name'],
'change' => 'Validation errors: ' . json_encode($this->validationErrors) . ' Full Object: ' . json_encode($object),
));
return $this->validationErrors;
}

3
app/Plugin/OidcAuth/Controller/Component/Auth/OidcAuthenticate.php
View File

@ -47,6 +47,9 @@ class OidcAuthenticate extends BaseAuthenticate
if (property_exists($verifiedClaims, $roleProperty)) {
$roles = $verifiedClaims->{$roleProperty};
}
if (empty($roles)) {
$roles = $oidc->requestUserInfo($roleProperty);
}
$this->settings['fields'] = ['username' => 'email'];
$user = $this->_findUser($mispUsername);

14
app/View/Elements/genericElements/IndexTable/Fields/links.ctp
View File

@ -18,16 +18,13 @@
$url_param_data_paths = implode('/', $temp);
} else {
$url_param_data_paths = Hash::extract($row, $field['url_params_data_paths']);
if (!empty($url_param_data_paths)) {
$url_param_data_paths = $url_param_data_paths[0];
} else {
if (empty($url_param_data_paths)) {
$url_param_data_paths = '';
}
}
$urlWithData .= '/' . $url_param_data_paths;
}
$links = array();
foreach ($data_elements as $data) {
foreach ($data_elements as $k => $data) {
if (!empty($data['name'])) {
$field['title'] = $data['name'];
}
@ -40,6 +37,13 @@
$data
);
} elseif (!empty($field['url_params_data_paths'])) {
if (!empty($url_param_data_paths)) {
if (is_array($url_param_data_paths)) {
$urlWithData .= '/' . $url_param_data_paths[$k];
} else {
$urlWithData .= '/' . $url_param_data_paths;
}
}
$url = $urlWithData;
} else {
$url = $data;

18
app/View/Elements/genericElements/IndexTable/Fields/timestamp.ctp
View File

@ -12,15 +12,15 @@
$offset = ceil($offset / 60);
$unit = 'minute(s)';
$colour = 'orange';
}
if ($offset >= 60) {
$offset = ceil($offset / 60);
$unit = 'hour(s)';
$colour = 'green';
}
if ($offset >= 24) {
$offset = ceil($offset / 24);
$unit = 'day(s)';
if ($offset >= 60) {
$offset = ceil($offset / 60);
$unit = 'hour(s)';
$colour = 'green';
if ($offset >= 24) {
$offset = floor($offset / 24);
$unit = 'day(s)';
}
}
}
$raw = true;
$timestamp = sprintf('<span class="bold %s">%s %s ago</span>', $colour, $offset, $unit);

2
app/composer.json
View File

@ -9,8 +9,6 @@
"ext-dom": "*",
"ext-simplexml": "*",
"ext-pcre": "*",
"cakephp/cakephp": "2.10.24",
"paragonie/random_compat": ">=2",
"kamisama/cake-resque": "4.1.2",
"pear/crypt_gpg": "1.6.3",
"monolog/monolog": "1.24.0"

2
app/files/misp-galaxy

@ -1 +1 @@
Subproject commit eba33a1c91c59b41dd9bc58d80ed3a1e54cdb2e1
Subproject commit 5cdad926716caad5fec6049431cc6ae42a8d3bf7

2
app/files/warninglists

@ -1 +1 @@
Subproject commit 43592376c030195575976ae33594487d10c40a9f
Subproject commit 892ac72da4efc81531003102d5b3af446ebb44a5

10
app/webroot/index.php
View File

@ -58,16 +58,6 @@ if (!defined('APP_DIR')) {
*/
define('CAKE_CORE_INCLUDE_PATH', ROOT . DS . APP_DIR . DS .'Lib' . DS . 'cakephp' . DS . 'lib');
/**
* This auto-detects CakePHP as a composer installed library.
* You may remove this if you are not planning to use composer (not recommended, though).
*/
$vendorPath = ROOT . DS . APP_DIR . DS . 'Vendor' . DS . 'cakephp' . DS . 'cakephp' . DS . 'lib';
$dispatcher = 'Cake' . DS . 'Console' . DS . 'ShellDispatcher.php';
if (!defined('CAKE_CORE_INCLUDE_PATH') && file_exists($vendorPath . DS . $dispatcher)) {
define('CAKE_CORE_INCLUDE_PATH', $vendorPath);
}
/**
* Editing below this line should NOT be necessary.
* Change at your own risk.

1
docs/INSTALL.rhel7.md
View File

@ -158,6 +158,7 @@ yumInstallCoreDeps7 () {
PHP_INI="/etc/php.ini"
sudo dnf install gcc git zip unzip \
mod_ssl \
moreutils \
redis \
libxslt-devel zlib-devel ssdeep-devel -y

1
docs/INSTALL.rhel8.md
View File

@ -151,6 +151,7 @@ yumInstallCoreDeps8 () {
sudo dnf install gcc git zip unzip \
httpd \
mod_ssl \
moreutils \
redis \
mariadb \
mariadb-server \