MISP
/
MISP
mirror of https://github.com/MISP/MISP
2
2
Fork 0
Code Issues Releases Wiki Activity

Merge branch '2.4' of github.com:MISP/MISP into pr-view_picutre

pull/5553/head
mokaddem 2020-01-24 10:42:35 +01:00
parent 3d81f769bb 0c68bc7de7
commit d16369f4b3
No known key found for this signature in database
GPG Key ID: 164C473F627A06FA
223 changed files with 16620 additions and 3815 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
.github/ISSUE_TEMPLATE.md → .github/ISSUE_TEMPLATE/bug_report.md vendored
View File

@ -1,6 +1,11 @@
# This template is meant for bug reports, if you have a feature request, please be as descriptive as possible and delete the template
---
name: Bug report
about: Create a report to help us improve
labels: bug, potential-bug, needs triage
---
# This template is meant for bug reports, if you have a feature request user the other template.
@ -15,9 +20,9 @@
| Questions | Answers
|---------------------------|--------------------
| Type of issue | Bug, Question, Feature Request, support...
| OS version (server) | Debian, ubuntu, CentOS, RedHat, ...
| OS version (client) | XP, Seven, 10, Ubuntu, ...
| Type of issue | Bug
| OS version (server) | Debian, Ubuntu, CentOS, RedHat, ...
| OS version (client) | MacOS, Win10, Ubuntu, ...
| PHP version | 5.4, 5.5, 5.6, 7.0, 7.1...
| MISP version / git hash | 2.4.XX, hash of the commit
| Browser | If applicable

19
.github/ISSUE_TEMPLATE/feature_request.md vendored Normal file
View File

@ -0,0 +1,19 @@
---
name: Feature request
about: Suggest an idea for this project
labels: feature request, needs triage
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.

34
.github/ISSUE_TEMPLATE/support_request.md vendored Normal file
View File

@ -0,0 +1,34 @@
---
name: Support
about: Support requests for MISP
labels: support, needs triage
---
## Please consider the following notes
- Critical security bugs can be reported, preferably PGP encrypted, by mail to: info@circl.lu
- Bug reports and feature requests can be filed as an issue on github: https://github.com/MISP/MISP/issues
- For interactive support please join the Gitter chat on: https://gitter.im/MISP/MISP
The official documentation of MISP can be found here: https://www.circl.lu/doc/misp/
We also periodically do user/admin/developer trainings and have our training material online: https://www.circl.lu/services/misp-training-materials/
Nevertheless you can of course file a Support request as an issue. Please be as precise as possible and fill the template as detailed as possible too.
Please remove this text until the line below. Thanks a lot.
--------8<------
### Work environment
| Questions | Answers
|---------------------------|--------------------
| Type of issue | Support
| OS version (server) | Debian, Ubuntu, CentOS, RedHat, ...
| OS version (client) | MacOS, Win10, Ubuntu, ...
| PHP version | 5.4, 5.5, 5.6, 7.0, 7.1... (if relevant)
| MISP version / git hash | 2.4.XX, hash of the commit
| Browser | If applicable
### Support Questions
### Logs, screenshots, configuration dump, ...

38
.travis.yml
View File

@ -1,9 +1,9 @@
language: php
php:
- 7.1
- 7.2
- 7.3
- 7.4
- nightly
services:
@ -11,7 +11,6 @@ services:
sudo: required
dist: bionic
group: edge
addons:
mariadb: '10.2'
@ -95,14 +94,13 @@ install:
# Get authkey
- sudo usermod -a -G www-data $USER
- sudo -E su $USER -c 'app/Console/cake userInit -q | sudo tee ./key.txt'
- sudo -E su $USER -c 'app/Console/cake Admin updateDatabase'
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "MISP.python_bin" "/usr/bin/python3"'
- sudo -E su $USER -c 'app/Console/cake Admin runUpdates'
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "Session.autoRegenerate" 0'
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "Session.timeout" 600'
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "Session.cookieTimeout" 3600'
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "MISP.host_org_id" 1'
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "MISP.email" "info@admin.test"'
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "MISP.disable_emailing" true'
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "MISP.disable_emailing" false'
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "debug" true'
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "Plugin.CustomAuth_disable_logout" false'
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "MISP.redis_host" "127.0.0.1"'
@ -113,7 +111,7 @@ install:
- sudo -E su $USER -c 'app/Console/cake Admin updateTaxonomies'
- sudo -E su $USER -c 'app/Console/cake Admin updateWarningLists'
- sudo -E su $USER -c 'app/Console/cake Admin updateNoticeLists'
- sudo -E su $USER -c 'app/Console/cake Admin updateObjectTemplates "1337"'
- sudo -E su $USER -c 'app/Console/cake Admin updateObjectTemplates 1'
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "Plugin.ZeroMQ_enable" true'
- sudo -E su $USER -c 'app/Console/cake Live 1'
- sudo chmod 777 ./key.txt
@ -123,27 +121,18 @@ install:
- app/Console/worker/start.sh &
- sleep 10
# Dirty install python stuff
- virtualenv -p python3.6 ./venv
- sudo -E su $USER -c 'app/Console/cake Admin setSetting "MISP.python_bin" "$TRAVIS_BUILD_DIR/venv/bin/python"'
- . ./venv/bin/activate
- pushd cti-python-stix2
- sudo pip3 install .
- popd
- pushd app/files/scripts
- sudo -H -u $USER git clone https://github.com/CybOXProject/python-cybox.git
- sudo -H -u $USER git clone https://github.com/STIXProject/python-stix.git
- sudo -H -u $USER git clone https://github.com/MAECProject/python-maec.git
- sudo -H -u $USER git clone https://github.com/CybOXProject/mixbox.git
- pushd mixbox
- sudo pip3 install .
- popd
- pushd python-cybox
- sudo pip3 install .
- popd
- pushd python-stix
- sudo pip3 install .
- popd
- pushd python-maec
- sudo pip3 install .
- pip install .
- popd
- pushd PyMISP
- pip install .[fileobjects]
- popd
- pip install stix zmq redis plyara
- pip install -I antlr4-python3-runtime==4.7.2 # antlr4-python3-runtime==4.8 causes issues
- deactivate
before_script:
- curl http://misp.local
@ -164,7 +153,6 @@ script:
- git submodule update
- pipenv install -d
- pipenv run python tests/testlive_comprehensive.py
- pipenv run python tests/test.py
- pipenv run python tests/test_mispevent.py
- popd
- cp PyMISP/tests/keys.py PyMISP/examples/events/

112
INSTALL/INSTALL.sh
View File

@ -148,9 +148,9 @@ MISPvars () {
# sudo config to run $LUSER commands
if [[ "$(groups ${MISP_USER} |grep -o 'staff')" == "staff" ]]; then
SUDO_USER="sudo -H -u ${MISP_USER} -g staff"
SUDO_CMD="sudo -H -u ${MISP_USER} -g staff"
else
SUDO_USER="sudo -H -u ${MISP_USER}"
SUDO_CMD="sudo -H -u ${MISP_USER}"
fi
SUDO_WWW="sudo -H -u ${WWW_USER} "
@ -544,12 +544,12 @@ checkID () {
sudo adduser $MISP_USER $WWW_USER
fi
# FIXME: the below SUDO_USER check is a duplicate from global variables, try to have just one check
# FIXME: the below SUDO_CMD check is a duplicate from global variables, try to have just one check
# sudo config to run $LUSER commands
if [[ "$(groups ${MISP_USER} |grep -o 'staff')" == "staff" ]]; then
SUDO_USER="sudo -H -u ${MISP_USER} -g staff"
SUDO_CMD="sudo -H -u ${MISP_USER} -g staff"
else
SUDO_USER="sudo -H -u ${MISP_USER}"
SUDO_CMD="sudo -H -u ${MISP_USER}"
fi
}
@ -953,7 +953,8 @@ composer73 () {
# Update composer.phar
# If hash changes, check here: https://getcomposer.org/download/ and replace with the correct one
# Current Sum for: v1.8.3
SHA384_SUM='48e3236262b34d30969dca3c37281b3b4bbe3221bda826ac6a9a62d6444cdb0dcd0615698a5cbe587c3f0fe57a54d8f5'
SHA384_SUM="$(wget -q -O - https://composer.github.io/installer.sig)"
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
$SUDO_WWW php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
$SUDO_WWW php -r "if (hash_file('SHA384', 'composer-setup.php') === '$SHA384_SUM') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); exit(137); } echo PHP_EOL;"
checkFail "composer.phar checksum failed, please investigate manually. " $?
@ -985,8 +986,8 @@ genRCLOCAL () {
# Run PyMISP tests
runTests () {
echo "url = ${MISP_BASEURL}
key = ${AUTH_KEY}" |sudo tee ${PATH_TO_MISP}/PyMISP/tests/keys.py
echo "url = '${MISP_BASEURL}'
key = '${AUTH_KEY}'" |sudo tee ${PATH_TO_MISP}/PyMISP/tests/keys.py
sudo chown -R $WWW_USER:$WWW_USER $PATH_TO_MISP/PyMISP/
sudo -H -u $WWW_USER sh -c "cd $PATH_TO_MISP/PyMISP && git submodule foreach git pull origin master"
@ -1008,11 +1009,12 @@ nuke () {
# Final function to let the user know what happened
theEnd () {
space
echo "Admin (root) DB Password: $DBPASSWORD_ADMIN" |$SUDO_USER tee /home/${MISP_USER}/mysql.txt
echo "User (misp) DB Password: $DBPASSWORD_MISP" |$SUDO_USER tee -a /home/${MISP_USER}/mysql.txt
echo "Authkey: $AUTH_KEY" |$SUDO_USER tee -a /home/${MISP_USER}/MISP-authkey.txt
echo "Admin (root) DB Password: $DBPASSWORD_ADMIN" |$SUDO_CMD tee /home/${MISP_USER}/mysql.txt
echo "User (misp) DB Password: $DBPASSWORD_MISP" |$SUDO_CMD tee -a /home/${MISP_USER}/mysql.txt
echo "Authkey: $AUTH_KEY" |$SUDO_CMD tee -a /home/${MISP_USER}/MISP-authkey.txt
clear
# Commenting out, see: https://github.com/MISP/MISP/issues/5368
# clear -x
space
echo -e "${LBLUE}MISP${NC} Installed, access here: ${MISP_BASEURL}"
echo
@ -1224,7 +1226,7 @@ EOF
}
apacheConfig () {
debug "Generating Apache config"
debug "Generating Apache config, if this hangs, make sure you have enough entropy (install: haveged or wait)"
sudo cp ${PATH_TO_MISP}/INSTALL/apache.24.misp.ssl /etc/apache2/sites-available/misp-ssl.conf
if [[ ! -z ${MISP_BASEURL} ]] && [[ "$(echo $MISP_BASEURL|cut -f 1 -d :)" == "http" || "$(echo $MISP_BASEURL|cut -f 1 -d :)" == "https" ]]; then
@ -1293,6 +1295,8 @@ installCore () {
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install .
cd $PATH_TO_MISP/app/files/scripts/python-maec
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install .
# FIXME: Remove once stix-fixed
$SUDO_WWW $PATH_TO_MISP/venv/bin/pip install -I antlr4-python3-runtime==4.7.2
# install STIX2.0 library to support STIX 2.0 export:
cd ${PATH_TO_MISP}/cti-python-stix2
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install .
@ -1308,7 +1312,7 @@ installCore () {
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install https://github.com/lief-project/packages/raw/lief-master-latest/pylief-0.9.0.dev.zip
# install zmq needed by mispzmq
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install zmq
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install zmq redis
# install python-magic
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install python-magic
@ -1395,7 +1399,7 @@ coreCAKE () {
$SUDO_WWW $RUN_PHP -- $CAKE userInit -q
# This makes sure all Database upgrades are done, without logging in.
$SUDO_WWW $RUN_PHP -- $CAKE Admin updateDatabase
$SUDO_WWW $RUN_PHP -- $CAKE Admin runUpdates
# The default install is Python >=3.6 in a virtualenv, setting accordingly
$SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.python_bin" "${PATH_TO_MISP}/venv/bin/python"
@ -1607,7 +1611,7 @@ mispmodules () {
cd /usr/local/src/
## TODO: checkUsrLocalSrc in main doc
debug "Cloning misp-modules"
$SUDO_USER git clone https://github.com/MISP/misp-modules.git
$SUDO_CMD git clone https://github.com/MISP/misp-modules.git
cd misp-modules
# some misp-modules dependencies
sudo apt install libpq5 libjpeg-dev tesseract-ocr libpoppler-cpp-dev imagemagick libopencv-dev zbar-tools libzbar0 libzbar-dev libfuzzy-dev -y
@ -1762,39 +1766,39 @@ mail2misp () {
debug "Installing Mail2${LBLUE}MISP${NC}"
cd /usr/local/src/
sudo apt-get install cmake libcaca-dev liblua5.3-dev -y
$SUDO_USER git clone https://github.com/MISP/mail_to_misp.git
$SUDO_USER git clone git://github.com/stricaud/faup.git faup
$SUDO_USER git clone git://github.com/stricaud/gtcaca.git gtcaca
$SUDO_CMD git clone https://github.com/MISP/mail_to_misp.git
$SUDO_CMD git clone git://github.com/stricaud/faup.git faup
$SUDO_CMD git clone git://github.com/stricaud/gtcaca.git gtcaca
sudo chown -R ${MISP_USER}:${MISP_USER} faup mail_to_misp gtcaca
cd gtcaca
$SUDO_USER mkdir -p build
$SUDO_CMD mkdir -p build
cd build
$SUDO_USER cmake .. && $SUDO_USER make
$SUDO_CMD cmake .. && $SUDO_CMD make
sudo make install
cd ../../faup
$SUDO_USER mkdir -p build
$SUDO_CMD mkdir -p build
cd build
$SUDO_USER cmake .. && $SUDO_USER make
$SUDO_CMD cmake .. && $SUDO_CMD make
sudo make install
sudo ldconfig
cd ../../mail_to_misp
$SUDO_USER virtualenv -p python3 venv
$SUDO_USER ./venv/bin/pip install https://github.com/lief-project/packages/raw/lief-master-latest/pylief-0.9.0.dev.zip
$SUDO_USER ./venv/bin/pip install -r requirements.txt
$SUDO_USER cp mail_to_misp_config.py-example mail_to_misp_config.py
$SUDO_CMD virtualenv -p python3 venv
$SUDO_CMD ./venv/bin/pip install https://github.com/lief-project/packages/raw/lief-master-latest/pylief-0.9.0.dev.zip
$SUDO_CMD ./venv/bin/pip install -r requirements.txt
$SUDO_CMD cp mail_to_misp_config.py-example mail_to_misp_config.py
##$SUDO cp mail_to_misp_config.py-example mail_to_misp_config.py
$SUDO_USER sed -i "s/^misp_url\ =\ 'YOUR_MISP_URL'/misp_url\ =\ 'https:\/\/localhost'/g" /usr/local/src/mail_to_misp/mail_to_misp_config.py
$SUDO_USER sed -i "s/^misp_key\ =\ 'YOUR_KEY_HERE'/misp_key\ =\ '${AUTH_KEY}'/g" /usr/local/src/mail_to_misp/mail_to_misp_config.py
$SUDO_CMD sed -i "s/^misp_url\ =\ 'YOUR_MISP_URL'/misp_url\ =\ 'https:\/\/localhost'/g" /usr/local/src/mail_to_misp/mail_to_misp_config.py
$SUDO_CMD sed -i "s/^misp_key\ =\ 'YOUR_KEY_HERE'/misp_key\ =\ '${AUTH_KEY}'/g" /usr/local/src/mail_to_misp/mail_to_misp_config.py
}
ssdeep () {
debug "Install ssdeep 2.14.1"
cd /usr/local/src
$SUDO_USER wget https://github.com/ssdeep-project/ssdeep/releases/download/release-2.14.1/ssdeep-2.14.1.tar.gz
$SUDO_USER tar zxvf ssdeep-2.14.1.tar.gz
$SUDO_CMD wget https://github.com/ssdeep-project/ssdeep/releases/download/release-2.14.1/ssdeep-2.14.1.tar.gz
$SUDO_CMD tar zxvf ssdeep-2.14.1.tar.gz
cd ssdeep-2.14.1
$SUDO_USER ./configure --datadir=/usr --prefix=/usr --localstatedir=/var --sysconfdir=/etc
$SUDO_USER make
$SUDO_CMD ./configure --datadir=/usr --prefix=/usr --localstatedir=/var --sysconfdir=/etc
$SUDO_CMD make
sudo make install
#installing ssdeep_php
@ -1807,8 +1811,10 @@ ssdeep () {
sudo service apache2 restart
}
# viper-web is broken ATM
# Main Viper install function
viper () {
export PATH=$PATH:/home/misp/.local/bin
debug "Installing Viper dependencies"
cd /usr/local/src/
sudo apt-get install \
@ -1820,30 +1826,26 @@ viper () {
fi
fi
echo "Cloning Viper"
$SUDO_USER git clone https://github.com/viper-framework/viper.git
$SUDO_CMD git clone https://github.com/viper-framework/viper.git
$SUDO_CMD git clone https://github.com/viper-framework/viper-web.git
sudo chown -R $MISP_USER:$MISP_USER viper
sudo chown -R $MISP_USER:$MISP_USER viper-web
cd viper
echo "Creating virtualenv"
$SUDO_USER virtualenv -p python3 venv
$SUDO_CMD virtualenv -p python3 venv
echo "Submodule update"
# TODO: Check for current user install permissions
$SUDO_USER git submodule update --init --recursive
##$SUDO git submodule update --init --recursive
echo "Pip install deps"
$SUDO_USER ./venv/bin/pip install SQLAlchemy PrettyTable python-magic
echo "pip install scrapy"
$SUDO_USER ./venv/bin/pip install scrapy
echo "install lief"
$SUDO_USER ./venv/bin/pip install https://github.com/lief-project/packages/raw/lief-master-latest/pylief-0.9.0.dev.zip
echo "pip install reqs"
$SUDO_USER ./venv/bin/pip install -r requirements.txt
$SUDO_USER sed -i '1 s/^.*$/\#!\/usr\/local\/src\/viper\/venv\/bin\/python/' viper-cli
$SUDO_USER sed -i '1 s/^.*$/\#!\/usr\/local\/src\/viper\/venv\/bin\/python/' viper-web
echo "Launching viper-cli"
$SUDO_USER /usr/local/src/viper/viper-cli -h > /dev/null
$SUDO_CMD git submodule update --init --recursive
echo "pip install deps"
$SUDO_CMD ./venv/bin/pip install pefile olefile jbxapi Crypto pypdns pypssl r2pipe pdftools virustotal-api SQLAlchemy PrettyTable python-magic scrapy https://github.com/lief-project/packages/raw/lief-master-latest/pylief-0.9.0.dev.zip
$SUDO_CMD ./venv/bin/pip install .
echo 'update-modules' |/usr/local/src/viper/venv/bin/viper
cd /usr/local/src/viper-web
$SUDO_CMD sed -i '1 s/^.*$/\#!\/usr\/local\/src\/viper\/venv\/bin\/python/' viper-web
$SUDO_CMD /usr/local/src/viper/venv/bin/pip install -r requirements.txt
echo "Launching viper-web"
$SUDO_USER /usr/local/src/viper/viper-web -p 8888 -H 0.0.0.0 &
echo 'PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/src/viper:/var/www/MISP/app/Console"' |sudo tee /etc/environment
$SUDO_CMD /usr/local/src/viper-web/viper-web -p 8888 -H 0.0.0.0 &
echo 'PATH="/home/misp/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/src/viper:/var/www/MISP/app/Console"' |sudo tee /etc/environment
echo ". /etc/environment" >> /home/${MISP_USER}/.profile
# TODO: Perms, MISP_USER_HOME, nasty hack cuz Kali on R00t
@ -1854,8 +1856,8 @@ viper () {
fi
echo "Setting misp_url/misp_key"
$SUDO_USER sed -i "s/^misp_url\ =/misp_url\ =\ http:\/\/localhost/g" ${VIPER_HOME}/viper.conf
$SUDO_USER sed -i "s/^misp_key\ =/misp_key\ =\ $AUTH_KEY/g" ${VIPER_HOME}/viper.conf
$SUDO_CMD sed -i "s/^misp_url\ =/misp_url\ =\ http:\/\/localhost/g" ${VIPER_HOME}/viper.conf
$SUDO_CMD sed -i "s/^misp_key\ =/misp_key\ =\ $AUTH_KEY/g" ${VIPER_HOME}/viper.conf
# Reset admin password to: admin/Password1234
echo "Fixing admin.db with default password"
VIPER_COUNT=0
@ -2006,7 +2008,7 @@ installSupported () {
# TODO: Double check how the user is added and subsequently used during the install.
# TODO: Work on possibility to install as user X and install MISP for user Y
# TODO: Check if logout needed. (run SUDO_USER in installer)
# TODO: Check if logout needed. (run SUDO_CMD in installer)
# <snippet-begin add-user.sh>
# TODO: Double check how to properly handle postfix
# <snippet-begin postfix.sh>
@ -2230,6 +2232,7 @@ installMISPonKali () {
# install STIX2.0 library to support STIX 2.0 export
debug "Installing cti-python-stix2"
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install -I antlr4-python3-runtime==4.7.2 2> /dev/null > /dev/null
cd ${PATH_TO_MISP}/cti-python-stix2
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install -I . 2> /dev/null > /dev/null
@ -2464,6 +2467,7 @@ x86_64-debian-stretch
x86_64-debian-buster
x86_64-ubuntu-bionic
x86_64-kali-2019.2
x86_64-kali-2019.3
armv6l-raspbian-stretch
armv7l-raspbian-stretch
armv7l-debian-jessie

6
INSTALL/INSTALL.sh.sfv
View File

@ -1,5 +1,5 @@
; Generated by RHash v1.3.8 on 2019-09-19 at 11:45.46
; Generated by RHash v1.3.9 on 2020-01-21 at 17:22.04
; Written by Kravchenko Aleksey (Akademgorodok) - http://rhash.sf.net/
;
; 98996 11:45.46 2019-09-19 INSTALL.sh
INSTALL.sh 590EBA6FCA2E8F2B5044843DF22FEED524ED3C7E 2B3E6DC0191D5D3AFD685E6ACF3E02D15261770E8E17A21F882EC2F07E908D55 13EB5B9EF3AEE3C96558DE068A0210F938491322F0AAD10DB958D9DCFA93B5313044013084E56647586AFCF8DE07C7D1 B169AA7096FAAF21E8CB8B7B7C9296C4CDB25E4B540637C15C040722EE52A2A8D2C4E8122CCEC25AFAD99F9CEC065DE14899E7C1DD571DE2B6127F08C6B79229
; 99445 17:22.04 2020-01-21 INSTALL.sh
INSTALL.sh 6E1976E8429DE5E28AD7B4F286904F0D6AE278FE 0032F7224EF5CAC309A1E3EDC4D185D6014BAE6DCCC07FBD12528D7574C6167E 1EEEA5E9C5FC2DDF6198442E8B301CB30DEC443798BD597B9D6DA3C8FD5F1C77AB5EB0F44988876BBA7314B3BEC4C297 8C950F3C6A5A8D5CCCC298551555CDC7C631E2A93177DF7BEA2E5C0A80D651C2D459A4328832BE96CDEB874C3CC2214FAB9DA0FA1B97DD642E7488ADB558E748

2
INSTALL/INSTALL.sh.sha1
View File

@ -1 +1 @@
590eba6fca2e8f2b5044843df22feed524ed3c7e INSTALL.sh
6e1976e8429de5e28ad7b4f286904f0d6ae278fe INSTALL.sh

2
INSTALL/INSTALL.sh.sha256
View File

@ -1 +1 @@
2b3e6dc0191d5d3afd685e6acf3e02d15261770e8e17a21f882ec2f07e908d55 INSTALL.sh
0032f7224ef5cac309a1e3edc4d185d6014bae6dccc07fbd12528d7574c6167e INSTALL.sh

2
INSTALL/INSTALL.sh.sha384
View File

@ -1 +1 @@
13eb5b9ef3aee3c96558de068a0210f938491322f0aad10db958d9dcfa93b5313044013084e56647586afcf8de07c7d1 INSTALL.sh
1eeea5e9c5fc2ddf6198442e8b301cb30dec443798bd597b9d6da3c8fd5f1c77ab5eb0f44988876bba7314b3bec4c297 INSTALL.sh

2
INSTALL/INSTALL.sh.sha512
View File

@ -1 +1 @@
b169aa7096faaf21e8cb8b7b7c9296c4cdb25e4b540637c15c040722ee52a2a8d2c4e8122ccec25afad99f9cec065de14899e7c1dd571de2b6127f08c6b79229 INSTALL.sh
8c950f3c6a5a8d5cccc298551555cdc7c631e2a93177df7bea2e5c0a80d651c2d459a4328832be96cdeb874c3cc2214fab9da0fa1b97dd642e7488adb558e748 INSTALL.sh

4
INSTALL/INSTALL.tpl.sh
View File

@ -220,7 +220,7 @@ installSupported () {
# TODO: Double check how the user is added and subsequently used during the install.
# TODO: Work on possibility to install as user X and install MISP for user Y
# TODO: Check if logout needed. (run SUDO_USER in installer)
# TODO: Check if logout needed. (run SUDO_CMD in installer)
# <snippet-begin add-user.sh>
# TODO: Double check how to properly handle postfix
# <snippet-begin postfix.sh>
@ -444,6 +444,7 @@ installMISPonKali () {
# install STIX2.0 library to support STIX 2.0 export
debug "Installing cti-python-stix2"
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install -I antlr4-python3-runtime==4.7.2 2> /dev/null > /dev/null
cd ${PATH_TO_MISP}/cti-python-stix2
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install -I . 2> /dev/null > /dev/null
@ -678,6 +679,7 @@ x86_64-debian-stretch
x86_64-debian-buster
x86_64-ubuntu-bionic
x86_64-kali-2019.2
x86_64-kali-2019.3
armv6l-raspbian-stretch
armv7l-raspbian-stretch
armv7l-debian-jessie

4
INSTALL/MYSQL.sql
View File

@ -199,6 +199,7 @@ CREATE TABLE IF NOT EXISTS `events` (
`locked` tinyint(1) NOT NULL DEFAULT 0,
`threat_level_id` int(11) NOT NULL,
`publish_timestamp` int(11) NOT NULL DEFAULT 0,
`sighting_timestamp` int(11) NOT NULL DEFAULT 0,
`disable_correlation` tinyint(1) NOT NULL DEFAULT 0,
`extends_uuid` varchar(40) COLLATE utf8_bin DEFAULT '',
PRIMARY KEY (`id`),
@ -821,6 +822,7 @@ CREATE TABLE IF NOT EXISTS `servers` (
`org_id` int(11) NOT NULL,
`push` tinyint(1) NOT NULL,
`pull` tinyint(1) NOT NULL,
`push_sightings` tinyint(1) NOT NULL DEFAULT 0,
`lastpulledid` int(11) DEFAULT NULL,
`lastpushedid` int(11) DEFAULT NULL,
`organization` varchar(10) COLLATE utf8_bin DEFAULT NULL,
@ -1358,7 +1360,7 @@ INSERT INTO `admin_settings` (`id`, `setting`, `value`) VALUES
INSERT INTO `feeds` (`id`, `provider`, `name`, `url`, `distribution`, `default`, `enabled`) VALUES
(1, 'CIRCL', 'CIRCL OSINT Feed', 'https://www.circl.lu/doc/misp/feed-osint', 3, 1, 0),
(2, 'Botvrij.eu', 'The Botvrij.eu Data', 'http://www.botvrij.eu/data/feed-osint', 3, 1, 0);
(2, 'Botvrij.eu', 'The Botvrij.eu Data', 'https://www.botvrij.eu/data/feed-osint', 3, 1, 0);
INSERT INTO `regexp` (`id`, `regexp`, `replacement`, `type`) VALUES
(1, '/.:.ProgramData./i', '%ALLUSERSPROFILE%\\\\', 'ALL'),

6
INSTALL/POSTGRESQL-data-initial.sql
View File

@ -100,7 +100,7 @@ COPY public.event_tags (id, event_id, tag_id) FROM stdin;
-- Data for Name: events; Type: TABLE DATA; Schema: public; Owner: -
--
COPY public.events (id, org_id, date, info, user_id, uuid, published, analysis, attribute_count, orgc_id, "timestamp", distribution, sharing_group_id, proposal_email_lock, locked, threat_level_id, publish_timestamp, disable_correlation, extends_uuid) FROM stdin;
COPY public.events (id, org_id, date, info, user_id, uuid, published, analysis, attribute_count, orgc_id, "timestamp", distribution, sharing_group_id, proposal_email_lock, locked, threat_level_id, publish_timestamp, sighting_timestamp, disable_correlation, extends_uuid) FROM stdin;
\.
@ -118,7 +118,7 @@ COPY public.favourite_tags (id, tag_id, user_id) FROM stdin;
COPY public.feeds (id, name, provider, url, rules, enabled, distribution, sharing_group_id, tag_id, "default", source_format, fixed_event, delta_merge, event_id, publish, override_ids, settings, input_source, delete_local_file, lookup_visible, headers, caching_enabled) FROM stdin;
1 CIRCL OSINT Feed CIRCL https://www.circl.lu/doc/misp/feed-osint \N f 3 0 0 t misp f f 0 f f \N network f f \N f
2 The Botvrij.eu Data Botvrij.eu http://www.botvrij.eu/data/feed-osint \N f 3 0 0 t misp f f 0 f f \N network f f \N f
2 The Botvrij.eu Data Botvrij.eu https://www.botvrij.eu/data/feed-osint \N f 3 0 0 t misp f f 0 f f \N network f f \N f
\.
@ -323,7 +323,7 @@ COPY public.roles (id, name, created, modified, perm_add, perm_modify, perm_modi
-- Data for Name: servers; Type: TABLE DATA; Schema: public; Owner: -
--
COPY public.servers (id, name, url, authkey, org_id, push, pull, lastpulledid, lastpushedid, organization, remote_org_id, publish_without_email, unpublish_event, self_signed, pull_rules, push_rules, cert_file, client_cert_file, internal) FROM stdin;
COPY public.servers (id, name, url, authkey, org_id, push, pull, push_sightings, lastpulledid, lastpushedid, organization, remote_org_id, publish_without_email, unpublish_event, self_signed, pull_rules, push_rules, cert_file, client_cert_file, internal) FROM stdin;
\.

2
INSTALL/POSTGRESQL-structure.sql
View File

@ -335,6 +335,7 @@ CREATE TABLE public.events (
locked boolean DEFAULT false NOT NULL,
threat_level_id bigint NOT NULL,
publish_timestamp bigint DEFAULT '0'::bigint NOT NULL,
sighting_timestamp bigint DEFAULT '0'::bigint NOT NULL,
disable_correlation boolean DEFAULT false NOT NULL,
extends_uuid character varying(40) DEFAULT ''::character varying
);
@ -1171,6 +1172,7 @@ CREATE TABLE public.servers (
org_id bigint NOT NULL,
push boolean NOT NULL,
pull boolean NOT NULL,
push_sightings boolean DEFAULT false NOT NULL,
lastpulledid bigint,
lastpushedid bigint,
organization character varying(10),

12
INSTALL/misplogrotate.te
View File

@ -1,8 +1,16 @@
module misplogrotate 1.0;
module misplogrotate 1.2;
require {
type httpd_t;
type logrotate_t;
type httpd_log_t;
type httpd_sys_script_t;
type httpd_sys_content_t;
class dir { ioctl read getattr lock search open };
type httpd_sys_rw_content_t;
class dir { ioctl read getattr lock search open remove_name };
class file { unlink write };
}
#============= logrotate_t ==============
allow logrotate_t httpd_sys_content_t:dir { ioctl read getattr lock search open };
allow logrotate_t httpd_sys_rw_content_t:dir { ioctl read getattr lock search open };
allow httpd_t httpd_log_t:dir remove_name;
allow { httpd_t httpd_sys_script_t } httpd_log_t:file { unlink write };

BIN
INSTALL/screenshots/sh-misp-attack.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 218 KiB

BIN
INSTALL/screenshots/sh-misp-decaying-of-indicators.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 160 KiB

BIN
INSTALL/screenshots/sh-misp-eventgraph.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 248 KiB

BIN
INSTALL/screenshots/sh-misp-threat-actor-attack.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 215 KiB

2
PyMISP

@ -1 +1 @@
Subproject commit f73571f30d39db8f80b5bc1c4a22c8eb9f54fcd4
Subproject commit eabc6481d039940ad30b7342a477ccf56c878419

2
VERSION.json
View File

@ -1 +1 @@
{"major":2, "minor":4, "hotfix":117}
{"major":2, "minor":4, "hotfix":120}

5
app/Config/config.default.php
View File

@ -17,7 +17,7 @@ $config = array(
'org' => 'ORGNAME',
'showorg' => true,
'threatlevel_in_email_subject' => true,
'email_subject_TLP_string' => 'TLP Amber',
'email_subject_TLP_string' => 'tlp:amber',
'email_subject_tag' => 'tlp',
'email_subject_include_tag_name' => true,
'background_jobs' => true,
@ -110,7 +110,6 @@ $config = array(
'ApacheShibbAuth' => // Configuration for shibboleth authentication
array(
'apacheEnv' => 'REMOTE_USER', // If proxy variable = HTTP_REMOTE_USER
'ssoAuth' => 'AUTH_TYPE',
'MailTag' => 'EMAIL_TAG',
'OrgTag' => 'FEDERATION_TAG',
'GroupTag' => 'GROUP_TAG',
@ -141,7 +140,7 @@ $config = array(
/*
'ApacheSecureAuth' => // Configuration for kerberos authentication
array(
'apacheEnv' => 'REMOTE_USER', // If proxy variable = HTTP_REMOTE_USER
'apacheEnv' => 'REMOTE_USER', // If proxy variable = HTTP_REMOTE_USER, If BasicAuth ldap = PHP_AUTH_USER
'ldapServer' => 'ldap://example.com', // FQDN or IP
'ldapProtocol' => 3,
'ldapNetworkTimeout' => -1, // use -1 for unlimited network timeout

73
app/Console/Command/AdminShell.php
View File

@ -159,7 +159,7 @@ class AdminShell extends AppShell
}
}
public function updateWarningLists()
public function updateWarningLists()
{
$result = $this->Warninglist->update();
$success = count($result['success']);
@ -182,13 +182,7 @@ class AdminShell extends AppShell
echo 'Usage: ' . APP . '/cake ' . 'Admin updateObjectTemplates [user_id]' . PHP_EOL;
} else {
$userId = $this->args[0];
$user = $this->User->find('first', array(
'recursive' => -1,
'conditions' => array(
'User.id' => $userId,
),
'fields' => array('User.id', 'User.org_id')
));
$user = $this->User->getAuthUser($userId);
# If the user_id passed does not exist, do a global update.
if (empty($user)) {
echo 'User with ID: ' . $userId . ' not found' . PHP_EOL;
@ -313,49 +307,18 @@ class AdminShell extends AppShell
}
}
public function updateDatabase() {
public function runUpdates() {
$whoami = exec('whoami');
if ($whoami === 'httpd' || $whoami === 'www-data' || $whoami === 'apache') {
if ($whoami === 'httpd' || $whoami === 'www-data' || $whoami === 'apache' || $whoami === 'wwwrun' || $whoami === 'travis') {
echo 'Executing all updates to bring the database up to date with the current version.' . PHP_EOL;
$this->Server->runUpdates(true);
$processId = $this->args[0];
$this->Server->runUpdates(true, false, $processId);
echo 'All updates completed.' . PHP_EOL;
} else {
die('This OS user is not allowed to run this command.'. PHP_EOL. 'Run it under `www-data` or `httpd`.' . PHP_EOL . 'You tried to run this command as: ' . $whoami . PHP_EOL);
die('This OS user is not allowed to run this command.'. PHP_EOL. 'Run it under `www-data` or `httpd` or `apache` or `wwwrun`.' . PHP_EOL . 'You tried to run this command as: ' . $whoami . PHP_EOL);
}
}
public function updateApp() {
$whoami = exec('whoami');
if ($whoami === 'httpd' || $whoami === 'www-data' || $whoami === 'apache') {
$command = $this->args[0];
if (!empty($this->args[1])) {
$processId = $this->args[1];
$job = $this->Job->read(null, $processId);
} else { // create worker
$this->Job->create();
$job_data = array(
'worker' => 'prio',
'job_type' => 'update_app',
'job_input' => 'command: ' . $command,
'status' => 0,
'retries' => 0,
'org_id' => '',
'org' => '',
'message' => 'Updating.',
);
$this->Job->save($job_data);
$job = $this->Job->read(null, $this->Job->id);
}
$result = $this->Server->updateDatabase($command, false);
$job['Job']['progress'] = 100;
$job['Job']['message'] = 'Update done';
$this->Job->save($job);
} else {
die('This OS user is not allowed to run this command.' . PHP_EOL . 'Run it under `www-data` or `httpd`.' . PHP_EOL . 'You tried to run this command as: ' . $whoami . PHP_EOL);
}
}
public function getAuthkey() {
if (empty($this->args[0])) {
echo 'Invalid parameters. Usage: ' . APP . 'Console/cake Admin getAuthkey [user_email]' . PHP_EOL;
@ -550,4 +513,26 @@ class AdminShell extends AppShell
}
}
}
public function dumpCurrentDatabaseSchema()
{
$dbActualSchema = $this->Server->getActualDBSchema();
$dbVersion = $this->AdminSetting->find('first', array(
'conditions' => array('setting' => 'db_version')
));
if (!empty($dbVersion) && !empty($dbActualSchema['schema'])) {
$dbVersion = $dbVersion['AdminSetting']['value'];
$data = array(
'schema' => $dbActualSchema['schema'],
'indexes' => $dbActualSchema['indexes'],
'db_version' => $dbVersion
);
$file = new File(ROOT . DS . 'db_schema.json', true);
$file->write(json_encode($data, JSON_PRETTY_PRINT));
$file->close();
echo __("> Database schema dumped on disk") . PHP_EOL;
} else {
echo __("Something went wrong. Could not find the existing db version or fetch the current database schema.") . PHP_EOL;
}
}
}

22
app/Console/Command/EventShell.php
View File

@ -508,6 +508,28 @@ class EventShell extends AppShell
$log->createLogEntry($user, 'publish', 'Event', $id, 'Event (' . $id . '): published.', 'published () => (1)');
}
public function publish_sightings() {
$id = $this->args[0];
$passAlong = $this->args[1];
$jobId = $this->args[2];
$userId = $this->args[3];
$user = $this->User->getAuthUser($userId);
$job = $this->Job->read(null, $jobId);
$this->Event->Behaviors->unload('SysLogLogable.SysLogLogable');
$result = $this->Event->publish_sightings($id, $passAlong);
$job['Job']['progress'] = 100;
$job['Job']['date_modified'] = date("Y-m-d H:i:s");
if ($result) {
$job['Job']['message'] = 'Sightings published.';
} else {
$job['Job']['message'] = 'Sightings published, but the upload to other instances may have failed.';
}
$this->Job->save($job);
$log = ClassRegistry::init('Log');
$log->create();
$log->createLogEntry($user, 'publish_sightings', 'Event', $id, 'Sightings for event (' . $id . '): published.', 'publish_sightings updated');
}
public function enrichment() {
if (empty($this->args[0]) || empty($this->args[1]) || empty($this->args[2])) {
die('Usage: ' . $this->Server->command_line_functions['enrichment'] . PHP_EOL);

37
app/Console/Command/ServerShell.php
View File

@ -6,12 +6,41 @@ class ServerShell extends AppShell
{
public $uses = array('Server', 'Task', 'Job', 'User', 'Feed');
public function listServers() {
$res = ['servers'=>[]];
$servers = $this->Server->find('all', [
'fields' => ['Server.id', 'Server.name', 'Server.url'],
'recursive' => 0
]);
foreach ($servers as $server)
$res['servers'][] = $server['Server'];
echo json_encode($res) . PHP_EOL;
}
public function test() {
if (empty($this->args[0])) {
die('Usage: ' . $this->Server->command_line_functions['console_automation_tasks']['data']['Test'] . PHP_EOL);
}
$serverId = intval($this->args[0]);
$res = @$this->Server->runConnectionTest($serverId);
if (!empty($res['message']))
$res['message'] = json_decode($res['message']);
echo json_encode($res) . PHP_EOL;
}
public function pull() {
if (empty($this->args[0]) || empty($this->args[1])) {
die('Usage: ' . $this->Server->command_line_functions['console_automation_tasks']['data']['pull'] . PHP_EOL);
}
$userId = $this->args[0];
$user = $this->User->getAuthUser($userId);
if (empty($user)) {
die('User ID do not match an existing user.' . PHP_EOL);
}
if (empty($this->args[1])) die();
$serverId = $this->args[1];
if (!empty($this->args[2])) {
@ -46,7 +75,7 @@ class ServerShell extends AppShell
'status' => 4
));
if (is_array($result)) {
$message = sprintf(__('Pull completed. %s events pulled, %s events could not be pulled, %s proposals pulled.', count($result[0]), count($result[1]), $result[2]));
$message = sprintf(__('Pull completed. %s events pulled, %s events could not be pulled, %s proposals pulled, %s sightings pulled.', count($result[0]), count($result[1]), $result[2], $result[3]));
} else {
$message = sprintf(__('ERROR: %s'), $result);
}
@ -104,7 +133,7 @@ class ServerShell extends AppShell
public function fetchFeed() {
if (empty($this->args[0]) || empty($this->args[1])) {
die('Usage: ' . $this->Server->command_line_functions['console_automation_tasks']['data']['fetchFeed'] . PHP_EOL);
die('Usage: ' . $this->Server->command_line_functions['console_automation_tasks']['data']['Fetch feeds as local data'] . PHP_EOL);
}
$userId = $this->args[0];
$user = $this->User->getAuthUser($userId);
@ -195,7 +224,7 @@ class ServerShell extends AppShell
$data = array(
'worker' => 'default',
'job_type' => 'cache_servers',
'job_input' => 'Server: ' . $id,
'job_input' => 'Server: ' . $scopeid,
'status' => 0,
'retries' => 0,
'org' => $user['Organisation']['name'],
@ -230,7 +259,7 @@ class ServerShell extends AppShell
public function cacheFeed() {
if (empty($this->args[0]) || empty($this->args[1])) {
die('Usage: ' . $this->Server->command_line_functions['console_automation_tasks']['data']['cacheFeed'] . PHP_EOL);
die('Usage: ' . $this->Server->command_line_functions['console_automation_tasks']['data']['Cache feeds for quick lookups'] . PHP_EOL);
}
$userId = $this->args[0];
$user = $this->User->getAuthUser($userId);

1
app/Console/dumpDBSchema.sh Normal file
View File

@ -0,0 +1 @@
./cake admin dumpCurrentDatabaseSchema

1
app/Console/worker/start.sh
View File

@ -13,6 +13,7 @@ cd "${0%/*}"
../cake CakeResque.CakeResque start --interval 5 --queue prio
../cake CakeResque.CakeResque start --interval 5 --queue cache
../cake CakeResque.CakeResque start --interval 5 --queue email
../cake CakeResque.CakeResque start --interval 5 --queue update
../cake CakeResque.CakeResque startscheduler --interval 5
exit 0

3
app/Console/worker/start_dynamic.sh
View File

@ -28,7 +28,7 @@ else
fi
if [[ "$ADVANCED" == "1" ]]; then
for worker in `echo cache default email prio scheduler`; do
for worker in `echo cache default email prio scheduler update`; do
workerStatus=$(../cake Admin getWorkers |tail -n +7 |jq -r ".$worker" |jq -r '.ok')
PIDcount=$(../cake admin getWorkers |tail -n +7 |jq -r ".$worker.workers" |grep pid | wc -l)
echo -n "$worker has $PIDcount PID(s)"
@ -51,6 +51,7 @@ else
../cake CakeResque.CakeResque start --interval 5 --queue prio
../cake CakeResque.CakeResque start --interval 5 --queue cache
../cake CakeResque.CakeResque start --interval 5 --queue email
../cake CakeResque.CakeResque start --interval 5 --queue update
../cake CakeResque.CakeResque startscheduler --interval 5
exit 0

160
app/Controller/AppController.php
View File

@ -46,10 +46,12 @@ class AppController extends Controller
public $helpers = array('Utility', 'OrgImg', 'FontAwesome', 'UserName');
private $__queryVersion = '90';
public $pyMispVersion = '2.4.117';
public $phpmin = '7.0';
public $phprec = '7.2';
private $__queryVersion = '97';
public $pyMispVersion = '2.4.120';
public $phpmin = '7.2';
public $phprec = '7.4';
public $pythonmin = '3.6';
public $pythonrec = '3.7';
public $isApiAuthed = false;
public $baseurl = '';
@ -62,6 +64,8 @@ class AppController extends Controller
'attributes' => array('text', 'downloadAttachment', 'returnAttributes', 'restSearch', 'rpz', 'bro'),
);
protected $_legacyParams = array();
public function __construct($id = false, $table = null, $ds = null)
{
parent::__construct($id, $table, $ds);
@ -90,7 +94,10 @@ class AppController extends Controller
'RestResponse',
'Flash',
'Toolbox',
'RateLimit'
'RateLimit',
'IndexFilter',
'Deprecation',
'RestSearch'
//,'DebugKit.Toolbar'
);
@ -123,7 +130,7 @@ class AppController extends Controller
}
if (!empty($this->params['named']['sql'])) {
$this->sql_dump = 1;
$this->sql_dump = intval($this->params['named']['sql']);
}
$this->_setupDatabaseConnection();
@ -164,10 +171,10 @@ class AppController extends Controller
} else {
$this->Auth->authenticate['Form']['userFields'] = $auth_user_fields;
}
$versionArray = $this->{$this->modelClass}->checkMISPVersion();
if (!empty($this->params['named']['disable_background_processing'])) {
Configure::write('MISP.background_jobs', 0);
}
$versionArray = $this->{$this->modelClass}->checkMISPVersion();
$this->mispVersion = implode('.', array_values($versionArray));
$this->Security->blackHoleCallback = 'blackHole';
$this->_setupBaseurl();
@ -284,7 +291,9 @@ class AppController extends Controller
if ($this->Auth->user()) {
// update script
$this->{$this->modelClass}->runUpdates();
if ($this->Auth->user('Role')['perm_site_admin'] || (Configure::read('MISP.live') && !$this->_isRest())) {
$this->{$this->modelClass}->runUpdates();
}
$user = $this->Auth->user();
if (!isset($user['force_logout']) || $user['force_logout']) {
$this->loadModel('User');
@ -350,7 +359,7 @@ class AppController extends Controller
$this->Auth->logout();
throw new MethodNotAllowedException($message);//todo this should pb be removed?
} else {
$this->Flash->error('Warning: MISP is currently disabled for all users. Enable it in Server Settings (Administration -> Server Settings -> MISP tab -> live). An update might also be in progress, you can see the progress in ' , array('params' => array('url' => $this->baseurl . '/servers/updateProgress/', 'urlName' => __('Update Progress')), 'clear' => 1));
$this->Flash->error(__('Warning: MISP is currently disabled for all users. Enable it in Server Settings (Administration -> Server Settings -> MISP tab -> live). An update might also be in progress, you can see the progress in ') , array('params' => array('url' => $this->baseurl . '/servers/updateProgress/', 'urlName' => __('Update Progress')), 'clear' => 1));
}
}
if ($this->Session->check(AuthComponent::$sessionKey)) {
@ -394,10 +403,8 @@ class AppController extends Controller
// instead of using checkAction(), like we normally do from controllers when trying to find out about a permission flag, we can use getActions()
// getActions returns all the flags in a single SQL query
if ($this->Auth->user()) {
$versionArray = $this->{$this->modelClass}->checkMISPVersion();
$this->mispVersionFull = implode('.', array_values($versionArray));
$this->set('mispVersion', implode('.', array($versionArray['major'], $versionArray['minor'], 0)));
$this->set('mispVersionFull', $this->mispVersionFull);
$this->set('mispVersionFull', $this->mispVersion);
$role = $this->getActions();
$this->set('me', $this->Auth->user());
$this->set('isAdmin', $role['perm_admin']);
@ -474,6 +481,19 @@ class AppController extends Controller
if ($this->_isRest()) {
$this->__rateLimitCheck();
}
if ($this->modelClass !== 'CakeError') {
$deprecationWarnings = $this->Deprecation->checkDeprecation($this->request->params['controller'], $this->action, $this->{$this->modelClass}, $this->Auth->user('id'));
if ($deprecationWarnings) {
$deprecationWarnings = __('WARNING: This functionality is deprecated and will be removed in the near future. ') . $deprecationWarnings;
if ($this->_isRest()) {
$this->response->header('X-Deprecation-Warning', $deprecationWarnings);
$this->components['RestResponse']['deprecationWarnings'] = $deprecationWarnings;
} else {
$this->Flash->warning($deprecationWarnings);
}
}
}
$this->components['RestResponse']['sql_dump'] = $this->sql_dump;
}
private function __rateLimitCheck()
@ -506,10 +526,6 @@ class AppController extends Controller
public function afterFilter()
{
if (Configure::read('debug') > 1 && !empty($this->sql_dump) && $this->_isRest()) {
$this->Log = ClassRegistry::init('Log');
echo json_encode($this->Log->getDataSource()->getLog(false, false), JSON_PRETTY_PRINT);
}
if ($this->isApiAuthed && $this->_isRest()) {
$this->Session->destroy();
}
@ -695,9 +711,8 @@ class AppController extends Controller
}
// generic function to standardise on the collection of parameters. Accepts posted request objects, url params, named url params
protected function _harvestParameters($options, &$exception)
protected function _harvestParameters($options, &$exception, $data = array())
{
$data = array();
if (!empty($options['request']->is('post'))) {
if (empty($options['request']->data)) {
$exception = $this->RestResponse->throwException(
@ -708,12 +723,25 @@ class AppController extends Controller
return false;
} else {
if (isset($options['request']->data['request'])) {
$data = $options['request']->data['request'];
$data = array_merge($data, $options['request']->data['request']);
} else {
$data = $options['request']->data;
$data = array_merge($data, $options['request']->data);
}
}
}
/*
* If we simply capture ordered URL params with func_get_args(), reassociate them.
* We can easily detect this by having ordered_url_params passed as a list instead of a dict.
*/
if (isset($options['ordered_url_params'][0])) {
$temp = array();
foreach ($options['ordered_url_params'] as $k => $url_param) {
if (!empty($options['paramArray'][$k])) {
$temp[$options['paramArray'][$k]] = $url_param;
}
}
$options['ordered_url_params'] = $temp;
}
if (!empty($options['paramArray'])) {
foreach ($options['paramArray'] as $p) {
if (
@ -1091,4 +1119,96 @@ class AppController extends Controller
}
}
protected function _legacyAPIRemap($options = array())
{
$ordered_url_params = array();
foreach ($options['paramArray'] as $k => $param) {
if (isset($options['ordered_url_params'][$k])) {
$ordered_url_params[$param] = $options['ordered_url_params'][$k];
} else {
$ordered_url_params[$param] = false;
}
}
$filterData = array(
'request' => $options['request'],
'named_params' => $options['named_params'],
'paramArray' => $options['paramArray'],
'ordered_url_params' => $ordered_url_params
);
$exception = false;
$filters = $this->_harvestParameters($filterData, $exception);
if (!empty($options['injectedParams'])) {
foreach ($options['injectedParams'] as $injectedParam => $injectedValue) {
$filters[$injectedParam] = $injectedValue;
}
}
if (!empty($options['alias'])) {
foreach ($options['alias'] as $from => $to) {
if (!empty($filters[$from])) {
$filters[$to] = $filters[$from];
}
}
}
$this->_legacyParams = $filters;
return true;
}
public function restSearch()
{
$ordered_url_params = func_get_args();
if (empty($this->RestSearch->paramArray[$this->modelClass])) {
throw new NotFoundException(__('RestSearch is not implemented (yet) for this scope.'));
}
$scope = empty($this->scopeOverride) ? $this->modelClass : $this->scopeOverride;
if (!isset($this->$scope)) {
$this->loadModel($scope);
}
$filterData = array(
'request' => $this->request,
'named_params' => $this->params['named'],
'paramArray' => $this->RestSearch->paramArray[$scope],
'ordered_url_params' => func_get_args()
);
$exception = false;
$filters = $this->_harvestParameters($filterData, $exception, $this->_legacyParams);
if (empty($filters['returnFormat'])) {
$filters['returnFormat'] = 'json';
}
unset($filterData);
if ($filters === false) {
return $exception;
}
$list = array();
$key = empty($filters['key']) ? $filters['returnFormat'] : $filters['key'];
$user = $this->_getApiAuthUser($key, $exception);
if ($user === false) {
return $exception;
}
if (isset($filters['returnFormat'])) {
$returnFormat = $filters['returnFormat'];
} else {
$returnFormat = 'json';
}
if ($returnFormat === 'download') {
$returnFormat = 'json';
}
if ($returnFormat === 'stix' && $this->_isJson()) {
$returnFormat = 'stix-json';
}
$elementCounter = 0;
$renderView = false;
$final = $this->$scope->restSearch($user, $returnFormat, $filters, false, false, $elementCounter, $renderView);
if (!empty($renderView) && !empty($final)) {
$this->layout = false;
$final = json_decode($final, true);
foreach ($final as $key => $data) {
$this->set($key, $data);
}
$this->render('/Events/module_views/' . $renderView);
} else {
$responseType = $this->$scope->validFormats[$returnFormat][0];
$filename = $this->RestSearch->getFilename($filters, $scope, $responseType);
return $this->RestResponse->viewData($final, $responseType, false, true, $filename, array('X-Result-Count' => $elementCounter, 'X-Export-Module-Used' => $returnFormat, 'X-Response-Format' => $responseType));
}
}
}

412
app/Controller/AttributesController.php
View File

@ -239,13 +239,8 @@ class AttributesController extends AppController
$failed = 1;
$message = sprintf('Attributes saved, however, %s attributes could not be saved. Click %s for more info', count($fails), '$flashErrorMessage');
} else {
if (!empty($fails["attribute_0"])) {
$failed = 1;
$message = '0: ' . $v[0];
} else {
$failed = 1;
$message = 'Attribute could not be saved.';
}
$failed = 1;
$message = 'Attribute could not be saved.';
}
}
if (!empty($failKeys)) {
@ -267,8 +262,14 @@ class AttributesController extends AppController
$flashErrorMessage = implode('<br />', $flashErrorMessage);
$this->Session->write('flashErrorMessage', $flashErrorMessage);
}
if (empty($failed)) {
$this->Flash->success($message);
} else {
$this->Flash->error($message);
}
if ($this->request->is('ajax')) {
$this->autoRender = false;
$this->layout = false;
$errors = ($attributeCount > 1) ? $message : $this->Attribute->validationErrors;
if (!empty($successes)) {
return new CakeResponse(array('body'=> json_encode(array('saved' => true, 'success' => $message)),'status' => 200, 'type' => 'json'));
@ -276,11 +277,6 @@ class AttributesController extends AppController
return new CakeResponse(array('body'=> json_encode(array('saved' => false, 'errors' => $errors)),'status' => 200, 'type' => 'json'));
}
} else {
if (empty($failed)) {
$this->Flash->success($message);
} else {
$this->Flash->error($message);
}
if ($successes > 0) {
$this->redirect(array('controller' => 'events', 'action' => 'view', $eventId));
}
@ -311,25 +307,31 @@ class AttributesController extends AppController
$this->loadModel('SharingGroup');
$sgs = $this->SharingGroup->fetchAllAuthorised($this->Auth->user(), 'name', 1);
$this->set('sharingGroups', $sgs);
$info = array();
$initialDistribution = 5;
$configuredDistribution = Configure::check('MISP.default_attribute_distribution');
if ($configuredDistribution != null && $configuredDistribution != 'event') {
$initialDistribution = $configuredDistribution;
}
$this->set('initialDistribution', $initialDistribution);
$fieldDesc = array();
$distributionLevels = $this->Attribute->distributionLevels;
if (empty($sgs)) {
unset($distributionLevels[4]);
}
$this->set('distributionLevels', $distributionLevels);
foreach ($distributionLevels as $key => $value) {
$fieldDesc['distribution'][$key] = $this->Attribute->distributionDescriptions[$key]['formdesc'];
}
foreach ($this->Attribute->categoryDefinitions as $key => $value) {
$info['category'][$key] = array('key' => $key, 'desc' => isset($value['formdesc'])? $value['formdesc'] : $value['desc']);
$fieldDesc['category'][$key] = isset($value['formdesc']) ? $value['formdesc'] : $value['desc'];
}
foreach ($this->Attribute->typeDefinitions as $key => $value) {
$info['type'][$key] = array('key' => $key, 'desc' => isset($value['formdesc'])? $value['formdesc'] : $value['desc']);
}
foreach ($distributionLevels as $key => $value) {
$info['distribution'][$key] = array('key' => $value, 'desc' => $this->Attribute->distributionDescriptions[$key]['formdesc']);
$fieldDesc['type'][$key] = isset($value['formdesc']) ? $value['formdesc'] : $value['desc'];
}
$this->loadModel('Noticelist');
$notice_list_triggers = $this->Noticelist->getTriggerData();
$this->set('notice_list_triggers', json_encode($notice_list_triggers, true));
$this->set('info', $info);
$this->set('fieldDesc', $fieldDesc);
$this->set('typeDefinitions', $this->Attribute->typeDefinitions);
$this->set('categoryDefinitions', $this->Attribute->categoryDefinitions);
$this->set('published', $events['Event']['published']);
@ -862,7 +864,7 @@ class AttributesController extends AppController
$skipTimeCheck = true;
}
if ($skipTimeCheck || $this->request->data['Attribute']['timestamp'] > $existingAttribute['Attribute']['timestamp']) {
$recoverFields = array('value', 'to_ids', 'distribution', 'category', 'type', 'comment');
$recoverFields = array('value', 'to_ids', 'distribution', 'category', 'type', 'comment', 'first_seen', 'last_seen');
foreach ($recoverFields as $rF) {
if (!isset($this->request->data['Attribute'][$rF])) {
$this->request->data['Attribute'][$rF] = $existingAttribute['Attribute'][$rF];
@ -1062,7 +1064,7 @@ class AttributesController extends AppController
if (!$this->_isRest()) {
$this->Attribute->Event->insertLock($this->Auth->user(), $this->Attribute->data['Attribute']['event_id']);
}
$validFields = array('value', 'category', 'type', 'comment', 'to_ids', 'distribution');
$validFields = array('value', 'category', 'type', 'comment', 'to_ids', 'distribution', 'first_seen', 'last_seen');
$changed = false;
if (empty($this->request->data['Attribute'])) {
$this->request->data = array('Attribute' => $this->request->data);
@ -1096,6 +1098,9 @@ class AttributesController extends AppController
$event['Event']['timestamp'] = $date->getTimestamp();
$event['Event']['published'] = 0;
$this->Attribute->Event->save($event, array('fieldList' => array('published', 'timestamp', 'info')));
if ($attribute['Attribute']['object_id'] != 0) {
$this->Attribute->Object->updateTimestamp($attribute['Attribute']['object_id'], $date->getTimestamp());
}
$this->autoRender = false;
return new CakeResponse(array('body'=> json_encode(array('saved' => true, 'success' => 'Field updated.', 'check_publish' => true)), 'status'=>200, 'type' => 'json'));
} else {
@ -1627,25 +1632,6 @@ class AttributesController extends AppController
}
}
// Deletes this specific attribute from all remote servers
private function __deleteAttributeFromServers($uuid)
{
// get a list of the servers with push active
$this->loadModel('Server');
$servers = $this->Server->find('all', array('conditions' => array('push' => 1)));
// iterate over the servers and upload the attribute
if (empty($servers)) {
return;
}
App::uses('SyncTool', 'Tools');
foreach ($servers as $server) {
$syncTool = new SyncTool();
$HttpSocket = $syncTool->setupHttpSocket($server);
$this->Attribute->deleteAttributeFromServer($uuid, $server, $HttpSocket);
}
}
public function search($continue = false)
{
$this->set('attrDescriptions', $this->Attribute->fieldDescriptions);
@ -1656,7 +1642,7 @@ class AttributesController extends AppController
if (isset($this->request->data['Attribute'])) {
$this->request->data = $this->request->data['Attribute'];
}
$checkForEmpty = array('value', 'tags', 'uuid', 'org', 'type', 'category');
$checkForEmpty = array('value', 'tags', 'uuid', 'org', 'type', 'category', 'first_seen', 'last_seen');
foreach ($checkForEmpty as $field) {
if (empty($this->request->data[$field]) || $this->request->data[$field] === 'ALL') {
unset($this->request->data[$field]);
@ -1666,12 +1652,12 @@ class AttributesController extends AppController
unset($this->request->data['to_ids']);
$this->request->data['ignore'] = 1;
}
$paramArray = array('value' , 'type', 'category', 'org', 'tags', 'from', 'to', 'last', 'eventid', 'withAttachments', 'uuid', 'publish_timestamp', 'timestamp', 'enforceWarninglist', 'to_ids', 'deleted', 'includeEventUuid', 'event_timestamp', 'threat_level_id', 'includeEventTags');
$paramArray = array('value' , 'type', 'category', 'org', 'tags', 'from', 'to', 'last', 'eventid', 'withAttachments', 'uuid', 'publish_timestamp', 'timestamp', 'enforceWarninglist', 'to_ids', 'deleted', 'includeEventUuid', 'event_timestamp', 'threat_level_id', 'includeEventTags', 'first_seen', 'last_seen');
$filterData = array(
'request' => $this->request,
'named_params' => $this->params['named'],
'paramArray' => $paramArray,
'ordered_url_params' => compact($paramArray),
'ordered_url_params' => @compact($paramArray),
'additional_delimiters' => PHP_EOL
);
$exception = false;
@ -1765,17 +1751,12 @@ class AttributesController extends AppController
private function __searchUI($attributes)
{
$sightingsData = array();
$sgids = $this->Attribute->Event->cacheSgids($this->Auth->user(), true);
$this->Feed = ClassRegistry::init('Feed');
if (!empty($options['overrideLimit'])) {
$overrideLimit = true;
} else {
$overrideLimit = false;
}
$this->loadModel('GalaxyCluster');
$cluster_names = $this->GalaxyCluster->find('list', array('fields' => array('GalaxyCluster.tag_name'), 'group' => array('GalaxyCluster.tag_name', 'GalaxyCluster.id')));
$this->loadModel('Sighting');
$user = $this->Auth->user();
foreach ($attributes as $k => $attribute) {
$attributeId = $attribute['Attribute']['id'];
if ($this->Attribute->isImage($attribute['Attribute'])) {
if (extension_loaded('gd')) {
// if extension is loaded, the data is not passed to the view because it is asynchronously fetched
@ -1787,22 +1768,18 @@ class AttributesController extends AppController
}
$attributes[$k]['Attribute']['AttributeTag'] = $attributes[$k]['AttributeTag'];
$attributes[$k]['Attribute'] = $this->Attribute->Event->massageTags($attributes[$k]['Attribute'], 'Attribute');
$attributes[$k]['Attribute'] = $this->Attribute->Event->massageTags($attributes[$k]['Attribute'], 'Attribute', $excludeGalaxy = false, $cullGalaxyTags = true);
unset($attributes[$k]['AttributeTag']);
foreach ($attributes[$k]['Attribute']['AttributeTag'] as $k2 => $attributeTag) {
if (in_array($attributeTag['Tag']['name'], $cluster_names)) {
unset($attributes[$k]['Attribute']['AttributeTag'][$k2]);
}
}
$sightingsData = array_merge(
$sightingsData,
$this->Sighting->attachToEvent($attribute, $this->Auth->user(), $attributes[$k]['Attribute']['id'], $extraConditions = false)
$this->Sighting->attachToEvent($attribute, $user, $attributeId, $extraConditions = false)
);
$correlations = $this->Attribute->Event->getRelatedAttributes($this->Auth->user(), $attributes[$k]['Attribute']['id'], false, false, 'attribute');
$correlations = $this->Attribute->Event->getRelatedAttributes($user, $attributeId, false, false, 'attribute');
if (!empty($correlations)) {
$attributes[$k]['Attribute']['RelatedAttribute'] = $correlations[$attributes[$k]['Attribute']['id']];
$attributes[$k]['Attribute']['RelatedAttribute'] = $correlations[$attributeId];
}
$temp = $this->Feed->attachFeedCorrelations(array($attributes[$k]['Attribute']), $this->Auth->user, $attributes[$k]['Event'], $overrideLimit);
$temp = $this->Feed->attachFeedCorrelations(array($attributes[$k]['Attribute']), $user, $attributes[$k]['Event']);
if (!empty($temp)) {
$attributes[$k]['Attribute'] = $temp[0];
}
@ -1878,62 +1855,6 @@ class AttributesController extends AppController
$this->set('fails', $this->Attribute->checkComposites());
}
public function restSearch(
$returnFormat = false, $value = false, $type = false, $category = false, $org = false, $tags = false, $from = false,
$to = false, $last = false, $eventid = false, $withAttachments = false, $uuid = false, $publish_timestamp = false, $published = false,
$timestamp = false, $enforceWarninglist = false, $to_ids = false, $deleted = false, $includeEventUuid = false, $event_timestamp = false,
$threat_level_id = false
)
{
$paramArray = array(
'value' , 'type', 'category', 'org', 'tags', 'from', 'to', 'last', 'eventid', 'withAttachments', 'uuid', 'publish_timestamp',
'timestamp', 'enforceWarninglist', 'to_ids', 'deleted', 'includeEventUuid', 'event_timestamp', 'threat_level_id', 'includeEventTags',
'includeProposals', 'returnFormat', 'published', 'limit', 'page', 'requested_attributes', 'includeContext', 'headerless',
'includeWarninglistHits', 'attackGalaxy', 'object_relation', 'includeSightings', 'includeCorrelations', 'includeDecayScore',
'decayingModel', 'excludeDecayed', 'modelOverrides', 'includeFullModel', 'score'
);
$filterData = array(
'request' => $this->request,
'named_params' => $this->params['named'],
'paramArray' => $paramArray,
'ordered_url_params' => compact($paramArray)
);
$validFormats = $this->Attribute->validFormats;
$exception = false;
$filters = $this->_harvestParameters($filterData, $exception);
unset($filterData);
if ($filters === false) {
return $exception;
}
$list = array();
$user = $this->_getApiAuthUser($returnFormat, $exception);
if ($user === false) {
return $exception;
}
if (isset($filters['returnFormat'])) {
$returnFormat = $filters['returnFormat'];
} else {
$returnFormat = 'json';
}
if ($returnFormat === 'download') {
$returnFormat = 'json';
}
$elementCounter = 0;
$renderView = '';
$final = $this->Attribute->restSearch($user, $returnFormat, $filters, false, false, $elementCounter, $renderView);
if (!empty($renderView) && !empty($final)) {
$this->layout = false;
$final = json_decode($final, true);
foreach ($final as $key => $data) {
$this->set($key, $data);
}
$this->render('/Events/module_views/' . $renderView);
} else {
$responseType = $this->Attribute->validFormats[$returnFormat][0];
return $this->RestResponse->viewData($final, $responseType, false, true, false, array('X-Result-Count' => $elementCounter, 'X-Export-Module-Used' => $returnFormat, 'X-Response-Format' => $responseType));
}
}
// returns an XML with attributes that belong to an event. The type of attributes to be returned can be restricted by type using the 3rd parameter.
// Similar to the restSearch, this parameter can be chained with '&&' and negations are accepted too. For example filename&&!filename|md5 would return all filenames that don't have an md5
// The usage of returnAttributes is the following: [MISP-url]/attributes/returnAttributes/<API-key>/<type>/<signature flag>
@ -2079,165 +2000,50 @@ class AttributesController extends AppController
$this->__downloadAttachment($this->Attribute->data['Attribute']);
}
public function text($key='download', $type = 'all', $tags = false, $eventId = false, $allowNonIDS = false, $from = false, $to = false, $last = false, $enforceWarninglist = false, $allowNotPublished = false)
public function text()
{
$simpleFalse = array('eventId', 'allowNonIDS', 'tags', 'from', 'to', 'last', 'enforceWarninglist', 'allowNotPublished');
foreach ($simpleFalse as $sF) {
if (!is_array(${$sF}) && (${$sF} === 'null' || ${$sF} == '0' || ${$sF} === false || strtolower(${$sF}) === 'false')) {
${$sF} = false;
}
$this->_legacyAPIRemap(array(
'paramArray' => array(
'key', 'type', 'tags', 'eventId', 'allowNonIDS', 'from', 'to', 'last', 'enforceWarninglist', 'allowNotPublished'
),
'request' => $this->request,
'named_params' => $this->params['named'],
'ordered_url_params' => func_get_args(),
'injectedParams' => array(
'returnFormat' => 'text'
),
'alias' => array(
'eventId' => 'eventid'
)
));
if (!empty($this->_legacyParams['allowNonIDS'])) {
$this->_legacyParams['to_ids'] = [0,1];
}
if ($type === 'null' || $type === '0' || $type === 'false') {
$type = 'all';
if (!empty($this->_legacyParams['allowNotPublished'])) {
$this->_legacyParams['published'] = [0,1];
}
if ($this->request->is('post')) {
$params = array('type', 'tags', 'eventId', 'allowNonIDS', 'from', 'to', 'last', 'enforceWarninglist', 'allowNotPublished');
foreach ($params as $param) {
if (isset($this->request->data[$param])) {
${$param} = $this->request->data[$param];
}
}
if (!empty($this->_legacyParams['type']) && $this->_legacyParams['type'] === 'all') {
unset($this->_legacyParams['type']);
}
if ($from) {
$from = $this->Attribute->Event->dateFieldCheck($from);
}
if ($to) {
$to = $this->Attribute->Event->dateFieldCheck($to);
}
if ($last) {
$last = $this->Attribute->Event->resolveTimeDelta($last);
}
if ($key != 'download') {
// check if the key is valid -> search for users based on key
$user = $this->checkAuthUser($key);
if (!$user) {
throw new UnauthorizedException(__('This authentication key is not authorized to be used for exports. Contact your administrator.'));
}
} else {
if (!$this->Auth->user('id')) {
throw new UnauthorizedException(__('You have to be logged in to do that.'));
}
}
$this->response->type('txt'); // set the content type
$this->header('Content-Disposition: download; filename="misp.' . (is_array($type) ? 'multi' : $type) . '.txt"');
$this->layout = 'text/default';
$attributes = $this->Attribute->text($this->Auth->user(), $type, $tags, $eventId, $allowNonIDS, $from, $to, $last, $enforceWarninglist, $allowNotPublished);
$this->loadModel('Whitelist');
$attributes = $this->Whitelist->removeWhitelistedFromArray($attributes, true);
$this->set('attributes', $attributes);
$this->render('/Attributes/text');
return $this->restSearch();
}
public function rpz($key='download', $tags=false, $eventId=false, $from=false, $to=false, $policy=false, $walled_garden = false, $ns = false, $email = false, $serial = false, $refresh = false, $retry = false, $expiry = false, $minimum_ttl = false, $ttl = false, $enforceWarninglist = false, $ns_alt = false)
public function rpz()
{
// request handler for POSTed queries. If the request is a post, the parameters (apart from the key) will be ignored and replaced by the terms defined in the posted json or xml object.
// The correct format for both is a "request" root element, as shown by the examples below:
// For Json: {"request":{"policy": "Local-Data","walled_garden":"my.stop.page.net"}}
// For XML: <request><policy>Local-Data</policy><walled_garden>my.stop.page.net</walled_garden></request>
// the response type is used to determine the parsing method (xml/json)
if ($this->request->is('post')) {
if ($this->request->input('json_decode', true)) {
$data = $this->request->input('json_decode', true);
} else {
$data = $this->request->data;
}
if (empty($data)) {
throw new BadRequestException(__('Either specify the search terms in the url, or POST a json array / xml (with the root element being "request" and specify the correct headers based on content type).'));
}
$paramArray = array('eventId', 'tags', 'from', 'to', 'policy', 'walled_garden', 'ns', 'email', 'serial', 'refresh', 'retry', 'expiry', 'minimum_ttl', 'ttl', 'enforceWarninglist', 'ns_alt');
foreach ($paramArray as $p) {
if (isset($data['request'][$p])) {
${$p} = $data['request'][$p];
} else {
${$p} = false;
}
}
}
$simpleFalse = array('eventId', 'tags', 'from', 'to', 'policy', 'walled_garden', 'ns', 'email', 'serial', 'refresh', 'retry', 'expiry', 'minimum_ttl', 'ttl', 'enforceWarninglist', 'ns_alt');
foreach ($simpleFalse as $sF) {
if (!is_array(${$sF}) && (${$sF} === 'null' || ${$sF} == '0' || ${$sF} === false || strtolower(${$sF}) === 'false')) {
${$sF} = false;
}
}
if (!in_array($policy, array('NXDOMAIN', 'NODATA', 'DROP', 'Local-Data', 'PASSTHRU', 'TCP-only'))) {
$policy = false;
}
App::uses('RPZExport', 'Export');
$rpzExport = new RPZExport();
if ($policy) {
$policy = $rpzExport->getIdByPolicy($policy);
}
$this->loadModel('Server');
$rpzSettings = array();
$lookupData = array('policy', 'walled_garden', 'ns', 'ns_alt', 'email', 'serial', 'refresh', 'retry', 'expiry', 'minimum_ttl', 'ttl');
foreach ($lookupData as $v) {
if (${$v} !== false) {
$rpzSettings[$v] = ${$v};
} else {
$tempSetting = Configure::read('Plugin.RPZ_' . $v);
if (isset($tempSetting)) {
$rpzSettings[$v] = Configure::read('Plugin.RPZ_' . $v);
} else {
$rpzSettings[$v] = $this->Server->serverSettings['Plugin']['RPZ_' . $v]['value'];
}
}
}
if ($from) {
$from = $this->Attribute->Event->dateFieldCheck($from);
}
if ($to) {
$to = $this->Attribute->Event->dateFieldCheck($to);
}
if ($key != 'download') {
// check if the key is valid -> search for users based on key
$user = $this->checkAuthUser($key);
if (!$user) {
throw new UnauthorizedException(__('This authentication key is not authorized to be used for exports. Contact your administrator.'));
}
} else {
if (!$this->Auth->user('id')) {
throw new UnauthorizedException(__('You have to be logged in to do that.'));
}
}
if (false === $eventId || $eventId === null) {
$eventIds = $this->Attribute->Event->fetchEventIds($this->Auth->user(), false, false, false, true);
} elseif (is_numeric($eventId)) {
$eventIds = array($eventId);
} else {
throw new MethodNotAllowedException(__('Invalid event ID format.'));
}
$values = array();
foreach ($eventIds as $k => $eventId) {
$values = array_merge_recursive($values, $this->Attribute->rpz($this->Auth->user(), $tags, $eventId, $from, $to, $enforceWarninglist));
}
$this->response->type('txt'); // set the content type
$file = '';
if ($tags) {
$file = 'filtered.';
}
if ($eventId) {
$file .= 'event-' . $eventId . '.';
}
if ($from) {
$file .= 'from-' . $from . '.';
}
if ($to) {
$file .= 'to-' . $to . '.';
}
if ($file == '') {
$file = 'all.';
}
$this->header('Content-Disposition: download; filename="misp.rpz.' . $file . 'txt"');
$this->layout = 'text/default';
$this->loadModel('Whitelist');
foreach ($values as $key => $value) {
$values[$key] = $this->Whitelist->removeWhitelistedValuesFromArray($value);
}
$this->set('values', $values);
$this->set('rpzSettings', $rpzSettings);
$this->render('/Attributes/rpz');
$this->_legacyAPIRemap(array(
'paramArray' => array(
'key', 'tags', 'eventid', 'from', 'to', 'policy', 'walled_garden', 'ns',
'email', 'serial', 'refresh', 'retry', 'expiry', 'minimum_ttl', 'ttl',
'enforceWarninglist', 'ns_alt'
),
'request' => $this->request,
'named_params' => $this->params['named'],
'ordered_url_params' => func_get_args(),
'injectedParams' => array(
'returnFormat' => 'rpz'
)
));
return $this->restSearch();
}
public function bro($key = 'download', $type = 'all', $tags = false, $eventId = false, $from = false, $to = false, $last = false, $enforceWarninglist = false)
@ -2345,7 +2151,7 @@ class AttributesController extends AppController
public function fetchViewValue($id, $field = null)
{
$validFields = array('value', 'comment', 'type', 'category', 'to_ids', 'distribution', 'timestamp');
$validFields = array('value', 'comment', 'type', 'category', 'to_ids', 'distribution', 'timestamp', 'first_seen', 'last_seen');
if (!isset($field) || !in_array($field, $validFields)) {
throw new MethodNotAllowedException(__('Invalid field requested.'));
}
@ -2392,7 +2198,7 @@ class AttributesController extends AppController
public function fetchEditForm($id, $field = null)
{
$validFields = array('value', 'comment', 'type', 'category', 'to_ids', 'distribution');
$validFields = array('value', 'comment', 'type', 'category', 'to_ids', 'distribution', 'first_seen', 'last_seen');
if (!isset($field) || !in_array($field, $validFields)) {
throw new MethodNotAllowedException(__('Invalid field requested.'));
}
@ -2978,6 +2784,7 @@ class AttributesController extends AppController
public function addTag($id = false, $tag_id = false)
{
$this->Taxonomy = $log = ClassRegistry::init('Taxonomy');
$rearrangeRules = array(
'request' => false,
'Attribute' => false,
@ -2987,20 +2794,11 @@ class AttributesController extends AppController
);
$RearrangeTool = new RequestRearrangeTool();
$this->request->data = $RearrangeTool->rearrangeArray($this->request->data, $rearrangeRules);
if ($id === false) {
if (!isset($this->request->data['attribute'])) {
throw new NotFoundException(__('Invalid attribute'));
}
$id = $this->request->data['attribute'];
}
if ($id === 'selected') {
if (!isset($this->request->data['attribute_ids'])) {
throw new NotFoundException(__('Invalid attribute'));
}
$idList = json_decode($this->request->data['attribute_ids'], true);
}
$local = empty($this->params['named']['local']) ? 0 : 1;
if (!$this->request->is('post')) {
if ($id === false) {
throw new NotFoundException(__('Invalid attribute'));
}
$this->set('local', $local);
$this->set('object_id', $id);
$this->set('scope', 'Attribute');
@ -3008,6 +2806,18 @@ class AttributesController extends AppController
$this->autoRender = false;
$this->render('/Events/add_tag');
} else {
if ($id === false) {
if (!isset($this->request->data['attribute'])) {
throw new NotFoundException(__('Invalid attribute'));
}
$id = $this->request->data['attribute'];
}
if ($id === 'selected') {
if (!isset($this->request->data['attribute_ids'])) {
throw new NotFoundException(__('Invalid attribute'));
}
$idList = json_decode($this->request->data['attribute_ids'], true);
}
if ($tag_id === false) {
if (!isset($this->request->data['tag'])) {
throw new NotFoundException(__('Invalid tag'));
@ -3118,6 +2928,20 @@ class AttributesController extends AppController
$fails++;
continue;
}
$tagsOnAttribute = $this->Attribute->AttributeTag->find('all', array(
'conditions' => array(
'AttributeTag.attribute_id' => $id,
'AttributeTag.local' => $local
),
'contain' => 'Tag',
'fields' => array('Tag.name'),
'recursive' => -1
));
$exclusiveTestPassed = $this->Taxonomy->checkIfNewTagIsAllowedByTaxonomy($tag['Tag']['name'], Hash::extract($tagsOnAttribute, '{n}.Tag.name'));
if (!$exclusiveTestPassed) {
$fails++;
continue;
}
$this->Attribute->AttributeTag->create();
if ($this->Attribute->AttributeTag->save(array('attribute_id' => $id, 'tag_id' => $tag_id, 'event_id' => $eventId, 'local' => $local))) {
if (!$local) {
@ -3126,6 +2950,9 @@ class AttributesController extends AppController
$event['Event']['timestamp'] = $date->getTimestamp();
$result = $this->Attribute->Event->save($event);
$attribute['Attribute']['timestamp'] = $date->getTimestamp();
if ($attribute['Attribute']['object_id'] != 0) {
$this->Attribute->Object->updateTimestamp($attribute['Attribute']['object_id'], $date->getTimestamp());
}
$this->Attribute->save($attribute);
}
$log = ClassRegistry::init('Log');
@ -3267,15 +3094,20 @@ class AttributesController extends AppController
'fields' => array('Tag.name')
));
if ($this->Attribute->AttributeTag->delete($attributeTag['AttributeTag']['id'])) {
$event['Event']['published'] = 0;
$date = new DateTime();
$event['Event']['timestamp'] = $date->getTimestamp();
$this->Attribute->Event->save($event);
$this->Attribute->data['Attribute']['timestamp'] = $date->getTimestamp();
$this->Attribute->save($this->Attribute->data);
if (empty($attributeTag['AttributeTag']['local'])) {
$event['Event']['published'] = 0;
$date = new DateTime();
$event['Event']['timestamp'] = $date->getTimestamp();
$this->Attribute->Event->save($event);
if ($this->Attribute->data['Attribute']['object_id'] != 0) {
$this->Attribute->Object->updateTimestamp($this->Attribute->data['Attribute']['object_id'], $date->getTimestamp());
}
$this->Attribute->data['Attribute']['timestamp'] = $date->getTimestamp();
$this->Attribute->save($this->Attribute->data);
}
$log = ClassRegistry::init('Log');
$log->createLogEntry($this->Auth->user(), 'tag', 'Attribute', $id, 'Removed tag (' . $tag_id . ') "' . $tag['Tag']['name'] . '" from attribute (' . $id . ')', 'Attribute (' . $id . ') untagged of Tag (' . $tag_id . ')');
return new CakeResponse(array('body'=> json_encode(array('saved' => true, 'success' => 'Tag removed.', 'check_publish' => true)), 'status' => 200));
return new CakeResponse(array('body'=> json_encode(array('saved' => true, 'success' => 'Tag removed.', 'check_publish' => empty($attributeTag['AttributeTag']['local']))), 'status' => 200));
} else {
return new CakeResponse(array('body'=> json_encode(array('saved' => false, 'errors' => 'Tag could not be removed.')), 'status' => 200, 'type' => 'json'));
}

27
app/Controller/CommunitiesController.php
View File

@ -17,32 +17,16 @@ class CommunitiesController extends AppController
public function index()
{
$paramsToHarvest = array('context', 'value');
foreach ($paramsToHarvest as $param) {
if (!empty($this->params['named'][$param])) {
${$param} = $this->params['named'][$param];
} else if ($this->request->is('post') && !empty($this->request->data[$param])) {
${$param} = trim($this->request->data[$param]);
} else if ($param === 'context') {
${$param} = 'vetted';
}
}
$filterData = array(
'request' => $this->request,
'paramArray' => array('context', 'value'),
'named_params' => $this->params['named']
);
$exception = false;
$filters = $this->_harvestParameters($filterData, $exception);
$filters = $this->IndexFilter->harvestParameters(array('context', 'value'));
if (empty($filters['context'])) {
$filters['context'] = 'vetted';
}
if (!empty($value)) {
$value = strtolower($value);
if (!empty($filters['value'])) {
$filters['value'] = strtolower($filters['value']);
} else {
$value = false;
$filters['value'] = false;
}
$community_list = $this->Community->getCommunityList($context, $value);
$community_list = $this->Community->getCommunityList($filters['context'], $filters['value']);
//foreach ($community)
if ($this->_isRest()) {
@ -53,7 +37,6 @@ class CommunitiesController extends AppController
$customPagination->truncateAndPaginate($community_list, $this->params, $this->modelClass, true);
$this->set('community_list', $community_list);
$this->set('context', $filters['context']);
$this->set('passedArgs', json_encode($this->passedArgs, true));
}
public function view($id)

26
app/Controller/Component/ACLComponent.php
View File

@ -25,6 +25,7 @@ class ACLComponent extends Component
'pruneDuplicateUUIDs' => array(),
'queryACL' => array(),
'removeDuplicateEvents' => array(),
'restSearch' => array('*'),
'updateDatabase' => array(),
'upgrade2324' => array(),
),
@ -122,8 +123,6 @@ class ACLComponent extends Component
'checkPublishedStatus' => array('*'),
'checkuuid' => array('perm_sync'),
'contact' => array('*'),
'create_dummy_event' => array(),
'create_massive_dummy_events' => array(),
'csv' => array('*'),
'cullEmptyEvents' => array(),
'delegation_index' => array('*'),
@ -132,7 +131,6 @@ class ACLComponent extends Component
'dot' => array(),
'downloadExport' => array('*'),
'downloadOpenIOCEvent' => array('*'),
'downloadSearchResult' => array('*'),
'edit' => array('perm_add'),
'enrichEvent' => array('perm_add'),
'export' => array('*'),
@ -146,6 +144,7 @@ class ACLComponent extends Component
'getEventGraphReferences' => array('*'),
'getEventGraphTags' => array('*'),
'getEventGraphGeneric' => array('*'),
'getEventTimeline' => array('*'),
'genDistributionGraph' => array('*'),
'getDistributionGraph' => array('*'),
'getReferenceData' => array('*'),
@ -161,6 +160,7 @@ class ACLComponent extends Component
'nids' => array('*'),
'proposalEventIndex' => array('*'),
'publish' => array('perm_publish'),
'publishSightings' => array('perm_sighting'),
'pushEventToZMQ' => array('perm_publish_zmq'),
'pushEventToKafka' => array('perm_publish_kafka'),
'pushProposals' => array('perm_sync'),
@ -278,6 +278,11 @@ class ACLComponent extends Component
'edit' => array('perm_add'),
'get_row' => array('perm_add'),
'orphanedObjectDiagnostics' => array(),
'editField' => array('perm_add'),
'fetchEditForm' => array('perm_add'),
'fetchViewValue' => array('*'),
'quickAddAttributeForm' => array('perm_add'),
'quickFetchTemplateWithValidObjectAttributes' => array('perm_add'),
'proposeObjectsFromAttributes' => array('*'),
'groupAttributesIntoObject' => array('perm_add'),
'revise_object' => array('perm_add'),
@ -356,9 +361,11 @@ class ACLComponent extends Component
),
'servers' => array(
'add' => array(),
'dbSchemaDiagnostic' => array(),
'cache' => array(),
'changePriority' => array(),
'checkout' => array(),
'clearWorkerQueue' => array(),
'createSync' => array('perm_sync'),
'delete' => array(),
'deleteFile' => array(),
@ -369,6 +376,7 @@ class ACLComponent extends Component
'getGit' => array(),
'getInstanceUUID' => array('perm_sync'),
'getPyMISPVersion' => array('*'),
'getRemoteUser' => array(),
'getSetting' => array(),
'getSubmodulesStatus' => array(),
'getSubmoduleQuickUpdateForm' => array(),
@ -383,6 +391,7 @@ class ACLComponent extends Component
'pull' => array(),
'purgeSessions' => array(),
'push' => array(),
'releaseUpdateLock' => array(),
'resetRemoteAuthKey' => array(),
'rest' => array('perm_auth'),
'restartWorkers' => array(),
@ -400,7 +409,7 @@ class ACLComponent extends Component
'updateProgress' => array(),
'updateSubmodule' => array(),
'uploadFile' => array(),
'clearWorkerQueue' => array()
'viewDeprecatedFunctionUse' => array()
),
'shadowAttributes' => array(
'accept' => array('perm_add'),
@ -440,8 +449,17 @@ class ACLComponent extends Component
'listSightings' => array('*'),
'quickDelete' => array('perm_sighting'),
'viewSightings' => array('*'),
'bulkSaveSightings' => array('OR' => array('perm_sync', 'perm_sighting')),
'quickAdd' => array('perm_sighting')
),
'sightingdb' => array(
'add' => array(),
'edit' => array(),
'delete' => array(),
'index' => array(),
'requestStatus' => array(),
'search' => array()
),
'tagCollections' => array(
'add' => array('perm_tag_editor'),
'addTag' => array('perm_tag_editor'),

95
app/Controller/Component/DeprecationComponent.php Normal file
View File

@ -0,0 +1,95 @@
<?php
class DeprecationComponent extends Component
{
public $redis = false;
/*
* Deprecated endpoints
* - simple controller->action structure
* - each endpoint can be set to to a deprecation warning message or false
*/
public $deprecatedEndpoints = false;
public function initialize(Controller $controller) {
$this->deprecatedEndpoints = array(
'attributes' => array(
'rpz' => __('Use /attributes/restSearch to export RPZ rules.'),
'text' => __('Use /attributes/restSearch to export flat indicator lists.')
),
'events' => array(
'addIOC' => __('Use MISP modules to import in OpenIOC format.'),
'csv' => __('Use /events/restSearch to export in CSV format.'),
'export' => __('Use the REST client to refine your search conditions and export in any of the given formats with much more control.'),
'hids' => __('Use /events/restSearch to export hashes.'),
'nids' => __('Use /events/restSearch to export in the various NIDS formats.'),
'stix' => __('Use /events/restSearch to export in STIX format.'),
'stix2' => __('Use /events/restSearch to export in STIX2 format.'),
'xml' => __('Use /events/restSearch to export in XML format. It is highly recommended to use JSON whenever possible.')
),
'posts' => array(
'add' => false,
'index' => false
),
'templates' => array(
'add' => false,
'populateEventFromTemplate' => false
),
'whitelists' => array(
'admin_add' => false
)
);
}
public function checkDeprecation($controller, $action, $model, $user_id)
{
if (isset($this->deprecatedEndpoints[$controller][$action])) {
$this->__logDeprecatedAccess($controller, $action, $model, $user_id);
if ($this->deprecatedEndpoints[$controller][$action]) {
return $this->deprecatedEndpoints[$controller][$action];
}
}
return false;
}
private function __logDeprecatedAccess($controller, $action, $model, $user_id)
{
$this->redis = $model->setupRedis();
if ($this->redis) {
@$this->redis->hincrby(
'misp:deprecation',
sprintf(
'%s:%s:%s',
$controller,
$action,
$user_id
),
1
);
$result = $this->redis->hGetAll('misp:deprecation');
}
return false;
}
public function getDeprecatedAccessList($model)
{
$this->redis = $model->setupRedis();
if ($this->redis) {
$rearranged = array();
@$result = $this->redis->hGetAll('misp:deprecation');
if (!empty($result)) {
foreach ($result as $key => $value) {
$key_components = explode(':', $key);
$rearranged[$key_components[0]][$key_components[1]][$key_components[2]] = (int)$value;
if (empty($rearranged[$key_components[0]][$key_components[1]]['total'])) {
$rearranged[$key_components[0]][$key_components[1]]['total'] = (int)$value;
} else {
$rearranged[$key_components[0]][$key_components[1]]['total'] += (int)$value;
}
}
}
}
return $rearranged;
}
}

80
app/Controller/Component/IndexFilterComponent.php Normal file
View File

@ -0,0 +1,80 @@
<?php
/**
* Get filter parameters from index searches
*/
class IndexFilterComponent extends Component
{
private $__Controller = false;
public function startup(Controller $controller) {
$this->__Controller = $controller;
}
// generic function to standardise on the collection of parameters. Accepts posted request objects, url params, named url params
public function harvestParameters($paramArray, &$exception = array())
{
$data = array();
if (!empty($this->__Controller->request->is('post'))) {
if (empty($this->__Controller->request->data)) {
$exception = $this->__Controller->RestResponse->throwException(
400,
__('Either specify the search terms in the url, or POST a json with the filter parameters.'),
'/' . $this->__Controller->request->params['controller'] . '/' . $this->__Controller->action
);
return false;
} else {
if (isset($this->__Controller->request->data['request'])) {
$data = $this->__Controller->request->data['request'];
} else {
$data = $this->__Controller->request->data;
}
}
}
if (!empty($paramArray)) {
foreach ($paramArray as $p) {
if (
isset($options['ordered_url_params'][$p]) &&
(!in_array(strtolower((string)$options['ordered_url_params'][$p]), array('null', '0', false, 'false', null)))
) {
$data[$p] = $options['ordered_url_params'][$p];
$data[$p] = str_replace(';', ':', $data[$p]);
}
if (isset($this->__Controller->params['named'][$p])) {
$data[$p] = str_replace(';', ':', $this->__Controller->params['named'][$p]);
}
}
}
foreach ($data as $k => $v) {
if (!is_array($data[$k])) {
$data[$k] = trim($data[$k]);
if (strpos($data[$k], '||')) {
$data[$k] = explode('||', $data[$k]);
}
}
}
if (!empty($options['additional_delimiters'])) {
if (!is_array($options['additional_delimiters'])) {
$options['additional_delimiters'] = array($options['additional_delimiters']);
}
foreach ($data as $k => $v) {
$found = false;
foreach ($options['additional_delimiters'] as $delim) {
if (strpos($v, $delim) !== false) {
$found = true;
}
}
if ($found) {
$data[$k] = explode($options['additional_delimiters'][0], str_replace($options['additional_delimiters'], $options['additional_delimiters'][0], $v));
foreach ($data[$k] as $k2 => $value) {
$data[$k][$k2] = trim($data[$k][$k2]);
}
}
}
}
$this->__Controller->set('passedArgs', json_encode($this->__Controller->passedArgs, true));
return $data;
}
}

118
app/Controller/Component/RestResponseComponent.php
View File

@ -22,13 +22,13 @@ class RestResponseComponent extends Component
'add' => array(
'description' => "POST a MISP Attribute JSON to this API to create an Attribute.",
'mandatory' => array('value', 'type'),
'optional' => array('category', 'to_ids', 'uuid', 'distribution', 'sharing_group_id', 'timestamp', 'comment'),
'optional' => array('category', 'to_ids', 'uuid', 'distribution', 'sharing_group_id', 'timestamp', 'comment', 'data', 'encrypt', 'first_seen', 'last_seen'),
'params' => array('event_id')
),
'edit' => array(
'description' => "POST a MISP Attribute JSON to this API to update an Attribute. If the timestamp is set, it has to be newer than the existing Attribute.",
'mandatory' => array(),
'optional' => array('value', 'type', 'category', 'to_ids', 'uuid', 'distribution', 'sharing_group_id', 'timestamp', 'comment'),
'optional' => array('value', 'type', 'category', 'to_ids', 'uuid', 'distribution', 'sharing_group_id', 'timestamp', 'comment', 'date', 'encrypt', 'first_seen', 'last_seen'),
'params' => array('attribute_id')
),
'deleteSelected' => array(
@ -40,7 +40,7 @@ class RestResponseComponent extends Component
'restSearch' => array(
'description' => "Search MISP using a list of filter parameters and return the data in the selected format. The search is available on an event and an attribute level, just select the scope via the URL (/events/restSearch vs /attributes/restSearch). Besides the parameters listed, other, format specific ones can be passed along (for example: requested_attributes and includeContext for the CSV export). This API allows pagination via the page and limit parameters.",
'mandatory' => array('returnFormat'),
'optional' => array('page', 'limit', 'value' , 'type', 'category', 'org', 'tags', 'date', 'last', 'eventid', 'withAttachments', 'uuid', 'publish_timestamp', 'timestamp', 'enforceWarninglist', 'to_ids', 'deleted', 'includeEventUuid', 'includeEventTags', 'event_timestamp', 'threat_level_id', 'eventinfo', 'includeProposals', 'includeDecayScore', 'includeFullModel', 'decayingModel', 'excludeDecayed', 'score'),
'optional' => array('page', 'limit', 'value' , 'type', 'category', 'org', 'tags', 'date', 'last', 'eventid', 'withAttachments', 'uuid', 'publish_timestamp', 'timestamp', 'attribute_timestamp', 'enforceWarninglist', 'to_ids', 'deleted', 'includeEventUuid', 'includeEventTags', 'event_timestamp', 'threat_level_id', 'eventinfo', 'includeProposals', 'includeDecayScore', 'includeFullModel', 'decayingModel', 'excludeDecayed', 'score', 'first_seen', 'last_seen'),
'params' => array()
)
),
@ -171,11 +171,11 @@ class RestResponseComponent extends Component
'add' => array(
'description' => "POST an Server object in JSON format to this API to add a server.",
'mandatory' => array('url', 'name', 'remote_org_id', 'authkey'),
'optional' => array('push', 'pull', 'push_rules', 'pull_rules', 'submitted_cert', 'submitted_client_cert', 'json')
'optional' => array('push', 'pull', 'push_sightings', 'push_rules', 'pull_rules', 'submitted_cert', 'submitted_client_cert', 'json')
),
'edit' => array(
'description' => "POST an Server object in JSON format to this API to edit a server.",
'optional' => array('url', 'name', 'authkey', 'json', 'push', 'pull', 'push_rules', 'pull_rules', 'submitted_cert', 'submitted_client_cert', 'remote_org_id')
'optional' => array('url', 'name', 'authkey', 'json', 'push', 'pull', 'push_sightings', 'push_rules', 'pull_rules', 'submitted_cert', 'submitted_client_cert', 'remote_org_id')
),
'serverSettings' => array(
'description' => "Send a GET request to this endpoint to get a full diagnostic along with all currently set settings of the current instance. This will also include the worker status"
@ -278,6 +278,7 @@ class RestResponseComponent extends Component
public function initialize(Controller $controller) {
$this->__configureFieldConstraints();
$this->Controller = $controller;
}
public function getAllApisFieldsConstraint($user)
@ -391,6 +392,7 @@ class RestResponseComponent extends Component
if (isset($this->__convertActionToMessage[$controller][$action['action']])) {
$stringifiedAction = $this->__convertActionToMessage[$controller][$action['action']];
}
$response['saved'] = false;
$response['name'] = 'Could not ' . $stringifiedAction . ' ' . Inflector::singularize($controller);
$response['message'] = $response['name'];
$response['url'] = $this->__generateURL($action, $controller, $id);
@ -404,6 +406,8 @@ class RestResponseComponent extends Component
if (!$message) {
$message = Inflector::singularize($controller) . ' ' . $action['action'] . ((substr($action['action'], -1) == 'e') ? 'd' : 'ed');
}
$response['saved'] = true;
$response['success'] = true;
$response['name'] = $message;
$response['message'] = $response['name'];
$response['url'] = $this->__generateURL($action, $controller, $id);
@ -443,7 +447,28 @@ class RestResponseComponent extends Component
if (is_string($response)) {
$response = array('message' => $response);
}
if (Configure::read('debug') > 1 && !empty($this->Controller->sql_dump)) {
$this->Log = ClassRegistry::init('Log');
if ($this->Content->sql_dump === 2) {
$response = array('sql_dump' => $this->Log->getDataSource()->getLog(false, false));
} else {
$response['sql_dump'] = $this->Log->getDataSource()->getLog(false, false);
}
}
$response = json_encode($response, JSON_PRETTY_PRINT);
} else {
if (Configure::read('debug') > 1 && !empty($this->Controller->sql_dump)) {
$this->Log = ClassRegistry::init('Log');
if ($this->Controller->sql_dump === 2) {
$response = json_encode(array('sql_dump' => $this->Log->getDataSource()->getLog(false, false)));
} else {
$response = substr_replace(
$response,
sprintf(', "sql_dump": %s}', json_encode($this->Log->getDataSource()->getLog(false, false))),
-2
);
}
}
}
}
$cakeResponse = new CakeResponse(array('body'=> $response, 'status' => $code, 'type' => $type));
@ -464,6 +489,9 @@ class RestResponseComponent extends Component
$cakeResponse->header($key, $value);
}
}
if (!empty($deprecationWarnings)) {
$cakeResponse->header('X-Deprecation-Warning', $deprecationWarnings);
}
if ($download) {
$cakeResponse->download($download);
}
@ -699,6 +727,12 @@ class RestResponseComponent extends Component
'autoclose' => true
),
),
'data' => array(
'input' => 'text',
'type' => 'string',
'operators' => array('equal'),
'help' => __('Base64 encoded file contents')
),
'date' => array(
'type' => 'date',
'validation' => array( 'format' => 'YYYY-MM-DD' ),
@ -798,6 +832,12 @@ class RestResponseComponent extends Component
'type' => 'integer',
'values' => array(1 => 'True', 0 => 'False' )
),
'encrypt' => array(
'input' => 'radio',
'type' => 'integer',
'values' => array(1 => 'True', 0 => 'False' ),
'help' => __('When uploading malicious samples, set this flag to tell MISP to encrpyt the sample and extract the file hashes. This will create a MISP object with the appropriate attributes.')
),
//'enforceWarningList' => array(
// 'input' => 'radio',
// 'type' => 'integer',
@ -820,7 +860,14 @@ class RestResponseComponent extends Component
'type' => 'integer',
'operators' => array('equal', 'not_equal'),
'validation' => array('min' => 0, 'step' => 1),
'help' => __('The timestamp at which the event was published')
'help' => __('The timestamp at which the event was last modified')
),
'attribute_timestamp' => array(
'input' => 'number',
'type' => 'integer',
'operators' => array('equal', 'not_equal'),
'validation' => array('min' => 0, 'step' => 1),
'help' => __('The timestamp at which the attribute was last modified')
),
'eventid' => array(
'input' => 'number',
@ -870,6 +917,12 @@ class RestResponseComponent extends Component
'operators' => array('equal'),
'help' => __('A valid external auth key')
),
'first_seen' => array(
'input' => 'text',
'type' => 'string',
'operators' => array('equal'),
'help' => 'A valid ISO 8601 datetime format, up to milli-seconds. i.e.: 2019-06-13T15:56:56.856074+02:00'
),
'fixed_event' => array(
'input' => 'select',
'type' => 'integer',
@ -992,6 +1045,12 @@ class RestResponseComponent extends Component
'operators' => array('equal', 'not_equal'),
'help' => __('Events published within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m)')
),
'last_seen' => array(
'input' => 'text',
'type' => 'string',
'operators' => array('equal'),
'help' => 'A valid ISO 8601 datetime format, up to milli-seconds. i.e.: 2019-06-13T15:56:56.856074+02:00'
),
'limit' => array(
'input' => 'number',
'type' => 'integer',
@ -1247,6 +1306,12 @@ class RestResponseComponent extends Component
'values' => array(1 => 'True', 0 => 'False' ),
'help' => __('Allow the upload of events and their attribute to the server')
),
'push_sightings' => array(
'input' => 'radio',
'type' => 'integer',
'values' => array(1 => 'True', 0 => 'False' ),
'help' => __('Allow the upload of sightings to the server')
),
'releasability' => array(
'input' => 'text',
'type' => 'string',
@ -1580,33 +1645,37 @@ class RestResponseComponent extends Component
// add dynamic data and overwrite name collisions
switch($field) {
case "returnFormat":
$this->__overwriteReturnFormat($scope, $fieldsConstraint[$field]);
$this->__overwriteReturnFormat($scope, $action, $fieldsConstraint[$field]);
break;
case "type":
$this->__overwriteType($scope, $fieldsConstraint[$field]);
$this->__overwriteType($scope, $action, $fieldsConstraint[$field]);
break;
case "category":
$this->__overwriteCategory($scope, $fieldsConstraint[$field]);
$this->__overwriteCategory($scope, $action, $fieldsConstraint[$field]);
break;
case "decayingModel":
$this->__overwriteDecayingModel($scope, $fieldsConstraint[$field]);
break;
case "distribution":
$this->__overwriteDistribution($scope, $fieldsConstraint[$field]);
$this->__overwriteDistribution($scope, $action, $fieldsConstraint[$field]);
break;
case "tag":
case "tags":
case "EventTag":
$this->__overwriteTags($scope, $fieldsConstraint[$field]);
$this->__overwriteTags($scope, $action, $fieldsConstraint[$field]);
break;
case "nationality":
$this->__overwriteNationality($scope, $fieldsConstraint[$field]);
$this->__overwriteNationality($scope, $action, $fieldsConstraint[$field]);
break;
case "action":
$this->__overwriteAction($scope, $fieldsConstraint[$field]);
$this->__overwriteAction($scope, $action, $fieldsConstraint[$field]);
break;
case "role_id":
$this->__overwriteRoleId($scope, $fieldsConstraint[$field]);
$this->__overwriteRoleId($scope, $action, $fieldsConstraint[$field]);
break;
case "first_seen":
case "last_seen":
$this->__overwriteSeen($scope, $action, $fieldsConstraint[$field]);
break;
default:
break;
@ -1622,7 +1691,7 @@ class RestResponseComponent extends Component
}
// Fetch the correct values based on the scope, then overwrite default value
private function __overwriteReturnFormat($scope, &$field) {
private function __overwriteReturnFormat($scope, $action, &$field) {
switch($scope) {
case "Attribute":
$field['values'] = array_keys(ClassRegistry::init($scope)->validFormats);
@ -1632,7 +1701,7 @@ class RestResponseComponent extends Component
break;
}
}
private function __overwriteType($scope, &$field) {
private function __overwriteType($scope, $action, &$field) {
$field['input'] = 'select';
switch($scope) {
case "Attribute":
@ -1650,10 +1719,10 @@ class RestResponseComponent extends Component
}
}
private function __overwriteCategory($scope, &$field) {
private function __overwriteCategory($scope, $action, &$field) {
$field['values'] = array_keys(ClassRegistry::init("Attribute")->categoryDefinitions);
}
private function __overwriteDistribution($scope, &$field) {
private function __overwriteDistribution($scope, $action, &$field) {
$field['values'] = array();
foreach(ClassRegistry::init("Attribute")->distributionLevels as $d => $text) {
$field['values'][] = array('label' => $text, 'value' => $d);
@ -1670,7 +1739,7 @@ class RestResponseComponent extends Component
$field['values'][] = array('label' => h($model_name), 'value' => $i);
}
}
private function __overwriteTags($scope, &$field) {
private function __overwriteTags($scope, $action, &$field) {
$this->{$scope} = ClassRegistry::init("Tag");
$tags = $this->{$scope}->find('list', array(
'recursive' => -1,
@ -1683,13 +1752,13 @@ class RestResponseComponent extends Component
}
$field['values'] = $tags;
}
private function __overwriteNationality($scope, &$field) {
private function __overwriteNationality($scope, $action, &$field) {
$field['values'] = ClassRegistry::init("Organisation")->countries;
}
private function __overwriteAction($scope, &$field) {
private function __overwriteAction($scope, $action, &$field) {
$field['values'] = array_keys(ClassRegistry::init("Log")->actionDefinitions);
}
private function __overwriteRoleId($scope, &$field) {
private function __overwriteRoleId($scope, $action, &$field) {
$this->{$scope} = ClassRegistry::init("Role");
$roles = $this->{$scope}->find('list', array(
'recursive' => -1,
@ -1697,5 +1766,10 @@ class RestResponseComponent extends Component
));
$field['values'] = $roles;
}
private function __overwriteSeen($scope, $action, &$field) {
if ($action == 'restSearch') {
$field['help'] = __('Seen within the last x amount of time, where x can be defined in days, hours, minutes (for example 5d or 12h or 30m)');
}
}
}

46
app/Controller/Component/RestSearchComponent.php Normal file
View File

@ -0,0 +1,46 @@
<?php
App::uses('Component', 'Controller');
class RestSearchComponent extends Component
{
public $paramArray = array(
'Attribute' => array(
'returnFormat', 'value' , 'type', 'category', 'org', 'tags', 'from', 'to', 'last', 'eventid', 'withAttachments', 'uuid', 'publish_timestamp',
'published', 'timestamp','enforceWarninglist', 'to_ids', 'deleted', 'includeEventUuid', 'event_timestamp', 'threat_level_id', 'includeEventTags',
'includeProposals', 'returnFormat', 'limit', 'page', 'requested_attributes', 'includeContext', 'headerless',
'includeWarninglistHits', 'attackGalaxy', 'object_relation', 'includeSightings', 'includeCorrelations', 'includeDecayScore',
'decayingModel', 'excludeDecayed', 'modelOverrides', 'includeFullModel', 'score', 'attribute_timestamp', 'first_seen', 'last_seen'
),
'Event' => array(
'returnFormat', 'value', 'type', 'category', 'org', 'tags', 'searchall', 'from', 'to', 'last', 'eventid', 'withAttachments',
'metadata', 'uuid', 'publish_timestamp', 'timestamp', 'published', 'enforceWarninglist', 'sgReferenceOnly',
'limit', 'page', 'requested_attributes', 'includeContext', 'headerless', 'includeWarninglistHits', 'attackGalaxy', 'deleted',
'excludeLocalTags', 'date', 'includeSightingdb', 'tag', 'object_relation'
),
'Sighting' => array(
'context', 'returnFormat', 'id', 'type', 'from', 'to', 'last', 'org_id', 'source', 'includeAttribute', 'includeEvent'
)
);
public function getFilename($filters, $scope, $responseType)
{
$filename = false;
if ($scope === 'Event') {
$filename = 'misp.event.';
if (!empty($filters['eventid']) && !is_array($filters['eventid'])) {
if (Validation::uuid(trim($filters['eventid']))) {
$filename .= trim($filters['eventid']);
} else if (!empty(intval(trim($filters['eventid'])))) {
$filename .= intval(trim($filters['eventid']));
}
} else {
$filename .= 'list';
}
}
if ($filename !== false) {
$filename .= '.' . $responseType;
}
return $filename;
}
}

3
app/Controller/DecayingModelController.php
View File

@ -591,7 +591,8 @@ class DecayingModelController extends AppController
$filters['uuid'] = $filters['id'];
} else {
$attributes = $this->User->Event->Attribute->fetchAttributes($this->Auth->user(), array(
'conditions' => array('Attribute.id' => $filters['id'])
'conditions' => array('Attribute.id' => $filters['id']),
'flatten' => 1
));
if (!empty($attributes)) {
$filters['uuid'] = $attributes[0]['Attribute']['uuid'];

1051
app/Controller/EventsController.php
View File

File diff suppressed because it is too large Load Diff

14
app/Controller/GalaxiesController.php
View File

@ -111,17 +111,19 @@ class GalaxiesController extends AppController
{
$mitreAttackGalaxyId = $this->Galaxy->getMitreAttackGalaxyId();
$local = !empty($this->params['named']['local']) ? $this->params['named']['local'] : '0';
$conditions = $namespace == '0' ? array() : array('namespace' => $namespace);
$conditions = $namespace === '0' ? array() : array('namespace' => $namespace);
$galaxies = $this->Galaxy->find('all', array(
'recursive' => -1,
'fields' => array('MAX(Galaxy.version) as latest_version', 'id', 'kill_chain_order', 'name', 'icon', 'description'),
'conditions' => $conditions,
'group' => array('name', 'id', 'kill_chain_order', 'icon', 'description'),
'order' => array('name asc')
));
$items = array();
$items[] = array(
'name' => __('All clusters'),
'value' => "/galaxies/selectCluster/" . h($target_id) . '/' . h($target_type) . '/0'. '/local:' . $local
$items = array(
array(
'name' => __('All clusters'),
'value' => "/galaxies/selectCluster/" . h($target_id) . '/' . h($target_type) . '/0'. '/local:' . $local
)
);
foreach ($galaxies as $galaxy) {
if (!isset($galaxy['Galaxy']['kill_chain_order'])) {

2
app/Controller/JobsController.php
View File

@ -25,7 +25,7 @@ class JobsController extends AppController
$issueCount = 0;
$workers = $this->Server->workerDiagnostics($issueCount);
$this->recursive = 0;
$queues = array('email', 'default', 'cache', 'prio');
$queues = array('email', 'default', 'cache', 'prio', 'update');
if ($queue && in_array($queue, $queues)) {
$this->paginate['conditions'] = array('Job.worker' => $queue);
}

2
app/Controller/LogsController.php
View File

@ -416,7 +416,7 @@ class LogsController extends AppController
$this->set('actions', $actions);
// combobox for models
$models = array('Attribute', 'Event', 'EventBlacklist', 'EventTag', 'DecayingModel', 'MispObject', 'Organisation', 'Post', 'Regexp', 'Role', 'Server', 'ShadowAttribute', 'SharingGroup', 'Tag', 'Task', 'Taxonomy', 'Template', 'Thread', 'User', 'Whitelist');
$models = array('Attribute', 'Event', 'EventBlacklist', 'EventTag', 'Feed', 'DecayingModel', 'MispObject', 'Organisation', 'Post', 'Regexp', 'Role', 'Server', 'ShadowAttribute', 'SharingGroup', 'Tag', 'Task', 'Taxonomy', 'Template', 'Thread', 'User', 'Whitelist');
$models = array('' => 'ALL') + $this->_arrayToValuesIndexArray($models);
$this->set('models', $models);
$this->set('actionDefinitions', $this->{$this->defaultModel}->actionDefinitions);

539
app/Controller/ObjectsController.php
View File

@ -198,6 +198,7 @@ class ObjectsController extends AppController
$this->MispObject->Event->insertLock($this->Auth->user(), $eventId);
}
$error = false;
$template = false;
if (!empty($templateId) || !$this->_isRest()) {
$templates = $this->MispObject->ObjectTemplate->find('all', array(
'conditions' => array('ObjectTemplate.id' => $templateId),
@ -247,10 +248,14 @@ class ObjectsController extends AppController
foreach ($object['Attribute'] as $k => $attribute) {
unset($object['Attribute'][$k]['id']);
$object['Attribute'][$k]['event_id'] = $eventId;
$this->MispObject->Event->Attribute->set($attribute);
$this->MispObject->Event->Attribute->set($object['Attribute'][$k]);
if (!$this->MispObject->Event->Attribute->validates()) {
if ($this->MispObject->Event->Attribute->validationErrors['value'][0] !== 'Composite type found but the value not in the composite (value1|value2) format.') {
$error = 'Could not save object as at least one attribute has failed validation (' . $attribute['object_relation'] . '). ' . json_encode($this->MispObject->Event->Attribute->validationErrors);
$error = sprintf(
'Could not save object as at least one attribute has failed validation (%s). %s',
isset($attribute['object_relation']) ? $attribute['object_relation'] : 'No object_relation',
json_encode($this->MispObject->Event->Attribute->validationErrors)
);
}
}
}
@ -353,43 +358,20 @@ class ObjectsController extends AppController
$this->set('element', $element);
}
public function edit($id, $update_template_available=false)
public function edit($id, $update_template_available=false, $onlyAddNewAttribute=false)
{
if (Validation::uuid($id)) {
$conditions = array('Object.uuid' => $id);
} else {
$conditions = array('Object.id' => $id);
}
if (!$this->userRole['perm_modify']) {
throw new MethodNotAllowedException(__('You don\'t have permissions to edit objects.'));
}
$object = $this->MispObject->find('first', array(
'conditions' => $conditions,
'recursive' => -1,
'contain' => array(
'Attribute' => array(
'conditions' => array(
'Attribute.deleted' => 0
)
)
)
));
$id = $this->Toolbox->findIdByUuid($this->MispObject, $id);
$object = $this->MispObject->fetchObjects($this->Auth->user(), array('conditions' => array('Object.id' => $id)));
if (empty($object)) {
throw new NotFoundException(__('Invalid object.'));
}
$id = $object['Object']['id'];
$eventFindParams = array(
'recursive' => -1,
'fields' => array('Event.id', 'Event.uuid', 'Event.orgc_id'),
'conditions' => array('Event.id' => $object['Object']['event_id'])
);
$event = $this->MispObject->Event->find('first', $eventFindParams);
if (empty($event) || (!$this->_isSiteAdmin() && $event['Event']['orgc_id'] != $this->Auth->user('org_id'))) {
throw new NotFoundException(__('Invalid object.'));
$object = $object[0];
if ((!$this->_isSiteAdmin() && $object['Event']['orgc_id'] != $this->Auth->user('org_id'))) {
throw new MethodNotAllowedException(__('Insufficient permissions to edit this object.'));
}
$event = $this->MispObject->Event->fetchEvent($this->Auth->user(), array('eventid' => $object['Event']['id'], 'metadata' => 1))[0];
if (!$this->_isRest()) {
$this->MispObject->Event->insertLock($this->Auth->user(), $event['Event']['id']);
$this->MispObject->Event->insertLock($this->Auth->user(), $object['Event']['id']);
}
$template = $this->MispObject->ObjectTemplate->find('first', array(
'conditions' => array(
@ -401,130 +383,28 @@ class ObjectsController extends AppController
'ObjectTemplateElement'
)
));
if (empty($template)) {
if (empty($template) && !$this->_isRest()) {
$this->Flash->error('Object cannot be edited, no valid template found.');
$this->redirect(array('controller' => 'events', 'action' => 'view', $object['Object']['event_id']));
}
$newer_template = $this->MispObject->ObjectTemplate->find('first', array(
'conditions' => array(
'ObjectTemplate.uuid' => $object['Object']['template_uuid'],
'ObjectTemplate.version >' => $object['Object']['template_version'],
),
'recursive' => -1,
'contain' => array(
'ObjectTemplateElement'
),
'order' => array('ObjectTemplate.version DESC')
));
if (!empty($newer_template)) {
$newer_template_version = $newer_template['ObjectTemplate']['version'];
// ignore IDs for comparison
$cur_template_temp = Hash::remove(Hash::remove($template['ObjectTemplateElement'], '{n}.id'), '{n}.object_template_id');
$newer_template_temp = Hash::remove(Hash::remove($newer_template['ObjectTemplateElement'], '{n}.id'), '{n}.object_template_id');
$template_difference = array();
// check how current template is included in the newer
foreach ($cur_template_temp as $cur_obj_rel) {
$flag_sim = false;
foreach ($newer_template_temp as $newer_obj_rel) {
$tmp = Hash::diff($cur_obj_rel, $newer_obj_rel);
if (count($tmp) == 0) {
$flag_sim = true;
break;
}
}
if (!$flag_sim) {
$template_difference[] = $cur_obj_rel;
}
}
$updateable_attribute = $object['Attribute'];
$not_updateable_attribute = array();
if (!empty($template_difference)) { // older template not completely embeded in newer
foreach ($template_difference as $temp_diff_element) {
foreach ($object['Attribute'] as $i => $attribute) {
if (
$attribute['object_relation'] == $temp_diff_element['object_relation']
&& $attribute['type'] == $temp_diff_element['type']
) { // This attribute cannot be merged automatically
$attribute['merge-possible'] = false;
$not_updateable_attribute[] = $attribute;
unset($updateable_attribute[$i]);
}
}
}
}
$this->set('updateable_attribute', $updateable_attribute);
$this->set('not_updateable_attribute', $not_updateable_attribute);
if ($update_template_available) { // template version bump requested
$template = $newer_template; // bump the template version
}
} else {
$newer_template_version = false;
$templateData = $this->MispObject->resolveUpdatedTemplate($template, $object, $update_template_available);
$this->set('updateable_attribute', $templateData['updateable_attribute']);
$this->set('not_updateable_attribute', $templateData['not_updateable_attribute']);
if (isset($this->params['named']['revised_object'])) {
$revisedData = $this->MispObject->reviseObject($this->params['named']['revised_object'], $object);
$this->set('revised_object', $revisedData['revised_object_both']);
$object = $revisedData['object'];
}
if (isset($this->params['named']['revised_object'])) { // revised object data to be injected
$revised_object = json_decode(base64_decode($this->params['named']['revised_object']), true);
$revised_object_both = array('mergeable' => array(), 'notMergeable' => array());
// Loop through attributes to inject and perform the correct action
// (inject, duplicate, add warnings, ...) when applicable
foreach ($revised_object['Attribute'] as $attribute_to_inject) {
$flag_no_collision = true;
foreach ($object['Attribute'] as $attribute) {
if (
$attribute['object_relation'] == $attribute_to_inject['object_relation']
&& $attribute['type'] == $attribute_to_inject['type']
&& $attribute['value'] !== $attribute_to_inject['value']
) { // Collision on value
$multiple = !empty(Hash::extract($template['ObjectTemplateElement'], sprintf('{n}[object_relation=%s][type=%s][multiple=true]', $attribute['object_relation'], $attribute['type'])));
if ($multiple) { // if multiple is set, check if an entry exists already
$flag_entry_exists = false;
foreach ($object['Attribute'] as $attr) {
if (
$attr['object_relation'] == $attribute_to_inject['object_relation']
&& $attr['type'] == $attribute_to_inject['type']
&& $attr['value'] === $attribute_to_inject['value']
) {
$flag_entry_exists = true;
break;
}
}
if (!$flag_entry_exists) { // entry does no exists, can be duplicated
$attribute_to_inject['is_multiple'] = true;
$revised_object_both['mergeable'][] = $attribute_to_inject;
$object['Attribute'][] = $attribute_to_inject;
}
} else { // Collision on value, multiple not set => propose overwrite
$attribute_to_inject['current_value'] = $attribute['value'];
$attribute_to_inject['merge-possible'] = true; // the user can still swap value
$revised_object_both['notMergeable'][] = $attribute_to_inject;
}
$flag_no_collision = false;
} else if (
$attribute['object_relation'] == $attribute_to_inject['object_relation']
&& $attribute['type'] == $attribute_to_inject['type']
&& $attribute['value'] === $attribute_to_inject['value']
) { // all good, they are basically the same, do nothing
$revised_object_both['mergeable'][] = $attribute_to_inject;
$flag_no_collision = false;
}
}
if ($flag_no_collision) { // no collision, nor equalities => inject it straight away
$revised_object_both['mergeable'][] = $attribute_to_inject;
$object['Attribute'][] = $attribute_to_inject;
}
}
$this->set('revised_object', $revised_object_both);
if (!empty($templateData['template'])) {
$template = $this->MispObject->prepareTemplate($templateData['template'], $object);
}
$template = $this->MispObject->prepareTemplate($template, $object);
$enabledRows = false;
if ($this->request->is('post') || $this->request->is('put')) {
if (isset($this->request->data['request'])) {
$this->request->data = $this->request->data['request'];
}
if (empty($this->request->data['Object'])) {
$this->request->data['Object'] = $this->request->data;
}
if (isset($this->request->data['Object']['data'])) {
$this->request->data = json_decode($this->request->data['Object']['data'], true);
}
@ -533,34 +413,41 @@ class ObjectsController extends AppController
unset($this->request->data['Object']);
}
$objectToSave = $this->MispObject->attributeCleanup($this->request->data);
$objectToSave = $this->MispObject->deltaMerge($object, $objectToSave);
$objectToSave = $this->MispObject->deltaMerge($object, $objectToSave, $onlyAddNewAttribute);
// we pre-validate the attributes before we create an object at this point
// This allows us to stop the process and return an error (API) or return
// to the add form
if (empty($error)) {
if ($this->_isRest()) {
if ($this->_isRest()) {
if (is_numeric($objectToSave)) {
$objectToSave = $this->MispObject->fetchObjects($this->Auth->user(), array('conditions' => array('Object.id' => $id)));
if (!empty($objectToSave)) {
$objectToSave = $objectToSave[0];
$objectToSave['Object']['Attribute'] = $objectToSave['Attribute'];
unset($objectToSave['Attribute']);
}
$this->MispObject->Event->unpublishEvent($objectToSave['Object']['event_id']);
return $this->RestResponse->viewData($objectToSave, $this->response->type());
} else {
return $this->RestResponse->saveFailResponse('Objects', 'add', false, $id, $this->response->type());
}
} else {
$message = __('Object attributes saved.');
$error_message = __('Object attributes could not be saved.');
if ($this->request->is('ajax')) {
$this->autoRender = false;
if (is_numeric($objectToSave)) {
$objectToSave = $this->MispObject->find('first', array(
'recursive' => -1,
'conditions' => array('Object.id' => $id),
'contain' => array(
'Attribute' => array(
'fields' => $this->MispObject->Attribute->defaultFields
)
)
));
if (!empty($objectToSave)) {
$objectToSave['Object']['Attribute'] = $objectToSave['Attribute'];
unset($objectToSave['Attribute']);
}
$this->MispObject->Event->unpublishEvent($object['Object']['event_id']);
return $this->RestResponse->viewData($objectToSave, $this->response->type());
return new CakeResponse(array('body'=> json_encode(array('saved' => true, 'success' => $message)), 'status'=>200, 'type' => 'json'));
} else {
return $this->RestResponse->saveFailResponse('Objects', 'add', false, $id, $this->response->type());
return new CakeResponse(array('body'=> json_encode(array('saved' => true, 'errors' => $error_message)), 'status'=>200, 'type' => 'json'));
}
} else {
$this->MispObject->Event->unpublishEvent($object['Object']['event_id']);
$this->Flash->success('Object saved.');
if (is_numeric($objectToSave)) {
$this->MispObject->Event->unpublishEvent($object['Object']['event_id']);
$this->Flash->success('Object saved.');
} else {
$this->Flash->error($error_message);
}
$this->redirect(array('controller' => 'events', 'action' => 'view', $object['Object']['event_id']));
}
}
@ -596,10 +483,324 @@ class ObjectsController extends AppController
$this->set('action', 'edit');
$this->set('object', $object);
$this->set('update_template_available', $update_template_available);
$this->set('newer_template_version', $newer_template_version);
$this->set('newer_template_version', empty($templateData['newer_template_version']) ? false : $templateData['newer_template_version']);
$this->render('add');
}
// ajax edit - post a single edited field and this method will attempt to save it and return a json with the validation errors if they occur.
public function editField($id)
{
if (Validation::uuid($id)) {
$this->MispObject->recursive = -1;
$temp = $this->MispObject->findByUuid($id);
if ($temp == null) {
throw new NotFoundException(__('Invalid object'));
}
$id = $temp['Object']['id'];
} elseif (!is_numeric($id)) {
throw new NotFoundException(__('Invalid event id.'));
}
if ((!$this->request->is('post') && !$this->request->is('put'))) {
throw new MethodNotAllowedException(__('This function can only be accessed via POST or PUT'));
}
$object = $this->MispObject->find('first', array(
'conditions' => array('Object.id' => $id),
'contain' => 'Event',
'recursive' => -1
));
if (empty($object)) {
return $this->RestResponse->saveFailResponse('Objects', 'edit', false, 'Invalid object');
}
if (!$this->_isSiteAdmin()) {
if ($this->MispObject->data['Event']['orgc_id'] == $this->Auth->user('org_id')
&& (($this->userRole['perm_modify'] && $this->MispObject->data['Event']['user_id'] != $this->Auth->user('id'))
|| $this->userRole['perm_modify_org'])) {
// Allow the edit
} else {
return $this->RestResponse->saveFailResponse('Objects', 'edit', false, 'Invalid attribute');
}
}
$validFields = array('comment', 'distribution', 'first_seen', 'last_seen');
$changed = false;
if (empty($this->request->data['Object'])) {
$this->request->data = array('Object' => $this->request->data);
if (empty($this->request->data['Object'])) {
throw new MethodNotAllowedException('Invalid input.');
}
}
$seen_changed = false;
foreach ($this->request->data['Object'] as $changedKey => $changedField) {
if (!in_array($changedKey, $validFields)) {
throw new MethodNotAllowedException('Invalid field.');
}
if ($object['Object'][$changedKey] == $changedField) {
$this->autoRender = false;
return $this->RestResponse->saveSuccessResponse('Objects', 'edit', $id, false, 'nochange');
}
$seen_changed = $changedKey == 'first_seen' || $changedKey == 'last_seen';
$object['Object'][$changedKey] = $changedField;
$changed = true;
}
$forcedSeenOnElements = array();
if (!$changed) {
return $this->RestResponse->saveSuccessResponse('Objects', 'edit', $id, false, 'nochange');
} elseif ($seen_changed) {
$forcedSeenOnElements[$changedKey] = $changedField;
}
$date = new DateTime();
$object['Object']['timestamp'] = $date->getTimestamp();
$object = $this->MispObject->syncObjectAndAttributeSeen($object, $forcedSeenOnElements);
if ($this->MispObject->save($object)) {
$event = $this->MispObject->Event->find('first', array(
'recursive' => -1,
'fields' => array('id', 'published', 'timestamp', 'info', 'uuid'),
'conditions' => array(
'id' => $object['Object']['event_id'],
)));
$event['Event']['timestamp'] = $date->getTimestamp();
$event['Event']['published'] = 0;
if ($seen_changed) {
$this->MispObject->Attribute->saveAttributes($object['Attribute']);
}
$this->MispObject->Event->save($event, array('fieldList' => array('published', 'timestamp', 'info')));
return $this->RestResponse->saveSuccessResponse('Objects', 'edit', $id, false, 'Field updated');
} else {
return $this->RestResponse->saveFailResponse('Objects', 'edit', false, $this->MispObject->validationErrors);
}
}
public function fetchViewValue($id, $field = null)
{
$validFields = array('timestamp', 'comment', 'distribution', 'first_seen', 'last_seen');
if (!isset($field) || !in_array($field, $validFields)) {
throw new MethodNotAllowedException('Invalid field requested.');
}
if (!$this->request->is('ajax')) {
throw new MethodNotAllowedException('This function can only be accessed via AJAX.');
}
$params = array(
'conditions' => array('Object.id' => $id),
'fields' => array('id', 'distribution', 'event_id', $field),
'contain' => array(
'Event' => array(
'fields' => array('distribution', 'id', 'org_id'),
)
),
'flatten' => 1
);
$object = $this->MispObject->fetchObjectSimple($this->Auth->user(), $params);
if (empty($object)) {
throw new NotFoundException(__('Invalid object'));
}
$object = $object[0];
$result = $object['Object'][$field];
if ($field == 'distribution') {
$result=$this->MispObject->shortDist[$result];
}
$this->set('value', $result);
$this->layout = 'ajax';
$this->render('ajax/objectViewFieldForm');
}
public function fetchEditForm($id, $field = null)
{
$validFields = array('distribution', 'comment', 'first_seen', 'last_seen');
if (!isset($field) || !in_array($field, $validFields)) {
throw new MethodNotAllowedException('Invalid field requested.');
}
if (!$this->request->is('ajax')) {
throw new MethodNotAllowedException('This function can only be accessed via AJAX.');
}
$fields = array('id', 'distribution', 'event_id');
$fields[] = $field;
$params = array(
'conditions' => array('Object.id' => $id),
'fields' => $fields,
'contain' => array(
'Event' => array(
'fields' => array('distribution', 'id', 'user_id', 'orgc_id', 'org_id'),
)
)
);
$object = $this->MispObject->fetchObjectSimple($this->Auth->user(), $params);
if (empty($object)) {
throw new NotFoundException(__('Invalid attribute'));
}
$object = $object[0];
if (!$this->_isSiteAdmin()) {
if ($object['Event']['orgc_id'] == $this->Auth->user('org_id')
&& (($this->userRole['perm_modify'] && $object['Event']['user_id'] != $this->Auth->user('id'))
|| $this->userRole['perm_modify_org'])) {
// Allow the edit
} else {
throw new NotFoundException(__('Invalid object'));
}
}
$this->layout = 'ajax';
if ($field == 'distribution') {
$distributionLevels = $this->MispObject->shortDist;
unset($distributionLevels[4]);
$this->set('distributionLevels', $distributionLevels);
}
$this->set('object', $object['Object']);
$fieldURL = ucfirst($field);
$this->render('ajax/objectEdit' . $fieldURL . 'Form');
}
// Construct a template with valid object attributes to add to an object
public function quickFetchTemplateWithValidObjectAttributes($id) {
$this->MispObject->id = $id;
if (!$this->MispObject->exists()) {
if ($this->request->is('ajax')) {
return $this->RestResponse->saveFailResponse('Objects', 'add', false, 'Invalid object', $this->response->type());
} else {
throw new NotFoundException(__('Invalid object'));
}
}
$fields = array('template_uuid', 'template_version', 'id');
$params = array(
'conditions' => array('Object.id' => $id),
'fields' => $fields,
'flatten' => 1,
);
// fetchObjects restrict access based on user
$object = $this->MispObject->fetchObjects($this->Auth->user(), $params);
if (empty($object)) {
if ($this->request->is('ajax')) {
return $this->RestResponse->saveFailResponse('Objects', 'add', false, 'Invalid object', $this->response->type());
} else {
throw new NotFoundException(__('Invalid object'));
}
} else {
$object = $object[0];
}
// get object attributes already set
$objectRelation = array();
foreach($object['Attribute'] as $attr) {
$objectRelation[$attr['object_relation']] = 1;
}
$objectRelation = array_keys($objectRelation);
// get object attribute defined in the object's template
$template = $this->MispObject->ObjectTemplate->find('first', array(
'conditions' => array(
'ObjectTemplate.uuid' => $object['Object']['template_uuid'],
'ObjectTemplate.version' => $object['Object']['template_version'],
),
'recursive' => -1,
'flatten' => 1,
'contain' => 'ObjectTemplateElement'
));
if (empty($template)) {
if ($this->request->is('ajax')) {
return $this->RestResponse->saveFailResponse('Objects', 'add', false, 'Invalid template', $this->response->type());
} else {
throw new NotFoundException(__('Invalid template'));
}
}
// unset object invalid object attribute
foreach($template['ObjectTemplateElement'] as $i => $objAttr) {
if (in_array($objAttr['object_relation'], $objectRelation) && !$objAttr['multiple']) {
unset($template['ObjectTemplateElement'][$i]);
}
}
if ($this->request->is('get') || $this->request->is('post')) {
$this->set('template', $template);
$this->set('objectId', $object['Object']['id']);
$items = array();
foreach ($template['ObjectTemplateElement'] as $objectAttribute) {
$name = sprintf('%s :: %s', $objectAttribute['object_relation'], $objectAttribute['type']);
$items[] = array(
'name' => $name,
'value' => '/objects/quickAddAttributeForm/' . $object['Object']['id'] . '/' . $objectAttribute['object_relation'],
'template' => array(
'name' => $name,
'infoExtra' => $objectAttribute['description'],
)
);
}
$this->set('options', array(
'flag_redraw_chosen' => true
));
$this->set('items', $items);
$this->render('/Elements/generic_picker');
} else {
return $template;
}
}
/**
* GET: Returns a form allowing to add a valid object attribute to an object
* POST/PUT: Add the attribute to the object
*/
public function quickAddAttributeForm($id, $fieldName = null) {
if ($this->request->is('GET')) {
if (!isset($fieldName)) {
throw new MethodNotAllowedException('No field requested.');
}
$this->MispObject->id = $id;
if (!$this->MispObject->exists()) {
throw new NotFoundException(__('Invalid object'));
}
$fields = array('template_uuid', 'template_version', 'id', 'event_id');
$params = array(
'conditions' => array('Object.id' => $id),
'fields' => $fields,
'flatten' => 1,
);
// fetchObjects restrict access based on user
$object = $this->MispObject->fetchObjects($this->Auth->user(), $params);
if (empty($object)) {
throw new NotFoundException(__('Invalid object'));
} else {
$object = $object[0];
}
$template = $this->MispObject->ObjectTemplate->find('first', array(
'conditions' => array(
'ObjectTemplate.uuid' => $object['Object']['template_uuid'],
'ObjectTemplate.version' => $object['Object']['template_version'],
),
'recursive' => -1,
'flatten' => 1,
'contain' => array(
'ObjectTemplateElement' => array('conditions' => array(
'object_relation' => $fieldName
))
)
));
if (empty($template)) {
throw new NotFoundException(__('Invalid object'));
}
if (empty($template['ObjectTemplateElement'])) {
throw new NotFoundException(__('Invalid fields') . ' `' . h($fieldName) . '`');
}
// check if fields can be added
foreach($object['Attribute'] as $i => $objAttr) {
$objectAttrFromTemplate = $template['ObjectTemplateElement'][0];
if ($objAttr['object_relation'] == $fieldName && !$objectAttrFromTemplate['multiple']) {
throw new NotFoundException(__('Invalid field'));
}
}
$template = $this->MispObject->prepareTemplate($template, $object);
$this->layout = 'ajax';
$this->set('object', $object['Object']);
$template_element = $template['ObjectTemplateElement'][0];
unset($template_element['value']); // avoid filling if multiple
$this->set('template_element', $template_element);
$distributionData = $this->MispObject->Event->Attribute->fetchDistributionData($this->Auth->user());
$this->set('distributionData', $distributionData);
$info = array();
foreach ($distributionData['levels'] as $key => $value) {
$info['distribution'][$key] = array('key' => $value, 'desc' => $this->MispObject->Event->Attribute->distributionDescriptions[$key]['formdesc']);
}
$this->set('info', $info);
$this->render('ajax/quickAddAttributeForm');
} else if ($this->request->is('post') || $this->request->is('put')) {
return $this->edit($this->request->data['Object']['id'], false, true);
}
}
public function delete($id, $hard = false)
{
$id = $this->Toolbox->findIdByUuid($this->MispObject, $id);

161
app/Controller/ServersController.php
View File

@ -250,6 +250,7 @@ class ServersController extends AppController
$defaults = array(
'push' => 0,
'pull' => 0,
'push_sightings' => 0,
'caching_enabled' => 0,
'json' => '[]',
'push_rules' => '[]',
@ -393,6 +394,7 @@ class ServersController extends AppController
}
$this->set('allTags', $allTags);
$this->set('host_org_id', Configure::read('MISP.host_org_id'));
$this->render('edit');
}
}
@ -444,7 +446,7 @@ class ServersController extends AppController
}
if (!$fail) {
// say what fields are to be updated
$fieldList = array('id', 'url', 'push', 'pull', 'caching_enabled', 'unpublish_event', 'publish_without_email', 'remote_org_id', 'name' ,'self_signed', 'cert_file', 'client_cert_file', 'push_rules', 'pull_rules', 'internal', 'skip_proxy');
$fieldList = array('id', 'url', 'push', 'pull', 'push_sightings', 'caching_enabled', 'unpublish_event', 'publish_without_email', 'remote_org_id', 'name' ,'self_signed', 'cert_file', 'client_cert_file', 'push_rules', 'pull_rules', 'internal', 'skip_proxy');
$this->request->data['Server']['id'] = $id;
if (isset($this->request->data['Server']['authkey']) && "" != $this->request->data['Server']['authkey']) {
$fieldList[] = 'authkey';
@ -663,13 +665,14 @@ class ServersController extends AppController
if (!Configure::read('MISP.background_jobs')) {
$result = $this->Server->pull($this->Auth->user(), $id, $technique, $s);
if (is_array($result)) {
$success = sprintf(__('Pull completed. %s events pulled, %s events could not be pulled, %s proposals pulled.', count($result[0]), count($result[1]), $result[2]));
$success = sprintf(__('Pull completed. %s events pulled, %s events could not be pulled, %s proposals pulled, %s sightings pulled.', count($result[0]), count($result[1]), $result[2], $result[3]));
} else {
$error = $result;
}
$this->set('successes', $result[0]);
$this->set('fails', $result[1]);
$this->set('pulledProposals', $result[2]);
$this->set('pulledSightings', $result[3]);
} else {
$this->loadModel('Job');
$this->Job->create();
@ -1052,6 +1055,7 @@ class ServersController extends AppController
// get the DB diagnostics
$dbDiagnostics = $this->Server->dbSpaceUsage();
$dbSchemaDiagnostics = $this->Server->dbSchemaDiagnostic();
$redisInfo = $this->Server->redisInfo();
@ -1065,7 +1069,7 @@ class ServersController extends AppController
$sessionStatus = $this->Server->sessionDiagnostics($diagnostic_errors, $sessionCount);
$this->set('sessionCount', $sessionCount);
$additionalViewVars = array('gpgStatus', 'sessionErrors', 'proxyStatus', 'sessionStatus', 'zmqStatus', 'stixVersion', 'cyboxVersion', 'mixboxVersion', 'maecVersion', 'stix2Version', 'pymispVersion', 'moduleStatus', 'yaraStatus', 'gpgErrors', 'proxyErrors', 'zmqErrors', 'stixOperational', 'stix', 'moduleErrors', 'moduleTypes', 'dbDiagnostics', 'redisInfo');
$additionalViewVars = array('gpgStatus', 'sessionErrors', 'proxyStatus', 'sessionStatus', 'zmqStatus', 'stixVersion', 'cyboxVersion', 'mixboxVersion', 'maecVersion', 'stix2Version', 'pymispVersion', 'moduleStatus', 'yaraStatus', 'gpgErrors', 'proxyErrors', 'zmqErrors', 'stixOperational', 'stix', 'moduleErrors', 'moduleTypes', 'dbDiagnostics', 'dbSchemaDiagnostics', 'redisInfo');
}
// check whether the files are writeable
$writeableDirs = $this->Server->writeableDirsDiagnostics($diagnostic_errors);
@ -1106,6 +1110,9 @@ class ServersController extends AppController
'writeableDirs' => $writeableDirs,
'writeableFiles' => $writeableFiles,
'readableFiles' => $readableFiles,
'dbDiagnostics' => $dbDiagnostics,
'dbSchemaDiagnostics' => $dbSchemaDiagnostics,
'redisInfo' => $redisInfo,
'finalSettings' => $dumpResults,
'extensions' => $extensions,
'workers' => $worker_array
@ -1129,6 +1136,9 @@ class ServersController extends AppController
$this->set('phpversion', phpversion());
$this->set('phpmin', $this->phpmin);
$this->set('phprec', $this->phprec);
$this->set('pythonmin', $this->pythonmin);
$this->set('pythonrec', $this->pythonrec);
$this->set('pymisp', $this->pymisp);
}
}
@ -1137,12 +1147,23 @@ class ServersController extends AppController
if (!$this->_isSiteAdmin() || !$this->request->is('post')) {
throw new MethodNotAllowedException();
}
$validTypes = array('default', 'email', 'scheduler', 'cache', 'prio');
$validTypes = array('default', 'email', 'scheduler', 'cache', 'prio', 'update');
if (!in_array($type, $validTypes)) {
throw new MethodNotAllowedException('Invalid worker type.');
}
$prepend = '';
if ($type != 'scheduler') {
$workerIssueCount = 0;
$workerDiagnostic = $this->Server->workerDiagnostics($workerIssueCount);
if ($type == 'update' && isset($workerDiagnostic['update']['ok']) && $workerDiagnostic['update']['ok']) {
$message = __('Only one `update` worker can run at a time');
if ($this->_isRest()) {
return $this->RestResponse->saveFailResponse('Servers', 'startWorker', false, $message, $this->response->type());
} else {
$this->Flash->error($message);
$this->redirect('/servers/serverSettings/workers');
}
}
shell_exec($prepend . APP . 'Console' . DS . 'cake CakeResque.CakeResque start --interval 5 --queue ' . $type .' > /dev/null 2>&1 &');
} else {
shell_exec($prepend . APP . 'Console' . DS . 'cake CakeResque.CakeResque startscheduler -i 5 > /dev/null 2>&1 &');
@ -1455,6 +1476,20 @@ class ServersController extends AppController
}
}
public function getRemoteUser($id)
{
$this->Server->id = $id;
if (!$this->Server->exists()) {
throw new NotFoundException(__('Invalid server'));
}
$user = $this->Server->getRemoteUser($id);
if (empty($user)) {
throw new NotFoundException(__('Invalid user or user not found.'));
} else {
return $this->RestResponse->viewData($user);
}
}
public function testConnection($id = false)
{
if (!$this->Auth->user('Role')['perm_sync'] && !$this->Auth->user('Role')['perm_site_admin']) {
@ -1472,6 +1507,10 @@ class ServersController extends AppController
if (isset($version['perm_sync'])) {
$perm_sync = $version['perm_sync'];
}
$perm_sighting = false;
if (isset($version['perm_sighting'])) {
$perm_sighting = $version['perm_sighting'];
}
App::uses('Folder', 'Utility');
$file = new File(ROOT . DS . 'VERSION.json', true);
$local_version = json_decode($file->read(), true);
@ -1498,10 +1537,14 @@ class ServersController extends AppController
if (!$mismatch && $version[2] < 111) {
$mismatch = 'proposal';
}
if (!$perm_sync) {
if (!$perm_sync && !$perm_sighting) {
$result['status'] = 7;
return new CakeResponse(array('body'=> json_encode($result), 'type' => 'json'));
}
if (!$perm_sync && $perm_sighting) {
$result['status'] = 8;
return new CakeResponse(array('body'=> json_encode($result), 'type' => 'json'));
}
return new CakeResponse(
array(
'body'=> json_encode(
@ -1599,7 +1642,7 @@ class ServersController extends AppController
throw new MethodNotAllowedException('This action requires API access.');
}
$versionArray = $this->Server->checkMISPVersion();
$this->set('response', array('version' => $versionArray['major'] . '.' . $versionArray['minor'] . '.' . $versionArray['hotfix'], 'perm_sync' => $this->userRole['perm_sync']));
$this->set('response', array('version' => $versionArray['major'] . '.' . $versionArray['minor'] . '.' . $versionArray['hotfix'], 'perm_sync' => $this->userRole['perm_sync'], 'perm_sighting' => $this->userRole['perm_sighting']));
$this->set('_serialize', 'response');
}
@ -1666,36 +1709,49 @@ class ServersController extends AppController
$this->set('updateLocked', $this->Server->isUpdateLocked());
}
public function updateProgress()
public function updateProgress($ajaxHtml=false)
{
if (!$this->_isSiteAdmin()) {
throw new MethodNotAllowedException('You are not authorised to do that.');
}
$update_progress = $this->Server->getUpdateProgress();
$current_index = $update_progress['current'];
$current_command = !isset($update_progress['commands'][$current_index]) ? '' : $update_progress['commands'][$current_index];
$lookup_string = preg_replace('/\s{2,}/', '', substr($current_command, 0, -1));
$sql_info = $this->Server->query("SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST;");
if (empty($sql_info)) {
$update_progress['process_list'] = array();
$this->AdminSetting = ClassRegistry::init('AdminSetting');
$dbVersion = $this->AdminSetting->getSetting('db_version');
$updateProgress = $this->Server->getUpdateProgress();
$updateProgress['db_version'] = $dbVersion;
$maxUpdateNumber = max(array_keys($this->Server->db_changes));
$updateProgress['complete_update_remaining'] = max($maxUpdateNumber - $dbVersion, 0);
$updateProgress['update_locked'] = $this->Server->isUpdateLocked();
$updateProgress['lock_remaining_time'] = $this->Server->getLockRemainingTime();
$updateProgress['update_fail_number_reached'] = $this->Server->UpdateFailNumberReached();
$currentIndex = $updateProgress['current'];
$currentCommand = !isset($updateProgress['commands'][$currentIndex]) ? '' : $updateProgress['commands'][$currentIndex];
$lookupString = preg_replace('/\s{2,}/', '', substr($currentCommand, 0, -1));
$sqlInfo = $this->Server->query("SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST;");
if (empty($sqlInfo)) {
$updateProgress['process_list'] = array();
} else {
// retrieve current update process
foreach($sql_info as $row) {
if (preg_replace('/\s{2,}/', '', $row['PROCESSLIST']['INFO']) == $lookup_string) {
$sql_info = $row['PROCESSLIST'];
foreach($sqlInfo as $row) {
if (preg_replace('/\s{2,}/', '', $row['PROCESSLIST']['INFO']) == $lookupString) {
$sqlInfo = $row['PROCESSLIST'];
break;
}
}
$update_progress['process_list'] = array();
$update_progress['process_list']['STATE'] = isset($sql_info['STATE']) ? $sql_info['STATE'] : '';
$update_progress['process_list']['PROGRESS'] = isset($sql_info['PROGRESS']) ? $sql_info['PROGRESS'] : 0;
$update_progress['process_list']['STAGE'] = isset($sql_info['STAGE']) ? $sql_info['STAGE'] : 0;
$update_progress['process_list']['MAX_STAGE'] = isset($sql_info['MAX_STAGE']) ? $sql_info['MAX_STAGE'] : 0;
$updateProgress['process_list'] = array();
$updateProgress['process_list']['STATE'] = isset($sqlInfo['STATE']) ? $sqlInfo['STATE'] : '';
$updateProgress['process_list']['PROGRESS'] = isset($sqlInfo['PROGRESS']) ? $sqlInfo['PROGRESS'] : 0;
$updateProgress['process_list']['STAGE'] = isset($sqlInfo['STAGE']) ? $sqlInfo['STAGE'] : 0;
$updateProgress['process_list']['MAX_STAGE'] = isset($sqlInfo['MAX_STAGE']) ? $sqlInfo['MAX_STAGE'] : 0;
}
if ($this->request->is('ajax')) {
return $this->RestResponse->viewData(h($update_progress), $this->response->type());
$this->set('ajaxHtml', $ajaxHtml);
if ($this->request->is('ajax') && $ajaxHtml) {
$this->set('updateProgress', $updateProgress);
$this->layout = false;
} elseif ($this->request->is('ajax') || $this->_isRest()) {
return $this->RestResponse->viewData(h($updateProgress), $this->response->type());
} else {
$this->set('updateProgress', $update_progress);
$this->set('updateProgress', $updateProgress);
}
}
@ -1899,16 +1955,16 @@ misp_verifycert = %s
relative_path = \'%s\'
body = %s
from pymisp import PyMISP
from pymisp import ExpandedPyMISP
misp = PyMISP(misp_url, misp_key, misp_verifycert)
misp = ExpandedPyMISP(misp_url, misp_key, misp_verifycert)
misp.direct_call(relative_path, body)
',
$baseurl,
$request['header']['Authorization'],
$verifyCert,
$relative,
(empty($request['body']) ? 'Null' : $request['body'])
(empty($request['body']) ? 'None' : $request['body'])
);
return $python_script;
}
@ -2144,4 +2200,53 @@ misp.direct_call(relative_path, body)
return $this->RestResponse->saveFailResponse('Servers', 'changePriority', $id, $message, $this->response->type());
}
}
public function releaseUpdateLock()
{
if (!$this->request->is('post')) {
throw new MethodNotAllowedException(__('This endpoint expects POST requests.'));
}
if (!$this->_isSiteAdmin()) {
throw new MethodNotAllowedException(__('Only site admin accounts can release the update lock.'));
}
$this->Server->changeLockState(false);
$this->Server->resetUpdateFailNumber();
$this->redirect(array('action' => 'updateProgress'));
}
public function dbSchemaDiagnostic()
{
if (!$this->_isSiteAdmin()) {
throw new MethodNotAllowedException(__('Only site admin accounts get the DB schema diagnostic.'));
}
$dbSchemaDiagnostics = $this->Server->dbSchemaDiagnostic();
if ($this->_isRest()) {
return $this->RestResponse->viewData($dbSchemaDiagnostics, $this->response->type());
} else {
$this->set('checkedTableColumn', $dbSchemaDiagnostics['checked_table_column']);
$this->set('dbSchemaDiagnostics', $dbSchemaDiagnostics['diagnostic']);
$this->set('dbIndexDiagnostics', $dbSchemaDiagnostics['diagnostic_index']);
$this->set('expectedDbVersion', $dbSchemaDiagnostics['expected_db_version']);
$this->set('actualDbVersion', $dbSchemaDiagnostics['actual_db_version']);
$this->set('error', $dbSchemaDiagnostics['error']);
$this->set('remainingLockTime', $dbSchemaDiagnostics['remaining_lock_time']);
$this->set('updateFailNumberReached', $dbSchemaDiagnostics['update_fail_number_reached']);
$this->set('updateLocked', $dbSchemaDiagnostics['update_locked']);
$this->set('dataSource', $dbSchemaDiagnostics['dataSource']);
$this->set('columnPerTable', $dbSchemaDiagnostics['columnPerTable']);
$this->set('indexes', $dbSchemaDiagnostics['indexes']);
$this->render('/Elements/healthElements/db_schema_diagnostic');
}
}
public function viewDeprecatedFunctionUse()
{
$data = $this->Deprecation->getDeprecatedAccessList($this->Server);
if ($this->_isRest()) {
return $this->RestResponse->viewData($data, $this->response->type());
} else {
$this->layout = false;
$this->set('data', $data);
}
}
}

36
app/Controller/ShadowAttributesController.php
View File

@ -96,7 +96,7 @@ class ShadowAttributesController extends AppController
$this->Attribute->delete($activeAttribute['Attribute']['id']);
} else {
// Update the live attribute with the shadow data
$fieldsToUpdate = array('value1', 'value2', 'value', 'type', 'category', 'comment', 'to_ids');
$fieldsToUpdate = array('value1', 'value2', 'value', 'type', 'category', 'comment', 'to_ids', 'first_seen', 'last_seen');
foreach ($fieldsToUpdate as $f) {
$activeAttribute['Attribute'][$f] = $shadow[$f];
}
@ -197,24 +197,6 @@ class ShadowAttributesController extends AppController
}
}
// If we accept a proposed attachment, then the attachment itself needs to be moved from files/eventId/shadow/shadowId to files/eventId/attributeId
private function _moveFile($shadowId, $newId, $eventId)
{
$attachments_dir = Configure::read('MISP.attachments_dir');
if (empty($attachments_dir)) {
$attachments_dir = $this->ShadowAttribute->getDefaultAttachments_dir();
}
$pathOld = $attachments_dir . DS . 'shadow' . DS . $shadowId;
$pathNew = $attachments_dir . DS . $newId;
if (rename($pathOld, $pathNew)) {
return true;
} else {
$this->Flash->error(__('Moving of the file that this attachment references failed.', true), 'default', array());
$this->redirect(array('controller' => 'events', 'action' => 'view', $eventId));
}
}
private function __discard($id)
{
$sa = $this->ShadowAttribute->find(
@ -433,7 +415,7 @@ class ShadowAttributesController extends AppController
array(
'conditions' => array('ShadowAttribute.id' => $this->ShadowAttribute->id),
'recursive' => -1,
'fields' => array('id', 'old_id', 'event_id', 'type', 'category', 'value', 'comment','to_ids', 'uuid', 'event_org_id', 'email', 'deleted', 'timestamp')
'fields' => array('id', 'old_id', 'event_id', 'type', 'category', 'value', 'comment','to_ids', 'uuid', 'event_org_id', 'email', 'deleted', 'timestamp', 'first_seen', 'last_seen')
)
);
$this->set('ShadowAttribute', $sa['ShadowAttribute']);
@ -707,12 +689,12 @@ class ShadowAttributesController extends AppController
if ($attachment) {
$fields = array(
'static' => array('old_id' => 'Attribute.id', 'uuid' => 'Attribute.uuid', 'event_id' => 'Attribute.event_id', 'event_uuid' => 'Event.uuid', 'event_org_id' => 'Event.orgc_id', 'category' => 'Attribute.category', 'type' => 'Attribute.type'),
'optional' => array('value', 'to_ids', 'comment')
'optional' => array('value', 'to_ids', 'comment', 'first_seen', 'last_seen')
);
} else {
$fields = array(
'static' => array('old_id' => 'Attribute.id', 'uuid' => 'Attribute.uuid', 'event_id' => 'Attribute.event_id', 'event_uuid' => 'Event.uuid', 'event_org_id' => 'Event.orgc_id'),
'optional' => array('category', 'type', 'value', 'to_ids', 'comment')
'optional' => array('category', 'type', 'value', 'to_ids', 'comment', 'first_seen', 'last_seen')
);
if ($existingAttribute['Attribute']['object_id']) {
unset($fields['optional']['type']);
@ -749,7 +731,7 @@ class ShadowAttributesController extends AppController
array(
'conditions' => array('ShadowAttribute.id' => $this->ShadowAttribute->id),
'recursive' => -1,
'fields' => array('id', 'old_id', 'event_id', 'type', 'category', 'value', 'comment','to_ids', 'uuid', 'event_org_id', 'email', 'deleted', 'timestamp')
'fields' => array('id', 'old_id', 'event_id', 'type', 'category', 'value', 'comment','to_ids', 'uuid', 'event_org_id', 'email', 'deleted', 'timestamp', 'first_seen', 'last_seen')
)
);
$this->set('ShadowAttribute', $sa['ShadowAttribute']);
@ -851,6 +833,8 @@ class ShadowAttributesController extends AppController
'type' => $existingAttribute['Attribute']['type'],
'to_ids' => $existingAttribute['Attribute']['to_ids'],
'value' => $existingAttribute['Attribute']['value'],
'first_seen' => $existingAttribute['Attribute']['first_seen'],
'last_seen' => $existingAttribute['Attribute']['last_seen'],
'email' => $this->Auth->user('email'),
'org_id' => $this->Auth->user('org_id'),
'proposal_to_delete' => true,
@ -891,7 +875,7 @@ class ShadowAttributesController extends AppController
'recursive' => -1,
'contain' => 'Event',
'fields' => array(
'ShadowAttribute.id', 'ShadowAttribute.old_id', 'ShadowAttribute.event_id', 'ShadowAttribute.type', 'ShadowAttribute.category', 'ShadowAttribute.uuid', 'ShadowAttribute.to_ids', 'ShadowAttribute.value', 'ShadowAttribute.comment', 'ShadowAttribute.org_id',
'ShadowAttribute.id', 'ShadowAttribute.old_id', 'ShadowAttribute.event_id', 'ShadowAttribute.type', 'ShadowAttribute.category', 'ShadowAttribute.uuid', 'ShadowAttribute.to_ids', 'ShadowAttribute.value', 'ShadowAttribute.comment', 'ShadowAttribute.org_id', 'ShadowAttribute.first_seen', 'ShadowAttribute.last_seen',
'Event.id', 'Event.orgc_id', 'Event.org_id', 'Event.distribution', 'Event.uuid'
),
'conditions' => array('AND' => array('ShadowAttribute.id' => $id, $distConditions, 'ShadowAttribute.deleted' => 0))
@ -962,7 +946,7 @@ class ShadowAttributesController extends AppController
}
$params = array(
'conditions' => $conditions,
'fields' => array('ShadowAttribute.id', 'ShadowAttribute.old_id', 'ShadowAttribute.event_id', 'ShadowAttribute.type', 'ShadowAttribute.category', 'ShadowAttribute.uuid', 'ShadowAttribute.to_ids', 'ShadowAttribute.value', 'ShadowAttribute.comment', 'ShadowAttribute.org_id', 'ShadowAttribute.timestamp'),
'fields' => array('ShadowAttribute.id', 'ShadowAttribute.old_id', 'ShadowAttribute.event_id', 'ShadowAttribute.type', 'ShadowAttribute.category', 'ShadowAttribute.uuid', 'ShadowAttribute.to_ids', 'ShadowAttribute.value', 'ShadowAttribute.comment', 'ShadowAttribute.org_id', 'ShadowAttribute.timestamp', 'ShadowAttribute.first_seen', 'ShadowAttribute.last_seen'),
'contain' => array(
'Event' => array(
'fields' => array('id', 'org_id', 'info', 'orgc_id', 'uuid'),
@ -1142,7 +1126,7 @@ class ShadowAttributesController extends AppController
}
}
$keys = array_flip(array('uuid', 'event_id', 'value', 'type', 'category', 'to_ids'));
$keys = array_flip(array('uuid', 'event_id', 'value', 'type', 'category', 'to_ids', 'first_seen', 'last_seen'));
$proposal = array_intersect_key($attribute['Attribute'], $keys);
$proposal['email'] = $this->Auth->user('email');

224
app/Controller/SightingdbController.php Normal file
View File

@ -0,0 +1,224 @@
<?php
App::uses('AppController', 'Controller');
class SightingdbController extends AppController
{
public $components = array('Session', 'RequestHandler');
public $paginate = array(
'limit' => 60,
'maxLimit' => 9999, // LATER we will bump here on a problem once we have more than 9999 events <- no we won't, this is the max a user van view/page.
'order' => array(
'Sightingdb.id' => 'DESC'
),
'recursive' => -1,
'contain' => array('SightingdbOrg' => 'Organisation')
);
public function beforeFilter()
{
parent::beforeFilter();
$this->Security->unlockedActions = array('search');
}
public function add()
{
if ($this->request->is('post')) {
if (empty($this->request->data['Sightingdb'])) {
$this->request->data = array('Sightingdb' => $this->request->data);
}
$this->Sightingdb->create();
$result = $this->Sightingdb->save($this->request->data);
$message = $result ? __('SightingDB connection added.') : __('SightingDB connection could not be added.');
if ($result) {
if (isset($this->request->data['Sightingdb']['org_id'])) {
$this->Sightingdb->SightingdbOrg->resetOrgs($this->Sightingdb->id, $this->request->data['Sightingdb']['org_id']);
}
}
if ($this->_isRest()) {
if ($result) {
return $this->RestResponse->saveSuccessResponse('Sightingdb', 'add', $this->response->type(), $message);
} else {
return $this->RestResponse->saveFailResponse('Sightingdb', 'add', $message, $this->response->type());
}
} else {
if ($result) {
$this->Flash->success($message);
$this->redirect(array('action' => 'index'));
} else {
$message .= __(' Reason: %s', json_encode($this->Sightingdb->validationErrors, true));
$this->Flash->error($message);
}
}
}
$orgs = $this->Sightingdb->SightingdbOrg->Organisation->find('list', array(
'conditions' => array('Organisation.local' => 1),
'order' => array('LOWER(Organisation.name)'),
'fields' => array('Organisation.id', 'Organisation.name')
));
$this->set('orgs', $orgs);
}
public function edit($id)
{
$existingEntry = $this->Sightingdb->find('first', array(
'recursive' => -1,
'conditions' => array('Sightingdb.id' => $id),
'contain' => array('SightingdbOrg.org_id')
));
$existingEntry = $this->Sightingdb->extractOrgIds($existingEntry);
if (empty($id) || empty($existingEntry)) {
throw new NotFoundException(__('Invalid SightingDB entry.'));
}
if ($this->request->is('post') || $this->request->is('put')) {
if (empty($this->request->data['Sightingdb'])) {
$this->request->data = array('Sightingdb' => $this->request->data);
}
$keys = array('host', 'port', 'description', 'name', 'owner', 'enabled', 'skip_proxy', 'ssl_skip_verification', 'namespace');
foreach ($keys as $key) {
if (!empty($this->request->data['Sightingdb'][$key])) {
$existingEntry['Sightingdb'][$key] = $this->request->data['Sightingdb'][$key];
}
}
$result = $this->Sightingdb->save($existingEntry);
if (isset($this->request->data['Sightingdb']['org_id'])) {
$this->Sightingdb->SightingdbOrg->resetOrgs($this->Sightingdb->id, $this->request->data['Sightingdb']['org_id']);
}
$message = $result ? __('SightingDB connection updated.') : __('SightingDB connection could not be updated.');
if ($this->_isRest()) {
if ($result) {
return $this->RestResponse->saveSuccessResponse('Sightingdb', 'edit', $id, $this->response->type(), $message);
} else {
return $this->RestResponse->saveFailResponse('Sightingdb', 'edit', $id, $message, $this->response->type());
}
} else {
if ($result) {
$this->Flash->success($message);
$this->redirect(array('action' => 'index'));
} else {
$message .= __(' Reason: %s', json_encode($this->Sightingdb->validationErrors, true));
$this->Flash->error($message);
}
}
} else {
$this->request->data = $existingEntry;
}
$orgs = $this->Sightingdb->SightingdbOrg->Organisation->find('list', array(
'conditions' => array('Organisation.local' => 1),
'order' => array('LOWER(Organisation.name)'),
'fields' => array('Organisation.id', 'Organisation.name')
));
$this->set('id', $id);
$this->set('orgs', $orgs);
$this->render('/Sightingdb/add');
}
public function delete($id)
{
$existingEntry = $this->Sightingdb->find('first', array(
'recursive' => -1,
'conditions' => array('Sightingdb.id' => $id)
));
if (empty($id) || empty($existingEntry)) {
throw new NotFoundException(__('Invalid SightingDB entry.'));
}
if ($this->request->is('post') || $this->request->is('delete')) {
$result = $this->Sightingdb->delete($existingEntry['Sightingdb']['id']);
if ($result) {
$message = __('SightingDB connection removed.');
} else {
$message = __('SightingDB connection could not be removed.');
}
if ($this->_isRest()) {
if ($result) {
return $this->RestResponse->saveSuccessResponse('Sightingdb', 'edit', $id, $this->response->type(), $message);
} else {
return $this->RestResponse->saveFailResponse('Sightingdb', 'edit', $id, $message, $this->response->type());
}
} else {
if ($result) {
$this->Flash->success($message);
$this->redirect(array('action' => 'index'));
} else {
$message .= __(' Reason: %s', json_encode($this->Sightingdb->validationErrors, true));
$this->Flash->error($message);
}
$this->redirect(array('action' => 'index'));
}
}
}
public function index()
{
$filters = $this->IndexFilter->harvestParameters(array('value'));
if (!empty($filters['value'])) {
if (is_array($filters['value'])) {
foreach ($filters['value'] as &$value) {
$value = '%' . strtolower($value) . '%';
}
} else {
$filters['value'] = '%' . strtolower($filters['value']) . '%';
}
$this->paginate['conditions']['AND'][] = array(
'OR' => array(
'Sightingdb.name LIKE' => $filters['value'],
'Sightingdb.owner LIKE' => $filters['value'],
'Sightingdb.host LIKE' => $filters['value']
)
);
}
if ($this->_isRest()) {
$params = array(
'contain' => $this->paginate['contain'],
'conditions' => empty($this->paginate['conditions']) ? array() : $this->paginate['conditions'],
);
$data = $this->Sightingdb->find('all', $params);
$data = $this->Sightingdb->extractOrgIdsFromList($data);
return $this->RestResponse->viewData($data, $this->response->type());
} else {
$this->set('data', $this->paginate());
}
}
public function requestStatus($id)
{
$result = $this->Sightingdb->requestStatus($id);
if (is_array($result)) {
return $this->RestResponse->viewData($result, $this->response->type());
} else {
return $this->RestResponse->saveFailResponse('Sightingdb', 'requestStatus', $id, $result, $this->response->type());
}
}
public function search($id)
{
if (empty($id)) {
throw new InvalidArgumentException(__('Pass a valid SightingDB ID'));
}
$sightingdb = $this->Sightingdb->find('first', array(
'recursive' => -1,
'conditions' => array('Sightingdb.id' => $id),
'contain' => array('SightingdbOrg')
));
if (empty($sightingdb)) {
throw new NotFoundException('Invalid sightingDB');
}
if (!empty($this->request->data['value'])) {
$requestValue = trim($this->request->data['value']);
$result = $this->Sightingdb->queryValues(array($requestValue => array()), $sightingdb);
if (!empty($result[$requestValue][$sightingdb['Sightingdb']['id']])) {
$result = $result[$requestValue][$sightingdb['Sightingdb']['id']];
$result = array(
'first_seen' => date('Y-m-d H:i:s', $result['first_seen']),
'last_seen' => date('Y-m-d H:i:s', $result['last_seen']),
'count' => $result['count']
);
} else {
$result = array('count' => 0);
}
} else {
$result = array('count' => 0);
}
return $this->RestResponse->viewData($result, $this->response->type());
}
}

65
app/Controller/SightingsController.php
View File

@ -66,7 +66,7 @@ class SightingsController extends AppController
$source = isset($this->request->data['source']) ? trim($this->request->data['source']) : '';
}
if (!$error) {
$result = $this->Sighting->saveSightings($id, $values, $timestamp, $this->Auth->user(), $type, $source);
$result = $this->Sighting->saveSightings($id, $values, $timestamp, $this->Auth->user(), $type, $source, false, true);
}
if (!is_numeric($result)) {
$error = $result;
@ -276,45 +276,6 @@ class SightingsController extends AppController
return $this->RestResponse->viewData($sightings);
}
public function restSearch($context = false)
{
$allowedContext = array(false, 'event', 'attribute');
$paramArray = array('returnFormat', 'id', 'type', 'from', 'to', 'last', 'org_id', 'source', 'includeAttribute', 'includeEvent');
$filterData = array(
'request' => $this->request,
'named_params' => $this->params['named'],
'paramArray' => $paramArray,
'ordered_url_params' => compact($paramArray)
);
$filters = $this->_harvestParameters($filterData, $exception);
// validate context
if (!in_array($context, $allowedContext, true)) {
throw new MethodNotAllowedException(_('Invalid context.'));
}
// ensure that an id is provided if context is set
if ($context !== false && !isset($filters['id'])) {
throw new MethodNotAllowedException(_('An id must be provided if the context is set.'));
}
$filters['context'] = $context;
if (!isset($returnFormat)) {
$returnFormat = 'json';
}
if (isset($filters['returnFormat'])) {
$returnFormat = $filters['returnFormat'];
}
if ($returnFormat === 'download') {
$returnFormat = 'json';
}
$sightings = $this->Sighting->restSearch($this->Auth->user(), $returnFormat, $filters);
$validFormats = $this->Sighting->validFormats;
$responseType = $validFormats[$returnFormat][0];
return $this->RestResponse->viewData($sightings, $responseType, false, true);
}
public function listSightings($id = false, $context = 'attribute', $org_id = false)
{
$rawId = $id;
@ -422,4 +383,28 @@ class SightingsController extends AppController
$this->layout = 'ajax';
$this->render('ajax/view_sightings');
}
// Save sightings synced over, restricted to sync users
public function bulkSaveSightings($eventId = false)
{
if ($this->request->is('post')) {
if (empty($this->request->data['Sighting'])) {
$sightings = $this->request->data;
} else {
$sightings = $this->request->data['Sighting'];
}
$saved = $this->Sighting->bulkSaveSightings($eventId, $sightings, $this->Auth->user());
if (is_numeric($saved)) {
if ($saved > 0) {
return new CakeResponse(array('body'=> json_encode(array('saved' => true, 'success' => $saved . ' sightings added.')), 'status' => 200, 'type' => 'json'));
} else {
return new CakeResponse(array('body'=> json_encode(array('saved' => false, 'success' => 'No sightings added.')), 'status' => 200, 'type' => 'json'));
}
} else {
throw new MethodNotAllowedException($saved);
}
} else {
throw new MethodNotAllowedException('This method is only accessible via POST requests.');
}
}
}

88
app/Controller/TagsController.php
View File

@ -459,6 +459,7 @@ class TagsController extends AppController
public function showEventTag($id)
{
$this->loadModel('EventTag');
$this->loadModel('Taxonomy');
if (!$this->EventTag->Event->checkIfAuthorised($this->Auth->user(), $id)) {
throw new MethodNotAllowedException('Invalid event.');
}
@ -487,6 +488,8 @@ class TagsController extends AppController
'conditions' => array('Event.id' => $id)
));
$this->set('required_taxonomies', $this->EventTag->Event->getRequiredTaxonomies());
$tagConflicts = $this->Taxonomy->checkIfTagInconsistencies($tags);
$this->set('tagConflicts', $tagConflicts);
$this->set('event', $event);
$this->layout = 'ajax';
$this->render('/Events/ajax/ajaxTags');
@ -496,6 +499,7 @@ class TagsController extends AppController
{
$this->helpers[] = 'TextColour';
$this->loadModel('AttributeTag');
$this->loadModel('Taxonomy');
$this->Tag->AttributeTag->Attribute->id = $id;
if (!$this->Tag->AttributeTag->Attribute->exists()) {
@ -528,6 +532,8 @@ class TagsController extends AppController
$this->set('event', $event);
$this->set('attributeTags', $attributeTags);
$this->set('attributeId', $id);
$tagConflicts = $this->Taxonomy->checkIfTagInconsistencies($attributeTags);
$this->set('tagConflicts', $tagConflicts);
$this->layout = 'ajax';
$this->render('/Attributes/ajax/ajaxAttributeTags');
}
@ -854,41 +860,56 @@ class TagsController extends AppController
$this->render('/Servers/json/simple');
}
private function __findObjectByUuid($object_uuid, &$type)
private function __findObjectByUuid($object_uuid, &$type, $scope = 'modify')
{
$this->loadModel('Event');
$object = $this->Event->find('first', array(
'conditions' => array(
'Event.uuid' => $object_uuid,
),
'fields' => array('Event.orgc_id', 'Event.id'),
'recursive' => -1
if (!$this->userRole['perm_tagger']) {
throw new MethodNotAllowedException(__('This functionality requires tagging permission.'));
}
$object = $this->Event->fetchEvent($this->Auth->user(), array(
'event_uuid' => $object_uuid,
'metadata' => 1
));
$type = 'Event';
if (!empty($object)) {
$object = $object[0];
if (
$scope !== 'view' &&
!$this->_isSiteAdmin() &&
!$this->userRole['perm_tagger'] &&
$object['Event']['orgc_id'] != $this->Auth->user('org_id')
) {
throw new MethodNotAllowedException('Invalid Target.');
$message = __('Cannot alter the tags of this data, only the organisation that has created the data (orgc) can modify global tags.');
if ($this->Auth->user('org_id') === Configure::read('MISP.host_org_id')) {
$message .= ' ' . __('Please consider using local tags if you are in the host organisation of the instance.');
}
throw new MethodNotAllowedException($message);
}
} else {
$type = 'Attribute';
$object = $this->Event->Attribute->find('first', array(
'conditions' => array(
'Attribute.uuid' => $object_uuid,
),
'fields' => array('Attribute.id'),
'recursive' => -1,
'contain' => array('Event.orgc_id')
));
$object = $this->Event->Attribute->fetchAttributes(
$this->Auth->user(),
array(
'conditions' => array(
'Attribute.uuid' => $object_uuid
),
'flatten' => 1
)
);
if (!empty($object)) {
if (!$this->_isSiteAdmin() && !$this->userRole['perm_tagger'] && $object['Event']['orgc_id'] != $this->Auth->user('org_id')) {
throw new MethodNotAllowedException('Invalid Target.');
$object = $object[0];
if (
$scope !== 'view' &&
!$this->_isSiteAdmin() &&
$object['Event']['orgc_id'] != $this->Auth->user('org_id')
) {
$message = __('Cannot alter the tags of this data, only the organisation that has created the data (orgc) can modify global tags.');
if ($this->Auth->user('org_id') === Configure::read('MISP.host_org_id')) {
$message .= ' ' . __('Please consider using local tags if you are in the host organisation of the instance.');
}
throw new MethodNotAllowedException($message);
}
} else {
throw new MethodNotAllowedException('Invalid Target.');
throw new MethodNotAllowedException(__('Invalid Target.'));
}
}
return $object;
@ -926,11 +947,11 @@ class TagsController extends AppController
$local = $this->request->data['local'];
}
}
if (!is_bool($local)) {
throw new InvalidArgumentException('Invalid local flag');
if (!empty($local) && $this->Auth->user('org_id') != Configure::read('MISP.host_org_id')) {
throw new MethodNotAllowedException(__('Local tags can only be added by users of the host organisation.'));
}
$objectType = '';
$object = $this->__findObjectByUuid($uuid, $objectType);
$object = $this->__findObjectByUuid($uuid, $objectType, $local ? 'view' : 'modify');
$existingTag = $this->Tag->find('first', array('conditions' => $conditions, 'recursive' => -1));
if (empty($existingTag)) {
if (!is_numeric($tag)) {
@ -938,7 +959,10 @@ class TagsController extends AppController
throw new MethodNotAllowedException('Tag not found and insufficient privileges to create it.');
}
$this->Tag->create();
$this->Tag->save(array('Tag' => array('name' => $tag, 'colour' => $this->Tag->random_color())));
$result = $this->Tag->save(array('Tag' => array('name' => $tag, 'colour' => $this->Tag->random_color())));
if (!$result) {
return $this->RestResponse->saveFailResponse('Tags', 'attachTagToObject', false, __('Unable to create tag. Reason: ' . json_encode($this->Tag->validationErrors)), $this->response->type());
}
$existingTag = $this->Tag->find('first', array('recursive' => -1, 'conditions' => array('Tag.id' => $this->Tag->id)));
} else {
throw new NotFoundException('Invalid Tag.');
@ -946,10 +970,10 @@ class TagsController extends AppController
}
if (!$this->_isSiteAdmin()) {
if (!in_array($existingTag['Tag']['org_id'], array(0, $this->Auth->user('org_id')))) {
throw new MethodNotAllowedException('Invalid Tag.');
throw new MethodNotAllowedException('Invalid Tag. This tag can only be set by a fixed organisation.');
}
if (!in_array($existingTag['Tag']['user_id'], array(0, $this->Auth->user('id')))) {
throw new MethodNotAllowedException('Invalid Tag.');
throw new MethodNotAllowedException('Invalid Tag. This tag can only be set by a fixed user.');
}
}
$this->loadModel($objectType);
@ -1029,9 +1053,9 @@ class TagsController extends AppController
throw new MethodNotAllowedException('Invalid Tag.');
}
$objectType = '';
$object = $this->__findObjectByUuid($uuid, $objectType);
$object = $this->__findObjectByUuid($uuid, $objectType, 'view');
if (empty($object)) {
throw new MethodNotAllowedException('Invalid Target.');
throw new MethodNotAllowedException(__('Invalid Target.'));
}
$connectorObject = $objectType . 'Tag';
$this->loadModel($objectType);
@ -1043,6 +1067,14 @@ class TagsController extends AppController
));
if (empty($existingAssociation)) {
throw new MethodNotAllowedException('Could not remove tag as it is not attached to the target ' . $objectType);
} else {
if (empty($existingAssociation[$objectType . 'Tag']['local'])) {
$object = $this->__findObjectByUuid($uuid, $objectType);
} else {
if ($object['Event']['orgc_id'] !== $this->Auth->user('org_id') && $this->Auth->user('org_id') != Configure::read('MISP.host_org_id')) {
throw new MethodNotAllowedException(__('Insufficient privileges to remove local tags from events you do not own.'));
}
}
}
$result = $this->$objectType->$connectorObject->delete($existingAssociation[$connectorObject]['id']);
if ($result) {

113
app/Controller/UsersController.php
View File

@ -74,16 +74,25 @@ class UsersController extends AppController
$temp[$v['setting']] = $v['value'];
}
$user['UserSetting'] = $temp;
return $this->RestResponse->viewData(array(
'User' => $user['User'],
'Role' => $user['Role'],
'UserSetting' => $user['UserSetting']
), $this->response->type());
return $this->RestResponse->viewData($this->__massageUserObject($user), $this->response->type());
} else {
$this->set('user', $user);
}
}
private function __massageUserObject($user)
{
unset($user['User']['server_id']);
$user['User']['password'] = '*****';
$objectsToInclude = array('User', 'Role', 'UserSetting', 'Organisation');
foreach ($objectsToInclude as $objectToInclude) {
if (isset($user[$objectToInclude])) {
$temp[$objectToInclude] = $user[$objectToInclude];
}
}
return $temp;
}
public function request_API()
{
if (Configure::read('MISP.disable_emailing')) {
@ -108,14 +117,20 @@ class UsersController extends AppController
if (!$this->_isAdmin() && Configure::read('MISP.disableUserSelfManagement')) {
throw new MethodNotAllowedException('User self-management has been disabled on this instance.');
}
$id = $this->Auth->user('id');
$this->User->read(null, $id);
if (!$this->User->exists()) {
$currentUser = $this->User->find('first', array(
'conditions' => array('User.id' => $this->Auth->user('id')),
'recursive' => -1
));
if (empty($currentUser)) {
throw new NotFoundException('Something went wrong. Your user account could not be accessed.');
}
$id = $currentUser['User']['id'];
if ($this->request->is('post') || $this->request->is('put')) {
if (empty($this->request->data['User'])) {
$this->request->data = array('User' => $this->request->data);
}
$abortPost = false;
if (!$this->_isSiteAdmin() && !empty($this->request->data['User']['email'])) {
if (!empty($this->request->data['User']['email']) && !$this->_isSiteAdmin()) {
$organisation = $this->User->Organisation->find('first', array(
'conditions' => array('Organisation.id' => $this->Auth->user('org_id')),
'recursive' => -1
@ -132,7 +147,7 @@ class UsersController extends AppController
}
}
if ($abortPost) {
$this->Flash->error(__('Invalid e-mail domain. Your user is restricted to creating users for the following domain(s): ') . implode(', ', $organisation['Organisation']['restricted_to_domain']));
$message = __('Invalid e-mail domain. Your user is restricted to creating users for the following domain(s): ') . implode(', ', $organisation['Organisation']['restricted_to_domain']);
}
}
}
@ -154,17 +169,49 @@ class UsersController extends AppController
if (!$abortPost) {
// What fields should be saved (allowed to be saved)
$fieldList = array('email', 'autoalert', 'gpgkey', 'certif_public', 'nids_sid', 'contactalert', 'disabled');
if ("" != $this->request->data['User']['password']) {
if (!empty($this->request->data['User']['password'])) {
$fieldList[] = 'password';
$fieldList[] = 'confirm_password';
}
foreach ($this->request->data['User'] as $k => $v) {
$currentUser['User'][$k] = $v;
}
// Save the data
if ($this->User->save($this->request->data, true, $fieldList)) {
$this->Flash->success(__('The profile has been updated'));
$this->_refreshAuth();
$this->redirect(array('action' => 'view', $id));
} else {
$this->Flash->error(__('The profile could not be updated. Please, try again.'));
if ($this->_isRest()) {
if (!empty($this->request->data['User']['password'])) {
if ($this->request->data['User']['password'] === '*****') {
unset($this->request->data['User']['password']);
} else {
$currentUser['User']['confirm_password'] = $this->request->data['User']['password'];
}
}
}
if ($this->User->save($currentUser, true, $fieldList)) {
if ($this->_isRest()) {
$user = $this->User->find('first', array(
'conditions' => array('User.id' => $id),
'recursive' => -1,
'contain' => array(
'Organisation',
'Role',
'UserSetting'
)
));
return $this->RestResponse->viewData($this->__massageUserObject($user), $this->response->type());
} else {
$this->Flash->success(__('The profile has been updated'));
$this->_refreshAuth();
$this->redirect(array('action' => 'view', $id));
}
} else {
$message = __('The profile could not be updated. Please, try again.');
$abortPost = true;
}
}
if ($abortPost) {
return $this->RestResponse->saveFailResponse('Users', 'edit', $id, $message, $this->response->type());
} else {
$this->Flash->error($message);
}
} else {
$this->User->set('password', '');
@ -483,7 +530,8 @@ class UsersController extends AppController
'conditions' => array('User.id' => $id),
'contain' => array(
'UserSetting',
'Role'
'Role',
'Organisation'
)
));
if (empty($user)) {
@ -516,9 +564,9 @@ class UsersController extends AppController
), $this->response->type());
return $this->RestResponse->viewData(array('User' => $user['User']), $this->response->type());
} else {
$temp = $this->User->data['User']['invited_by'];
$user2 = $this->User->find('first', array('conditions' => array('User.id' => $user['User']['invited_by']), 'recursive' => -1));
$this->set('id', $id);
$this->set('user2', $this->User->read(null, $temp));
$this->set('user2', $user2);
}
}
@ -838,9 +886,9 @@ class UsersController extends AppController
continue;
}
if ($field != 'confirm_password') {
array_push($fieldsOldValues, $this->User->field($field));
$fieldsOldValues[$field] = $this->User->field($field);
} else {
array_push($fieldsOldValues, $this->User->field('password'));
$fieldsOldValues[$field] = $this->User->field('password');
}
}
if (
@ -881,31 +929,28 @@ class UsersController extends AppController
}
$cP++;
}
array_push($fieldsNewValues, $newValueStr);
$fieldsNewValues[$field] = $newValueStr;
} else {
array_push($fieldsNewValues, $newValue);
$fieldsNewValues[$field] = $newValue;
}
} else {
array_push($fieldsNewValues, $this->data['User']['password']);
$fieldsNewValues[$field] = $this->data['User']['password'];
}
}
// compare
$fieldsResultStr = '';
$c = 0;
$fieldsResult = array();
foreach ($fields as $field) {
if (isset($fieldsOldValues[$c]) && $fieldsOldValues[$c] != $fieldsNewValues[$c]) {
if (isset($fieldsOldValues[$field]) && $fieldsOldValues[$field] != $fieldsNewValues[$field]) {
if ($field != 'confirm_password' && $field != 'enable_password') {
$fieldsResultStr = $fieldsResultStr . ', ' . $field . ' (' . $fieldsOldValues[$c] . ') => (' . $fieldsNewValues[$c] . ')';
$fieldsResult[$field] = array($fieldsOldValues[$field], $fieldsNewValues[$field]);
}
}
$c++;
}
$fieldsResultStr = substr($fieldsResultStr, 2);
$user = $this->User->find('first', array(
'recursive' => -1,
'conditions' => array('User.id' => $this->User->id)
));
$this->User->extralog($this->Auth->user(), "edit", "user", $fieldsResultStr, $user);
$this->User->extralog($this->Auth->user(), "edit", "user", $fieldsResult, $user);
if ($this->_isRest()) {
$user['User']['password'] = '******';
return $this->RestResponse->viewData($user, $this->response->type());
@ -1756,10 +1801,12 @@ class UsersController extends AppController
'group' => 'Event.orgc_id',
'conditions' => array('Event.orgc_id' => array_keys($orgs)),
'recursive' => -1,
'fields' => array('Event.orgc_id', 'count(*)')
'fields' => array('Event.orgc_id', 'count(*)', 'sum(Event.attribute_count) as attributeCount')
));
foreach ($events as $event) {
$orgs[$event['Event']['orgc_id']]['eventCount'] = $event[0]['count(*)'];
$orgs[$event['Event']['orgc_id']]['attributeCount'] = $event[0]['attributeCount'];
$orgs[$event['Event']['orgc_id']]['orgActivity'] = $this->User->getOrgActivity($event['Event']['orgc_id'], array('event_timestamp' => '365d'));
}
unset($events);
$orgs = Set::combine($orgs, '{n}.name', '{n}');

60
app/Lib/Export/HashesExport.php Normal file
View File

@ -0,0 +1,60 @@
<?php
class HashesExport
{
public $additional_params = array(
'flatten' => 1
);
public $validTypes = array(
'simple' => array(
'md5', 'sha1', 'sha256', 'sha224', 'sha512', 'sha512/224', 'sha512/256', 'ssdeep', 'imphash', 'tlsh',
'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'pehash', 'authentihash',
'impfuzzy'
),
'composite' => array(
'malware-sample', 'filename|md5', 'filename|sha1', 'filename|sha256', 'filename|sha224', 'filename|sha512',
'filename|sha512/224', 'filename|sha512/256', 'filename|ssdeep', 'filename|imphash', 'filename|tlsh',
'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'filename|pehash',
'filename|authentihash', 'filename|impfuzzy'
)
);
public function handler($data, $options = array())
{
if ($options['scope'] === 'Attribute') {
if (in_array($data['Attribute']['type'], $this->validTypes['composite'])) {
return explode('|', $data['Attribute']['value'])[1];
} else if (in_array($data['Attribute']['type'], $this->validTypes['simple'])) {
return $data['Attribute']['value'];
}
}
if ($options['scope'] === 'Event') {
$result = array();
foreach ($data['Attribute'] as $attribute) {
if (in_array($attribute['type'], $this->validTypes['composite'])) {
$result[] = explode('|', $attribute['value'])[1];
} else if (in_array($attribute['type'], $this->validTypes['simple'])) {
$result[] = $attribute['value'];
}
}
return implode($this->separator(), $result);
}
return '';
}
public function header($options = array())
{
return '';
}
public function footer()
{
return "\n";
}
public function separator()
{
return "\n";
}
}

4
app/Lib/Export/Stix1Export.php
View File

@ -15,13 +15,13 @@ class Stix1Export extends StixExport
$this->__org = escapeshellarg(Configure::read('MISP.org'));
$framing_file = $this->__scripts_dir . 'misp_framing.py ';
$my_server = ClassRegistry::init('Server');
return $my_server->getPythonVersion() . ' ' . $framing_file . $this->__return_type . ' ' . $this->__baseurl . ' ' . $this->__org . ' xml' . $this->__end_of_cmd;
return $my_server->getPythonVersion() . ' ' . $framing_file . $this->__return_type . ' ' . $this->__baseurl . ' ' . $this->__org . ' ' . $this->__return_format . ' ' . $this->__end_of_cmd;
}
protected function __parse_misp_events($filename)
{
$scriptFile = $this->__scripts_dir . $this->__script_name;
$my_server = ClassRegistry::init('Server');
return shell_exec($my_server->getPythonVersion() . ' ' . $scriptFile . ' ' . $filename . ' xml ' . $this->__baseurl . ' ' . $this->__org . $this->__end_of_cmd);
return shell_exec($my_server->getPythonVersion() . ' ' . $scriptFile . ' ' . $filename . ' ' . $this->__return_format . ' ' . $this->__baseurl . ' ' . $this->__org . $this->__end_of_cmd);
}
}

13
app/Lib/Export/StixExport.php
View File

@ -2,6 +2,11 @@
class StixExport
{
public $additional_params = array(
'includeEventTags' => 1,
'includeGalaxy' => 1
);
protected $__return_format = 'json';
protected $__scripts_dir = APP . 'files/scripts/';
protected $__end_of_cmd = ' 2>' . APP . 'tmp/logs/exec-errors.log';
protected $__return_type = null;
@ -54,6 +59,11 @@ class StixExport
public function header($options = array())
{
$this->__return_type = $options['returnFormat'];
if ($this->__return_type == 'stix-json') {
$this->__return_type = 'stix';
} else if ($this->__return_type == 'stix') {
$this->__return_format = 'xml';
}
$framing_cmd = $this->initiate_framing_params();
$randomFileName = $this->__generateRandomFileName();
$this->__tmp_dir = $this->__scripts_dir . 'tmp/';
@ -80,7 +90,8 @@ class StixExport
$decoded = json_decode($result, true);
if (!isset($decoded['success']) || !$decoded['success']) {
$this->__delete_temporary_files($f);
return 'Error while processing your query: ' . $decoded['error'];
$error = $decoded && !empty($decoded['error']) ? $decoded['error'] : $result;
return 'Error while processing your query: ' . $error;
}
$file = new File($this->__tmp_dir . $filename . '.out');
$stix_event = ($this->__return_type == 'stix') ? $file->read() : substr($file->read(), 1, -1);

8
app/Lib/Tools/EventGraphTool.php
View File

@ -232,12 +232,12 @@
$event = $this->__get_filtered_event($id);
$this->__json['items'] = array();
$this->__json['relations'] = array();
$this->__json['existing_object_relation'] = array();
if (empty($event)) {
return $this->__json;
}
if (!empty($event['Object'])) {
$object = $event['Object'];
} else {
@ -312,7 +312,7 @@
if (empty($event)) {
return $this->__json;
}
if (!empty($event['Object'])) {
$object = $event['Object'];
} else {
@ -419,7 +419,7 @@
if (empty($event)) {
return $this->__json;
}
if (!empty($event['Object'])) {
$object = $event['Object'];
} else {

138
app/Lib/Tools/EventTimelineTool.php Normal file
View File

@ -0,0 +1,138 @@
<?php
class EventTimelineTool
{
private $__lookupTables = array();
private $__user = false;
private $__json = array();
private $__eventModel = false;
private $__refModel = false;
# Will be use latter on
private $__related_events = array();
private $__related_attributes = array();
public function construct($eventModel, $user, $filterRules, $extended_view=0)
{
$this->__eventModel = $eventModel;
$this->__objectTemplateModel = $eventModel->Object->ObjectTemplate;
$this->__user = $user;
$this->__filterRules = $filterRules;
$this->__json = array();
$this->__extended_view = $extended_view;
$this->__lookupTables = array(
'analysisLevels' => $this->__eventModel->analysisLevels,
'distributionLevels' => $this->__eventModel->Attribute->distributionLevels
);
return true;
}
public function construct_for_ref($refModel, $user)
{
$this->__refModel = $refModel;
$this->__user = $user;
$this->__json = array();
return true;
}
private function __get_event($id)
{
$fullevent = $this->__eventModel->fetchEvent($this->__user, array('eventid' => $id, 'flatten' => 0, 'includeTagRelations' => 1, 'extended' => $this->__extended_view));
$event = array();
if (empty($fullevent)) {
return $event;
}
if (!empty($fullevent[0]['Object'])) {
$event['Object'] = $fullevent[0]['Object'];
} else {
$event['Object'] = array();
}
if (!empty($fullevent[0]['Attribute'])) {
$event['Attribute'] = $fullevent[0]['Attribute'];
} else {
$event['Attribute'] = array();
}
return $event;
}
public function get_timeline($id)
{
$event = $this->__get_event($id);
$this->__json['items'] = array();
if (empty($event)) {
return $this->__json;
}
if (!empty($event['Object'])) {
$object = $event['Object'];
} else {
$object = array();
}
if (!empty($event['Attribute'])) {
$attribute = $event['Attribute'];
} else {
$attribute = array();
}
// extract links and node type
foreach ($attribute as $attr) {
$toPush = array(
'id' => $attr['id'],
'uuid' => $attr['uuid'],
'content' => $attr['value'],
'event_id' => $attr['event_id'],
'group' => 'attribute',
'timestamp' => $attr['timestamp'],
'first_seen' => $attr['first_seen'],
'last_seen' => $attr['last_seen'],
);
$this->__json['items'][] = $toPush;
}
foreach ($object as $obj) {
$toPush_obj = array(
'id' => $obj['id'],
'uuid' => $obj['uuid'],
'content' => $obj['name'],
'group' => 'object',
'meta-category' => $obj['meta-category'],
'template_uuid' => $obj['template_uuid'],
'event_id' => $obj['event_id'],
'timestamp' => $obj['timestamp'],
'Attribute' => array(),
);
$toPush_obj['first_seen'] = $obj['first_seen'];
$toPush_obj['last_seen'] = $obj['last_seen'];
$toPush_obj['first_seen_overwrite'] = false;
$toPush_obj['last_seen_overwrite'] = false;
foreach ($obj['Attribute'] as $obj_attr) {
// replaced *_seen based on object attribute
if ($obj_attr['object_relation'] == 'first-seen' && is_null($toPush_obj['first_seen'])) {
$toPush_obj['first_seen'] = $obj_attr['value']; // replace first_seen of the object to seen of the element
$toPush_obj['first_seen_overwrite'] = true;
} elseif ($obj_attr['object_relation'] == 'last-seen' && is_null($toPush_obj['last_seen'])) {
$toPush_obj['last_seen'] = $obj_attr['value']; // replace last_seen of the object to seen of the element
$toPush_obj['last_seen_overwrite'] = true;
}
$toPush_attr = array(
'id' => $obj_attr['id'],
'uuid' => $obj_attr['uuid'],
'content' => $obj_attr['value'],
'contentType' => $obj_attr['object_relation'],
'event_id' => $obj_attr['event_id'],
'group' => 'object_attribute',
'timestamp' => $obj_attr['timestamp'],
);
$toPush_obj['Attribute'][] = $toPush_attr;
}
$this->__json['items'][] = $toPush_obj;
}
return $this->__json;
}
}

7
app/Lib/Tools/SyncTool.php
View File

@ -20,6 +20,9 @@ class SyncTool
$params['ssl_verify_peer'] = false;
}
}
if (!empty($server['Server']['skip_proxy'])) {
$params['skip_proxy'] = 1;
}
}
return $this->createHttpSocket($params);
@ -35,7 +38,7 @@ class SyncTool
* @return HttpSocket
* @throws Exception
*/
private function createHttpSocket($params = array())
public function createHttpSocket($params = array())
{
// Use own CA PEM file
$caPath = Configure::read('MISP.ca_path');
@ -49,7 +52,7 @@ class SyncTool
App::uses('HttpSocket', 'Network/Http');
$HttpSocket = new HttpSocket($params);
$proxy = Configure::read('Proxy');
if (isset($proxy['host']) && !empty($proxy['host'])) {
if (empty($params['skip_proxy']) && isset($proxy['host']) && !empty($proxy['host'])) {
$HttpSocket->configProxy($proxy['host'], $proxy['port'], $proxy['method'], $proxy['user'], $proxy['password']);
}
return $HttpSocket;

2
app/Locale/no/LC_MESSAGES/default.po
View File

@ -1,7 +1,7 @@
msgid ""
msgstr ""
"Project-Id-Version: misp\n"
"PO-Revision-Date: 2019-09-30 01:23\n"
"PO-Revision-Date: 2019-11-30 07:05\n"
"Last-Translator: Steve Clement (SteveClement)\n"
"Language-Team: Norwegian\n"
"MIME-Version: 1.0\n"

10
app/Model/AdminSetting.php
View File

@ -21,12 +21,9 @@ class AdminSetting extends AppModel
$setting_object = $this->find('first', array(
'conditions' => array('setting' => $setting)
));
if (!empty($setting_object)) {
$setting_object['AdminSetting']['value'] = $value;
} else {
$this->create();
$setting_object['AdminSetting'] = array('setting' => $setting, 'value' => $value);
}
$this->deleteAll(array('setting' => $setting));
$this->create();
$setting_object['AdminSetting'] = array('setting' => $setting, 'value' => $value);
if ($this->save($setting_object)) {
return true;
} else {
@ -34,6 +31,7 @@ class AdminSetting extends AppModel
}
}
public function getSetting($setting)
{
$setting_object = $this->find('first', array(

544
app/Model/AppModel.php
View File

@ -76,10 +76,22 @@ class AppModel extends Model
21 => false, 22 => false, 23 => false, 24 => false, 25 => false, 26 => false,
27 => false, 28 => false, 29 => false, 30 => false, 31 => false, 32 => false,
33 => false, 34 => false, 35 => false, 36 => false, 37 => false, 38 => false,
39 => false, 40 => false, 41 => false
39 => false, 40 => false, 41 => false, 42 => false, 43 => false, 44 => false,
45 => false, 46 => false
);
public $advanced_updates_description = array(
'seenOnAttributeAndObject' => array(
'title' => 'First seen/Last seen Attribute table',
'description' => 'Update the Attribute table to support first_seen and last_seen feature, with a microsecond resolution.',
'liveOff' => true, # should the instance be offline for users other than site_admin
'recommendBackup' => true, # should the update recommend backup
'exitOnError' => false, # should the update exit on error
'requirements' => 'MySQL version must be >= 5.6', # message stating the requirements necessary for the update
'record' => false, # should the update success be saved in the admin_table
// 'preUpdate' => 'seenOnAttributeAndObjectPreUpdate', # Function to execute before the update. If it throws an error, it cancels the update
'url' => '/servers/updateDatabase/seenOnAttributeAndObject/' # url pointing to the funcion performing the update
),
);
public $actions_description = array(
'verifyGnuPGkeys' => array(
@ -91,6 +103,12 @@ class AppModel extends Model
'title' => 'Database Cleanup Scripts',
'description' => 'If you run into an issue with an infinite upgrade loop (when upgrading from version ~2.4.50) that ends up filling your database with upgrade script log messages, run the following script.',
'url' => '/logs/pruneUpdateLogs/'
),
'releaseUpdateLock' => array(
'title' => 'Release update lock',
'description' => 'If your your database is locked and is not updating, unlock it here.',
'ignore_disabled' => true,
'url' => '/servers/releaseUpdateLock/'
)
);
@ -102,19 +120,41 @@ class AppModel extends Model
return true;
}
public function isAcceptedDatabaseError($errorMessage, $dataSource)
{
$isAccepted = false;
if ($dataSource == 'Database/Mysql') {
$errorDuplicateColumn = 'SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name';
$errorDuplicateIndex = 'SQLSTATE[42000]: Syntax error or access violation: 1061 Duplicate key name';
$errorDropIndex = "/SQLSTATE\[42000\]: Syntax error or access violation: 1091 Can't DROP '[\w]+'; check that column\/key exists/";
$isAccepted = substr($errorMessage, 0, strlen($errorDuplicateColumn)) === $errorDuplicateColumn ||
substr($errorMessage, 0, strlen($errorDuplicateIndex)) === $errorDuplicateIndex ||
preg_match($errorDropIndex, $errorMessage) !== 0;
} elseif ($dataSource == 'Database/Postgres') {
$errorDuplicateColumn = '/ERROR: column "[\w]+" specified more than once/';
$errorDuplicateIndex = '/ERROR: relation "[\w]+" already exists/';
$errorDropIndex = '/ERROR: index "[\w]+" does not exist/';
$isAccepted = preg_match($errorDuplicateColumn, $errorMessage) !== 0 ||
preg_match($errorDuplicateIndex, $errorMessage) !== 0 ||
preg_match($errorDropIndex, $errorMessage) !== 0;
}
return $isAccepted;
}
// Generic update script
// add special cases where the upgrade does more than just update the DB
// this could become useful in the future
public function updateMISP($command)
{
$dbUpdateSuccess = false;
switch ($command) {
case '2.4.20':
$this->updateDatabase($command);
$dbUpdateSuccess = $this->updateDatabase($command);
$this->ShadowAttribute = ClassRegistry::init('ShadowAttribute');
$this->ShadowAttribute->upgradeToProposalCorrelation();
break;
case '2.4.25':
$this->updateDatabase($command);
$dbUpdateSuccess = $this->updateDatabase($command);
$newFeeds = array(
array('provider' => 'CIRCL', 'name' => 'CIRCL OSINT Feed', 'url' => 'https://www.circl.lu/doc/misp/feed-osint', 'enabled' => 0),
);
@ -122,27 +162,27 @@ class AppModel extends Model
break;
case '2.4.27':
$newFeeds = array(
array('provider' => 'Botvrij.eu', 'name' => 'The Botvrij.eu Data','url' => 'http://www.botvrij.eu/data/feed-osint', 'enabled' => 0)
array('provider' => 'Botvrij.eu', 'name' => 'The Botvrij.eu Data','url' => 'https://www.botvrij.eu/data/feed-osint', 'enabled' => 0)
);
$this->__addNewFeeds($newFeeds);
break;
case '2.4.49':
$this->updateDatabase($command);
$dbUpdateSuccess = $this->updateDatabase($command);
$this->SharingGroup = ClassRegistry::init('SharingGroup');
$this->SharingGroup->correctSyncedSharingGroups();
$this->SharingGroup->updateRoaming();
break;
case '2.4.55':
$this->updateDatabase('addSightings');
$dbUpdateSuccess = $this->updateDatabase('addSightings');
break;
case '2.4.66':
$this->updateDatabase('2.4.66');
$dbUpdateSuccess = $this->updateDatabase('2.4.66');
$this->cleanCacheFiles();
$this->Sighting = Classregistry::init('Sighting');
$this->Sighting->addUuids();
break;
case '2.4.67':
$this->updateDatabase('2.4.67');
$dbUpdateSuccess = $this->updateDatabase('2.4.67');
$this->Sighting = Classregistry::init('Sighting');
$this->Sighting->addUuids();
$this->Sighting->deleteAll(array('NOT' => array('Sighting.type' => array(0, 1, 2))));
@ -160,15 +200,15 @@ class AppModel extends Model
$this->OrgBlacklist->save($value);
}
}
$this->updateDatabase($command);
$dbUpdateSuccess = $this->updateDatabase($command);
break;
case '2.4.86':
$this->MispObject = Classregistry::init('MispObject');
$this->MispObject->removeOrphanedObjects();
$this->updateDatabase($command);
$dbUpdateSuccess = $this->updateDatabase($command);
break;
case 5:
$this->updateDatabase($command);
$dbUpdateSuccess = $this->updateDatabase($command);
$this->Feed = Classregistry::init('Feed');
$this->Feed->setEnableFeedCachingDefaults();
break;
@ -177,7 +217,7 @@ class AppModel extends Model
$this->Server->restartWorkers();
break;
case 10:
$this->updateDatabase($command);
$dbUpdateSuccess = $this->updateDatabase($command);
$this->Role = Classregistry::init('Role');
$this->Role->setPublishZmq();
break;
@ -191,13 +231,17 @@ class AppModel extends Model
$this->__fixServerPullPushRules();
break;
case 38:
$this->updateDatabase($command);
$dbUpdateSuccess = $this->updateDatabase($command);
$this->__addServerPriority();
break;
case 46:
$dbUpdateSuccess = $this->updateDatabase('seenOnAttributeAndObject');
break;
default:
$this->updateDatabase($command);
$dbUpdateSuccess = $this->updateDatabase($command);
break;
}
return $dbUpdateSuccess;
}
private function __addServerPriority()
@ -236,51 +280,21 @@ class AppModel extends Model
}
// SQL scripts for updates
public function updateDatabase($command, $useWorker=false)
public function updateDatabase($command)
{
// Exit if updates are locked
if ($this->isUpdateLocked()) {
return false;
}
$this->__resetUpdateProgress();
// restart this function by a worker
if ($useWorker && Configure::read('MISP.background_jobs')) {
$job = ClassRegistry::init('Job');
$job->create();
$data = array(
'worker' => 'prio',
'job_type' => 'update_app',
'job_input' => 'command: ' . $command,
'status' => 0,
'retries' => 0,
'org_id' => 0,
'org' => '',
'message' => 'Updating.',
);
$job->save($data);
$jobId = $job->id;
$process_id = CakeResque::enqueue(
'prio',
'AdminShell',
array('updateApp', $command, $jobId),
true
);
$job->saveField('process_id', $process_id);
return true;
}
$this->Log = ClassRegistry::init('Log');
$liveOff = false;
$exitOnError = false;
if (isset($advanced_updates_description[$command])) {
$liveOff = isset($advanced_updates_description[$command]['liveOff']) ? $advanced_updates_description[$command]['liveOff'] : $liveOff;
$exitOnError = isset($advanced_updates_description[$command]['exitOnError']) ? $advanced_updates_description[$command]['exitOnError'] : $exitOnError;
if (isset($this->advanced_updates_description[$command])) {
$liveOff = isset($this->advanced_updates_description[$command]['liveOff']) ? $this->advanced_updates_description[$command]['liveOff'] : $liveOff;
$exitOnError = isset($this->advanced_updates_description[$command]['exitOnError']) ? $this->advanced_updates_description[$command]['exitOnError'] : $exitOnError;
}
$dataSourceConfig = ConnectionManager::getDataSource('default')->config;
$dataSource = $dataSourceConfig['datasource'];
$sqlArray = array();
$indexArray = array();
$this->Log = ClassRegistry::init('Log');
$clean = true;
switch ($command) {
case 'extendServerOrganizationLength':
@ -1268,6 +1282,43 @@ class AppModel extends Model
$sqlArray[] = "ALTER TABLE `roles` ADD `enforce_rate_limit` tinyint(1) NOT NULL DEFAULT 0;";
$sqlArray[] = "ALTER TABLE `roles` ADD `rate_limit_count` int(11) NOT NULL DEFAULT 0;";
break;
case 42:
$sqlArray[] = "CREATE TABLE IF NOT EXISTS sightingdbs (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) NOT NULL,
`description` text,
`owner` varchar(255) DEFAULT '',
`host` varchar(255) DEFAULT 'http://localhost',
`port` int(11) DEFAULT 9999,
`timestamp` int(11) NOT NULL,
`enabled` tinyint(1) NOT NULL DEFAULT 0,
`skip_proxy` tinyint(1) NOT NULL DEFAULT 0,
`ssl_skip_verification` tinyint(1) NOT NULL DEFAULT 0,
PRIMARY KEY (id),
INDEX `name` (`name`),
INDEX `owner` (`owner`),
INDEX `host` (`host`),
INDEX `port` (`port`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;";
$sqlArray[] = "CREATE TABLE IF NOT EXISTS sightingdb_orgs (
`id` int(11) NOT NULL AUTO_INCREMENT,
`sightingdb_id` int(11) NOT NULL,
`org_id` int(11) NOT NULL,
PRIMARY KEY (id),
INDEX `sightingdb_id` (`sightingdb_id`),
INDEX `org_id` (`org_id`)
) ENGINE=InnoDB;";
break;
case 43:
$sqlArray[] = "ALTER TABLE sightingdbs ADD namespace varchar(255) DEFAULT '';";
break;
case 44:
$sqlArray[] = "ALTER TABLE object_template_elements CHANGE `disable_correlation` `disable_correlation` tinyint(1);";
break;
case 45:
$sqlArray[] = "ALTER TABLE `events` ADD `sighting_timestamp` int(11) NOT NULL DEFAULT 0 AFTER `publish_timestamp`;";
$sqlArray[] = "ALTER TABLE `servers` ADD `push_sightings` tinyint(1) NOT NULL DEFAULT 0 AFTER `pull`;";
break;
case 'fixNonEmptySharingGroupID':
$sqlArray[] = 'UPDATE `events` SET `sharing_group_id` = 0 WHERE `distribution` != 4;';
$sqlArray[] = 'UPDATE `attributes` SET `sharing_group_id` = 0 WHERE `distribution` != 4;';
@ -1284,13 +1335,64 @@ class AppModel extends Model
$sqlArray[] = 'ALTER TABLE `threads` DROP `org`;';
$sqlArray[] = 'ALTER TABLE `users` DROP `org`;';
break;
case 'seenOnAttributeAndObject':
$sqlArray[] =
"ALTER TABLE `attributes`
DROP INDEX uuid,
DROP INDEX event_id,
DROP INDEX sharing_group_id,
DROP INDEX type,
DROP INDEX category,
DROP INDEX value1,
DROP INDEX value2,
DROP INDEX object_id,
DROP INDEX object_relation;
";
$sqlArray[] = "ALTER TABLE `attributes` DROP INDEX deleted"; // deleted index may not be present
$sqlArray[] = "ALTER TABLE `attributes` DROP INDEX comment"; // for replayability
$sqlArray[] = "ALTER TABLE `attributes` DROP INDEX first_seen"; // for replayability
$sqlArray[] = "ALTER TABLE `attributes` DROP INDEX last_seen"; // for replayability
$sqlArray[] =
"ALTER TABLE `attributes`
ADD COLUMN `first_seen` BIGINT(20) NULL DEFAULT NULL,
ADD COLUMN `last_seen` BIGINT(20) NULL DEFAULT NULL,
MODIFY comment TEXT COLLATE utf8_unicode_ci
;";
$indexArray[] = array('attributes', 'uuid');
$indexArray[] = array('attributes', 'event_id');
$indexArray[] = array('attributes', 'sharing_group_id');
$indexArray[] = array('attributes', 'type');
$indexArray[] = array('attributes', 'category');
$indexArray[] = array('attributes', 'value1', 255);
$indexArray[] = array('attributes', 'value2', 255);
$indexArray[] = array('attributes', 'object_id');
$indexArray[] = array('attributes', 'object_relation');
$indexArray[] = array('attributes', 'deleted');
$indexArray[] = array('attributes', 'first_seen');
$indexArray[] = array('attributes', 'last_seen');
$sqlArray[] = "
ALTER TABLE `objects`
ADD `first_seen` BIGINT(20) NULL DEFAULT NULL,
ADD `last_seen` BIGINT(20) NULL DEFAULT NULL,
MODIFY comment TEXT COLLATE utf8_unicode_ci
;";
$indexArray[] = array('objects', 'first_seen');
$indexArray[] = array('objects', 'last_seen');
$sqlArray[] = "
ALTER TABLE `shadow_attributes`
ADD `first_seen` BIGINT(20) NULL DEFAULT NULL,
ADD `last_seen` BIGINT(20) NULL DEFAULT NULL,
MODIFY comment TEXT COLLATE utf8_unicode_ci
;";
$indexArray[] = array('shadow_attributes', 'first_seen');
$indexArray[] = array('shadow_attributes', 'last_seen');
break;
default:
return false;
break;
}
$now = new DateTime();
$this->__changeLockState(time());
// switch MISP instance live to false
if ($liveOff) {
$this->Server = Classregistry::init('Server');
@ -1300,14 +1402,14 @@ class AppModel extends Model
$sql_update_count = count($sqlArray);
$index_update_count = count($indexArray);
$total_update_count = $sql_update_count + $index_update_count;
$this->__setUpdateProgress(0, $total_update_count);
$this->__setUpdateProgress(0, $total_update_count, $command);
$str_index_array = array();
foreach($indexArray as $toIndex) {
$str_index_array[] = __('Indexing ') . implode($toIndex, '->');
$str_index_array[] = __('Indexing ') . sprintf('%s -> %s', $toIndex[0], $toIndex[1]);
}
$this->__setUpdateCmdMessages(array_merge($sqlArray, $str_index_array));
$flag_stop = false;
$error_count = 0;
$flagStop = false;
$errorCount = 0;
// execute test before update. Exit if it fails
if (isset($this->advanced_updates_description[$command]['preUpdate'])) {
@ -1317,15 +1419,15 @@ class AppModel extends Model
} catch (Exception $e) {
$this->__setPreUpdateTestState(false);
$this->__setUpdateProgress(0, false);
$this->__setUpdateResMessages(0, __('Issues executing the pre-update test `') . $function_name . __('`. The returned error is: ') . PHP_EOL . $e->getMessage());
$this->__setUpdateResMessages(0, sprintf(__('Issues executing the pre-update test `%s`. The returned error is: %s'), $function_name, $e->getMessage()) . PHP_EOL);
$this->__setUpdateError(0);
$error_count++;
$errorCount++;
$exitOnError = true;
$flag_stop = true;
$flagStop = true;
}
}
if (!$flag_stop) {
if (!$flagStop) {
$this->__setPreUpdateTestState(true);
foreach ($sqlArray as $i => $sql) {
try {
@ -1340,47 +1442,63 @@ class AppModel extends Model
'action' => 'update_database',
'user_id' => 0,
'title' => __('Successfuly executed the SQL query for ') . $command,
'change' => __('The executed SQL query was: ') . $sql
'change' => sprintf(__('The executed SQL query was: %s'), $sql)
));
$this->__setUpdateResMessages($i, __('Successfuly executed the SQL query for ') . $command);
$this->__setUpdateResMessages($i, sprintf(__('Successfuly executed the SQL query for %s'), $command));
} catch (Exception $e) {
$errorMessage = $e->getMessage();
$this->Log->create();
$this->Log->save(array(
$logMessage = array(
'org' => 'SYSTEM',
'model' => 'Server',
'model_id' => 0,
'email' => 'SYSTEM',
'action' => 'update_database',
'user_id' => 0,
'title' => __('Issues executing the SQL query for ') . $command,
'change' => __('The executed SQL query was: ') . $sql . PHP_EOL . __(' The returned error is: ') . $e->getMessage()
));
$this->__setUpdateResMessages($i, __('Issues executing the SQL query for ') . $command . __('. The returned error is: ') . PHP_EOL . $e->getMessage());
$this->__setUpdateError($i);
$error_count++;
if ($exitOnError) {
$flag_stop = true;
break;
'title' => sprintf(__('Issues executing the SQL query for %s'), $command),
'change' => __('The executed SQL query was: ') . $sql . PHP_EOL . __(' The returned error is: ') . $errorMessage
);
$this->__setUpdateResMessages($i, sprintf(__('Issues executing the SQL query for `%s`. The returned error is: ' . PHP_EOL . '%s'), $command, $errorMessage));
if (!$this->isAcceptedDatabaseError($errorMessage, $dataSource)) {
$this->__setUpdateError($i);
$errorCount++;
if ($exitOnError) {
$flagStop = true;
break;
}
} else {
$logMessage['change'] = $logMessage['change'] . PHP_EOL . __('However, as this error is whitelisted, the update went through.');
}
$this->Log->save($logMessage);
}
}
}
if (!$flag_stop) {
if (!empty($indexArray)) {
if ($clean) {
$this->cleanCacheFiles();
}
foreach ($indexArray as $i => $iA) {
$this->__setUpdateProgress(count($sqlArray)+$i, false);
if (isset($iA[2])) {
$this->__addIndex($iA[0], $iA[1], $iA[2]);
} else {
$this->__addIndex($iA[0], $iA[1]);
}
$this->__setUpdateResMessages(count($sqlArray)+$i, __('Successfuly indexed ') . implode($iA, '->'));
}
}
$this->__setUpdateProgress(count($sqlArray)+count($indexArray), false);
if (!$flagStop) {
if (!empty($indexArray)) {
if ($clean) {
$this->cleanCacheFiles();
}
foreach ($indexArray as $i => $iA) {
$this->__setUpdateProgress(count($sqlArray)+$i, false);
if (isset($iA[2])) {
$indexSuccess = $this->__addIndex($iA[0], $iA[1], $iA[2]);
} else {
$indexSuccess = $this->__addIndex($iA[0], $iA[1]);
}
if ($indexSuccess['success']) {
$this->__setUpdateResMessages(count($sqlArray)+$i, __('Successfuly indexed ') . sprintf('%s -> %s', $iA[0], $iA[1]));
} else {
$this->__setUpdateResMessages(count($sqlArray)+$i, sprintf('%s %s %s %s',
__('Failed to add index'),
sprintf('%s -> %s', $iA[0], $iA[1]),
__('The returned error is:') . PHP_EOL,
$indexSuccess['errorMessage']
));
$this->__setUpdateError(count($sqlArray)+$i);
}
}
}
$this->__setUpdateProgress(count($sqlArray)+count($indexArray), false);
}
if ($clean) {
$this->cleanCacheFiles();
@ -1389,10 +1507,23 @@ class AppModel extends Model
$liveSetting = 'MISP.live';
$this->Server->serverSettingsSaveValue($liveSetting, true);
}
if (!$flag_stop && $error_count == 0) {
if (!$flagStop && $errorCount == 0) {
$this->__postUpdate($command);
}
$this->__changeLockState(false);
if ($flagStop && $errorCount > 0) {
$this->Log->create();
$this->Log->save(array(
'org' => 'SYSTEM',
'model' => 'Server',
'model_id' => 0,
'email' => 'SYSTEM',
'action' => 'update_database',
'user_id' => 0,
'title' => sprintf(__('Issues executing the SQL query for %s'), $command),
'change' => __('Database updates stopped as some errors occured and the stop flag is enabled.')
));
return false;
}
return true;
}
@ -1460,10 +1591,12 @@ class AppModel extends Model
}
$result = true;
$duplicate = false;
$errorMessage = '';
try {
$this->query($addIndex);
} catch (Exception $e) {
$duplicate = (strpos($e->getMessage(), '1061') !== false);
$errorMessage = $e->getMessage();
$result = false;
}
$this->Log->create();
@ -1474,9 +1607,14 @@ class AppModel extends Model
'email' => 'SYSTEM',
'action' => 'update_database',
'user_id' => 0,
'title' => ($result ? 'Added index ' : 'Failed to add index ') . $field . ' to ' . $table . ($duplicate ? ' (index already set)' : ''),
'change' => ($result ? 'Added index ' : 'Failed to add index ') . $field . ' to ' . $table . ($duplicate ? ' (index already set)' : ''),
'title' => ($result ? 'Added index ' : 'Failed to add index ') . $field . ' to ' . $table . ($duplicate ? ' (index already set)' : $errorMessage),
'change' => ($result ? 'Added index ' : 'Failed to add index ') . $field . ' to ' . $table . ($duplicate ? ' (index already set)' : $errorMessage),
));
$additionResult = array('success' => $result || $duplicate);
if (!$result) {
$additionResult['errorMessage'] = $errorMessage;
}
return $additionResult;
}
public function cleanCacheFiles()
@ -1497,7 +1635,6 @@ class AppModel extends Model
public function checkMISPVersion()
{
App::uses('Folder', 'Utility');
$file = new File(ROOT . DS . 'VERSION.json', true);
$version_array = json_decode($file->read(), true);
$file->close();
@ -1568,9 +1705,35 @@ class AppModel extends Model
return true;
}
public function runUpdates($verbose = false)
// Try to create a table with a BIGINT(20)
public function seenOnAttributeAndObjectPreUpdate() {
$sqlArray[] = "CREATE TABLE IF NOT EXISTS testtable (
`testfield` BIGINT(6) NULL DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;";
try {
foreach($sqlArray as $i => $sql) {
$this->query($sql);
}
} catch (Exception $e) {
throw new Exception('Pre update test failed: ' . PHP_EOL . $sql . PHP_EOL . ' The returned error is: ' . $e->getMessage());
}
// clean up
$sqlArray[] = "DROP TABLE testtable;";
foreach($sqlArray as $i => $sql) {
$this->query($sql);
}
}
public function failingPreUpdate() {
throw new Exception('Yolo fail');
}
public function runUpdates($verbose = false, $useWorker = true, $processId = false)
{
$this->AdminSetting = ClassRegistry::init('AdminSetting');
$this->Job = ClassRegistry::init('Job');
$this->Log = ClassRegistry::init('Log');
$this->Server = ClassRegistry::init('Server');
$db = ConnectionManager::getDataSource('default');
$tables = $db->listSources();
$requiresLogout = false;
@ -1591,22 +1754,136 @@ class AppModel extends Model
}
$db_version = $db_version[0];
$updates = $this->__findUpgrades($db_version['AdminSetting']['value']);
$job = $this->Job->find('first', array(
'conditions' => array('Job.id' => $processId)
));
if (!empty($updates)) {
// Exit if updates are locked.
// This is not as reliable as a real lock implementation
// However, as all updates are re-playable, there is no harm if they
// get played multiple time. The purpose of this lightweight lock
// is only to limit the load.
if ($this->isUpdateLocked()) { // prevent creation of useless workers
$this->Log->create();
$this->Log->save(array(
'org' => 'SYSTEM',
'model' => 'Server',
'model_id' => 0,
'email' => 'SYSTEM',
'action' => 'update_db_worker',
'user_id' => 0,
'title' => __('Issues executing run_updates'),
'change' => __('Database updates are locked. Worker not spawned')
));
if (!empty($job)) { // if multiple prio worker is enabled, want to mark them as done
$job['Job']['progress'] = 100;
$job['Job']['message'] = __('Update done');
$this->Job->save($job);
}
return true;
}
// restart this function by a worker
if ($useWorker && Configure::read('MISP.background_jobs')) {
$workerIssueCount = 0;
$workerDiagnostic = $this->Server->workerDiagnostics($workerIssueCount);
$workerType = '';
if (isset($workerDiagnostic['update']['ok']) && $workerDiagnostic['update']['ok']) {
$workerType = 'update';
} elseif (isset($workerDiagnostic['prio']['ok']) && $workerDiagnostic['prio']['ok']) {
$workerType = 'prio';
} else { // no worker running, doing inline update
return $this->runUpdates($verbose, false);
}
$this->Job->create();
$data = array(
'worker' => $workerType,
'job_type' => 'run_updates',
'job_input' => 'command: ' . implode(',', $updates),
'status' => 0,
'retries' => 0,
'org_id' => 0,
'org' => '',
'message' => 'Updating.',
);
$this->Job->save($data);
$jobId = $this->Job->id;
$processId = CakeResque::enqueue(
'prio',
'AdminShell',
array('runUpdates', $jobId),
true
);
$this->Job->saveField('process_id', $processId);
return true;
}
// See comment above for `isUpdateLocked()`
// prevent continuation of job if worker was already spawned
// (could happens if multiple prio workers are up)
if ($this->isUpdateLocked()) {
$this->Log->create();
$this->Log->save(array(
'org' => 'SYSTEM',
'model' => 'Server',
'model_id' => 0,
'email' => 'SYSTEM',
'action' => 'update_db_worker',
'user_id' => 0,
'title' => __('Issues executing run_updates'),
'change' => __('Updates are locked. Stopping worker gracefully')
));
if (!empty($job)) {
$job['Job']['progress'] = 100;
$job['Job']['message'] = __('Update done');
$this->Job->save($job);
}
return true;
}
$this->changeLockState(time());
$this->__resetUpdateProgress();
$update_done = 0;
foreach ($updates as $update => $temp) {
if ($verbose) {
echo str_pad('Executing ' . $update, 30, '.');
}
$this->updateMISP($update);
if (!empty($job)) {
$job['Job']['progress'] = floor($update_done / count($updates) * 100);
$job['Job']['message'] = sprintf(__('Running update %s'), $update);
$this->Job->save($job);
}
$dbUpdateSuccess = $this->updateMISP($update);
if ($temp) {
$requiresLogout = true;
}
$db_version['AdminSetting']['value'] = $update;
$this->AdminSetting->save($db_version);
if ($dbUpdateSuccess) {
$db_version['AdminSetting']['value'] = $update;
$this->AdminSetting->save($db_version);
$this->resetUpdateFailNumber();
} else {
$this->__increaseUpdateFailNumber();
}
if ($verbose) {
echo "\033[32mDone\033[0m" . PHP_EOL;
}
$update_done++;
}
if (!empty($job)) {
$job['Job']['message'] = __('Update done');
}
$this->changeLockState(false);
$this->__queueCleanDB();
} else {
if (!empty($job)) {
$job['Job']['message'] = __('Update done in another worker. Gracefuly stopping.');
}
}
// mark current worker as done, as well as queued workers than manages to pass the locks
// (happens if user hit reload before first worker start its job)
if (!empty($job)) {
$job['Job']['progress'] = 100;
$this->Job->save($job);
}
}
if ($requiresLogout) {
@ -1615,7 +1892,7 @@ class AppModel extends Model
return true;
}
private function __setUpdateProgress($current, $total=false)
private function __setUpdateProgress($current, $total=false, $toward_db_version=false)
{
$updateProgress = $this->getUpdateProgress();
$updateProgress['current'] = $current;
@ -1625,6 +1902,9 @@ class AppModel extends Model
$now = new DateTime();
$updateProgress['time']['started'][$current] = $now->format('Y-m-d H:i:s');
}
if ($toward_db_version !== false) {
$updateProgress['toward_db_version'] = $toward_db_version;
}
$this->__saveUpdateProgress($updateProgress);
}
@ -1642,16 +1922,22 @@ class AppModel extends Model
$this->__saveUpdateProgress($updateProgress);
}
private function __resetUpdateProgress()
private function __getEmptyUpdateMessage()
{
$updateProgress = array(
return array(
'commands' => array(),
'results' => array(),
'time' => array('started' => array(), 'elapsed' => array()),
'current' => '',
'total' => '',
'failed_num' => array()
'failed_num' => array(),
'toward_db_version' => ''
);
}
private function __resetUpdateProgress()
{
$updateProgress = $this->__getEmptyUpdateMessage();
$this->__saveUpdateProgress($updateProgress);
}
@ -1681,13 +1967,13 @@ class AppModel extends Model
if ($updateProgress !== false) {
$updateProgress = json_decode($updateProgress, true);
} else {
$this->__resetUpdateProgress();
$updateProgress = $this->AdminSetting->getSetting('update_progress');
$updateProgress = json_decode($updateProgress, true);
$updateProgress = $this->__getEmptyUpdateMessage();
}
foreach($updateProgress as $setting => $value) {
if (!is_array($value)) {
$value = $value !== false && $value !== '' ? intval($value) : 0;
if (is_numeric($value)) {
$value = intval($value);
}
}
$updateProgress[$setting] = $value;
}
@ -1696,12 +1982,18 @@ class AppModel extends Model
private function __saveUpdateProgress($updateProgress)
{
if (!isset($this->AdminSetting)) {
$this->AdminSetting = ClassRegistry::init('AdminSetting');
}
$data = json_encode($updateProgress);
$this->AdminSetting->changeSetting('update_progress', $data);
}
private function __changeLockState($locked)
public function changeLockState($locked)
{
if (!isset($this->AdminSetting)) {
$this->AdminSetting = ClassRegistry::init('AdminSetting');
}
$this->AdminSetting->changeSetting('update_locked', $locked);
}
@ -1714,7 +2006,7 @@ class AppModel extends Model
return is_null($locked) ? false : $locked;
}
public function isUpdateLocked()
public function getLockRemainingTime()
{
$lockState = $this->getUpdateLockState();
if ($lockState !== false && $lockState !== '') {
@ -1727,11 +2019,43 @@ class AppModel extends Model
$this->Server = ClassRegistry::init('Server');
$updateWaitThreshold = intval($this->Server->serverSettings['MISP']['updateTimeThreshold']['value']);
}
if ($diffSec < $updateWaitThreshold) {
return true;
}
$remainingTime = $updateWaitThreshold - $diffSec;
return $remainingTime > 0 ? $remainingTime : 0;
} else {
return 0;
}
return false;
}
public function isUpdateLocked()
{
$remainingTime = $this->getLockRemainingTime();
$failThresholdReached = $this->UpdateFailNumberReached();
return $remainingTime > 0 || $failThresholdReached;
}
public function getUpdateFailNumber()
{
$this->AdminSetting = ClassRegistry::init('AdminSetting');
$updateFailNumber = $this->AdminSetting->getSetting('update_fail_number');
return ($updateFailNumber !== false && $updateFailNumber !== '') ? $updateFailNumber : 0;
}
public function resetUpdateFailNumber()
{
$this->AdminSetting = ClassRegistry::init('AdminSetting');
$this->AdminSetting->changeSetting('update_fail_number', 0);
}
public function __increaseUpdateFailNumber()
{
$this->AdminSetting = ClassRegistry::init('AdminSetting');
$updateFailNumber = $this->AdminSetting->getSetting('update_fail_number');
$this->AdminSetting->changeSetting('update_fail_number', $updateFailNumber+1);
}
public function UpdateFailNumberReached()
{
return $this->getUpdateFailNumber() > 3;
}
private function __queueCleanDB()

235
app/Model/Attribute.php
View File

@ -35,7 +35,7 @@ class Attribute extends AppModel
);
public $defaultFields = array(
'id', 'event_id', 'object_id', 'object_relation', 'category', 'type', 'value', 'to_ids', 'uuid', 'timestamp', 'distribution', 'sharing_group_id', 'comment', 'deleted', 'disable_correlation'
'id', 'event_id', 'object_id', 'object_relation', 'category', 'type', 'value', 'to_ids', 'uuid', 'timestamp', 'distribution', 'sharing_group_id', 'comment', 'deleted', 'disable_correlation', 'first_seen', 'last_seen'
);
public $distributionDescriptions = array(
@ -89,16 +89,16 @@ class Attribute extends AppModel
'Payload delivery' => array(
'desc' => __('Information about how the malware is delivered'),
'formdesc' => __('Information about the way the malware payload is initially delivered, for example information about the email or web-site, vulnerability used, originating IP etc. Malware sample itself should be attached here.'),
'types' => array('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', 'sha512/224', 'sha512/256', 'ssdeep', 'imphash', 'impfuzzy', 'authentihash', 'pehash', 'tlsh', 'cdhash', 'filename', 'filename|md5', 'filename|sha1', 'filename|sha224', 'filename|sha256', 'filename|sha384', 'filename|sha512', 'filename|sha512/224', 'filename|sha512/256', 'filename|authentihash', 'filename|ssdeep', 'filename|tlsh', 'filename|imphash','filename|impfuzzy', 'filename|pehash', 'mac-address', 'mac-eui-64', 'ip-src', 'ip-dst', 'ip-dst|port', 'ip-src|port', 'hostname', 'domain', 'email-src', 'email-dst', 'email-subject', 'email-attachment', 'email-body', 'url', 'user-agent', 'AS', 'pattern-in-file', 'pattern-in-traffic', 'stix2-pattern', 'yara', 'sigma', 'mime-type', 'attachment', 'malware-sample', 'link', 'malware-type', 'comment', 'text', 'hex', 'vulnerability', 'weakness', 'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'ja3-fingerprint-md5', 'hassh-md5', 'hasshserver-md5', 'other', 'hostname|port', 'email-dst-display-name', 'email-src-display-name', 'email-header', 'email-reply-to', 'email-x-mailer', 'email-mime-boundary', 'email-thread-index', 'email-message-id', 'mobile-application-id', 'whois-registrant-email', 'anonymised')
'types' => array('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', 'sha512/224', 'sha512/256', 'ssdeep', 'imphash', 'impfuzzy', 'authentihash', 'pehash', 'tlsh', 'cdhash', 'filename', 'filename|md5', 'filename|sha1', 'filename|sha224', 'filename|sha256', 'filename|sha384', 'filename|sha512', 'filename|sha512/224', 'filename|sha512/256', 'filename|authentihash', 'filename|ssdeep', 'filename|tlsh', 'filename|imphash','filename|impfuzzy', 'filename|pehash', 'mac-address', 'mac-eui-64', 'ip-src', 'ip-dst', 'ip-dst|port', 'ip-src|port', 'hostname', 'domain', 'email-src', 'email-dst', 'email-subject', 'email-attachment', 'email-body', 'url', 'user-agent', 'AS', 'pattern-in-file', 'pattern-in-traffic', 'stix2-pattern', 'yara', 'sigma', 'mime-type', 'attachment', 'malware-sample', 'link', 'malware-type', 'comment', 'text', 'hex', 'vulnerability', 'weakness', 'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'ja3-fingerprint-md5', 'hassh-md5', 'hasshserver-md5', 'other', 'hostname|port', 'email-dst-display-name', 'email-src-display-name', 'email-header', 'email-reply-to', 'email-x-mailer', 'email-mime-boundary', 'email-thread-index', 'email-message-id', 'mobile-application-id', 'chrome-extension-id', 'whois-registrant-email', 'anonymised')
),
'Artifacts dropped' => array(
'desc' => __('Any artifact (files, registry keys etc.) dropped by the malware or other modifications to the system'),
'types' => array('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', 'sha512/224', 'sha512/256', 'ssdeep', 'imphash', 'impfuzzy', 'authentihash', 'cdhash', 'filename', 'filename|md5', 'filename|sha1', 'filename|sha224', 'filename|sha256', 'filename|sha384', 'filename|sha512', 'filename|sha512/224', 'filename|sha512/256', 'filename|authentihash', 'filename|ssdeep', 'filename|tlsh', 'filename|imphash', 'filename|impfuzzy','filename|pehash', 'regkey', 'regkey|value', 'pattern-in-file', 'pattern-in-memory','pdb', 'stix2-pattern', 'yara', 'sigma', 'attachment', 'malware-sample', 'named pipe', 'mutex', 'windows-scheduled-task', 'windows-service-name', 'windows-service-displayname', 'comment', 'text', 'hex', 'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'other', 'cookie', 'gene', 'mime-type', 'anonymised')
'types' => array('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', 'sha512/224', 'sha512/256', 'ssdeep', 'imphash', 'impfuzzy', 'authentihash', 'cdhash', 'filename', 'filename|md5', 'filename|sha1', 'filename|sha224', 'filename|sha256', 'filename|sha384', 'filename|sha512', 'filename|sha512/224', 'filename|sha512/256', 'filename|authentihash', 'filename|ssdeep', 'filename|tlsh', 'filename|imphash', 'filename|impfuzzy','filename|pehash', 'regkey', 'regkey|value', 'pattern-in-file', 'pattern-in-memory','pdb', 'stix2-pattern', 'yara', 'sigma', 'attachment', 'malware-sample', 'named pipe', 'mutex', 'windows-scheduled-task', 'windows-service-name', 'windows-service-displayname', 'comment', 'text', 'hex', 'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'other', 'cookie', 'gene', 'kusto-query', 'mime-type', 'anonymised')
),
'Payload installation' => array(
'desc' => __('Info on where the malware gets installed in the system'),
'formdesc' => __('Location where the payload was placed in the system and the way it was installed. For example, a filename|md5 type attribute can be added here like this: c:\\windows\\system32\\malicious.exe|41d8cd98f00b204e9800998ecf8427e.'),
'types' => array('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', 'sha512/224', 'sha512/256', 'ssdeep', 'imphash', 'impfuzzy', 'authentihash', 'pehash', 'tlsh', 'cdhash', 'filename', 'filename|md5', 'filename|sha1', 'filename|sha224', 'filename|sha256', 'filename|sha384', 'filename|sha512', 'filename|sha512/224', 'filename|sha512/256', 'filename|authentihash', 'filename|ssdeep', 'filename|tlsh', 'filename|imphash', 'filename|impfuzzy', 'filename|pehash', 'pattern-in-file', 'pattern-in-traffic', 'pattern-in-memory', 'stix2-pattern', 'yara', 'sigma', 'vulnerability', 'weakness', 'attachment', 'malware-sample', 'malware-type', 'comment', 'text', 'hex', 'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'mobile-application-id', 'other', 'mime-type', 'anonymised')
'types' => array('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', 'sha512/224', 'sha512/256', 'ssdeep', 'imphash', 'impfuzzy', 'authentihash', 'pehash', 'tlsh', 'cdhash', 'filename', 'filename|md5', 'filename|sha1', 'filename|sha224', 'filename|sha256', 'filename|sha384', 'filename|sha512', 'filename|sha512/224', 'filename|sha512/256', 'filename|authentihash', 'filename|ssdeep', 'filename|tlsh', 'filename|imphash', 'filename|impfuzzy', 'filename|pehash', 'pattern-in-file', 'pattern-in-traffic', 'pattern-in-memory', 'stix2-pattern', 'yara', 'sigma', 'vulnerability', 'weakness', 'attachment', 'malware-sample', 'malware-type', 'comment', 'text', 'hex', 'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'mobile-application-id', 'chrome-extension-id', 'other', 'mime-type', 'anonymised')
),
'Persistence mechanism' => array(
'desc' => __('Mechanisms used by the malware to start at boot'),
@ -107,7 +107,7 @@ class Attribute extends AppModel
),
'Network activity' => array(
'desc' => __('Information about network traffic generated by the malware'),
'types' => array('ip-src', 'ip-dst', 'ip-dst|port', 'ip-src|port', 'port', 'hostname', 'domain', 'domain|ip', 'mac-address', 'mac-eui-64', 'email-dst', 'url', 'uri', 'user-agent', 'http-method', 'AS', 'snort', 'pattern-in-file', 'stix2-pattern', 'pattern-in-traffic', 'attachment', 'comment', 'text', 'x509-fingerprint-md5', 'x509-fingerprint-sha1', 'x509-fingerprint-sha256', 'ja3-fingerprint-md5', 'hassh-md5', 'hasshserver-md5', 'other', 'hex', 'cookie', 'hostname|port', 'bro', 'zeek', 'anonymised', 'community-id', 'email-subject')
'types' => array('ip-src', 'ip-dst', 'ip-dst|port', 'ip-src|port', 'port', 'hostname', 'domain', 'domain|ip', 'mac-address', 'mac-eui-64', 'email-dst', 'email-src', 'eppn', 'url', 'uri', 'user-agent', 'http-method', 'AS', 'snort', 'pattern-in-file', 'stix2-pattern', 'pattern-in-traffic', 'attachment', 'comment', 'text', 'x509-fingerprint-md5', 'x509-fingerprint-sha1', 'x509-fingerprint-sha256', 'ja3-fingerprint-md5', 'hassh-md5', 'hasshserver-md5', 'other', 'hex', 'cookie', 'hostname|port', 'bro', 'zeek', 'anonymised', 'community-id', 'email-subject')
),
'Payload type' => array(
'desc' => __('Information about the final payload(s)'),
@ -135,7 +135,7 @@ class Attribute extends AppModel
'Social network' => array(
'desc' => __('Social networks and platforms'),
// email-src and email-dst or should we go with a new email type that is neither / both?
'types' => array('github-username', 'github-repository', 'github-organisation', 'jabber-id', 'twitter-id', 'email-src', 'email-dst', 'comment', 'text', 'other', 'whois-registrant-email', 'anonymised')
'types' => array('github-username', 'github-repository', 'github-organisation', 'jabber-id', 'twitter-id', 'email-src', 'email-dst', 'eppn','comment', 'text', 'other', 'whois-registrant-email', 'anonymised')
),
'Person' => array(
'desc' => __('A human being - natural person'),
@ -165,6 +165,7 @@ class Attribute extends AppModel
'domain' => array('desc' => __('A domain name used in the malware'), 'formdesc' => __("A domain name used in the malware. Use this instead of hostname when the upper domain is important or can be used to create links between events."), 'default_category' => 'Network activity', 'to_ids' => 1),
'domain|ip' => array('desc' => __('A domain name and its IP address (as found in DNS lookup) separated by a |'),'formdesc' => __("A domain name and its IP address (as found in DNS lookup) separated by a | (no spaces)"), 'default_category' => 'Network activity', 'to_ids' => 1),
'email-src' => array('desc' => __("The email address used to send the malware."), 'default_category' => 'Payload delivery', 'to_ids' => 1),
'eppn' => array('desc' => __("eduPersonPrincipalName - eppn - the NetId of the person for the purposes of inter-institutional authentication. Should be stored in the form of user@univ.edu, where univ.edu is the name of the local security domain."), 'default_category' => 'Network activity', 'to_ids' => 1),
'email-dst' => array('desc' => __("A recipient email address"), 'formdesc' => __("A recipient email address that is not related to your constituency."), 'default_category' => 'Network activity', 'to_ids' => 1),
'email-subject' => array('desc' => __("The subject of the email"), 'default_category' => 'Payload delivery', 'to_ids' => 0),
'email-attachment' => array('desc' => __("File name of the email attachment."), 'default_category' => 'Payload delivery', 'to_ids' => 1),
@ -190,6 +191,7 @@ class Attribute extends AppModel
'stix2-pattern' => array('desc' => __('STIX 2 pattern'), 'default_category' => 'Payload installation', 'to_ids' => 1),
'sigma' => array('desc' => __('Sigma - Generic Signature Format for SIEM Systems'), 'default_category' => 'Payload installation', 'to_ids' => 1),
'gene' => array('desc' => __('GENE - Go Evtx sigNature Engine'), 'default_category' => 'Artifacts dropped', 'to_ids' => 0),
'kusto-query' => array('desc' => __('Kusto query - Kusto from Microsoft Azure is a service for storing and running interactive analytics over Big Data.'), 'default_category' => 'Artifacts dropped', 'to_ids' => 0),
'mime-type' => array('desc' => __('A media type (also MIME type and content type) is a two-part identifier for file formats and format contents transmitted on the Internet'), 'default_category' => 'Artifacts dropped', 'to_ids' => 0),
'identity-card-number' => array('desc' => __('Identity card number'), 'default_category' => 'Person', 'to_ids' => 0),
'cookie' => array('desc' => __('HTTP cookie as often stored on the user web client. This can include authentication cookie or session cookie.'), 'default_category' => 'Network activity', 'to_ids' => 0),
@ -316,6 +318,7 @@ class Attribute extends AppModel
'place-port-of-onward-foreign-destination' => array('desc' => __('A Port where the passenger is transiting to'), 'default_category' => 'Person', 'to_ids' => 0),
'passenger-name-record-locator-number' => array('desc' => __('The Passenger Name Record Locator is a key under which the reservation for a trip is stored in the system. The PNR contains, among other data, the name, flight segments and address of the passenger. It is defined by a combination of five or six letters and numbers.'), 'default_category' => 'Person', 'to_ids' => 0),
'mobile-application-id' => array('desc' => __('The application id of a mobile application'), 'default_category' => 'Payload delivery', 'to_ids' => 1),
'chrome-extension-id' => array('desc' => __('Chrome extension id'), 'default_category' => 'Payload delivery', 'to_ids' => 1),
'cortex' => array('desc' => __('Cortex analysis result'), 'default_category' => 'External analysis', 'to_ids' => 0),
'boolean' => array('desc' => __('Boolean value - to be used in objects'), 'default_category' => 'Other', 'to_ids' => 0),
'anonymised' => array('desc' => __('Anonymised value - described with the anonymisation object via a relationship'), 'formdesc' => __('Anonymised value - described with the anonymisation object via a relationship.'), 'default_category' => 'Other', 'to_ids' => 0)
@ -373,7 +376,9 @@ class Attribute extends AppModel
'deleted',
'disable_correlation',
'object_id',
'object_relation'
'object_relation',
'first_seen',
'last_seen'
);
public $searchResponseTypes = array(
@ -401,6 +406,7 @@ class Attribute extends AppModel
'suricata' => array('txt', 'NidsSuricataExport', 'rules'),
'snort' => array('txt', 'NidsSnortExport', 'rules'),
'text' => array('txt', 'TextExport', 'txt'),
'hashes' => array('txt', 'HashesExport', 'txt'),
'yara' => array('txt', 'YaraExport', 'yara'),
'yara-json' => array('json', 'YaraExport', 'json'),
'rpz' => array('txt', 'RPZExport', 'rpz'),
@ -437,6 +443,7 @@ class Attribute extends AppModel
'mac-eui-64' => 'Network activity',
'hostname' => 'Network activity',
'domain' => 'Network activity',
'eppn' => 'Network activity',
'url' => 'Network activity',
'ja3-fingerprint-md5' => 'Network activity',
'hassh-md5' => 'Network activity',
@ -523,6 +530,16 @@ class Attribute extends AppModel
'rule' => array('inList', array('0', '1', '2', '3', '4', '5')),
'message' => 'Options: Your organisation only, This community only, Connected communities, All communities, Sharing group, Inherit event',
'required' => true
),
'first_seen' => array(
'rule' => array('datetimeOrNull'),
'required' => false,
'message' => array('Invalid ISO 8601 format')
),
'last_seen' => array(
'rule' => array('datetimeOrNull'),
'required' => false,
'message' => array('Invalid ISO 8601 format')
)
);
@ -601,6 +618,7 @@ class Attribute extends AppModel
if (isset($v['Attribute']['object_relation']) && $v['Attribute']['object_relation'] === null) {
$results[$k]['Attribute']['object_relation'] = '';
}
$results[$k] = $this->UTCToISODatetime($results[$k], $this->alias);
}
return $results;
}
@ -632,6 +650,13 @@ class Attribute extends AppModel
$this->data['Attribute']['value2'] = '';
}
}
$this->data = $this->ISODatetimeToUTC($this->data, $this->alias);
// update correlation... (only needed here if there's an update)
if ($this->id || !empty($this->data['Attribute']['id'])) {
$this->__beforeSaveCorrelation($this->data['Attribute']);
}
// always return true after a beforeSave()
return true;
}
@ -813,9 +838,11 @@ class Attribute extends AppModel
{
parent::beforeValidate();
if (!isset($this->data['Attribute']['type'])) {
$this->validationErrors['type'] = ['No type set.'];
return false;
}
if (is_array($this->data['Attribute']['value'])) {
$this->validationErrors['type'] = ['Value is an array.'];
return false;
}
App::uses('ComplexTypeTool', 'Tools');
@ -823,6 +850,7 @@ class Attribute extends AppModel
$this->data['Attribute']['value'] = $this->complexTypeTool->refangValue($this->data['Attribute']['value'], $this->data['Attribute']['type']);
if (!empty($this->data['Attribute']['object_id']) && empty($this->data['Attribute']['object_relation'])) {
$this->validationErrors['type'] = ['Object attribute sent, but no object_relation set.'];
return false;
}
// remove leading and trailing blanks
@ -847,6 +875,16 @@ class Attribute extends AppModel
$date = new DateTime();
$this->data['Attribute']['timestamp'] = $date->getTimestamp();
}
// parse first_seen different formats
if (isset($this->data['Attribute']['first_seen'])) {
$this->data['Attribute']['first_seen'] = $this->data['Attribute']['first_seen'] === '' ? null : $this->data['Attribute']['first_seen'];
}
// parse last_seen different formats
if (isset($this->data['Attribute']['last_seen'])) {
$this->data['Attribute']['last_seen'] = $this->data['Attribute']['last_seen'] === '' ? null : $this->data['Attribute']['last_seen'];
}
// TODO: add explanatory comment
// TODO: i18n?
$result = $this->runRegexp($this->data['Attribute']['type'], $this->data['Attribute']['value']);
@ -979,6 +1017,20 @@ class Attribute extends AppModel
return $this->runValidation($value, $this->data['Attribute']['type']);
}
// check whether the variable is null or datetime
public function datetimeOrNull($fields)
{
$k = array_keys($fields)[0];
$seen = $fields[$k];
try {
new DateTime($seen);
$returnValue = true;
} catch (Exception $e) {
$returnValue = false;
}
return $returnValue || is_null($seen);
}
private $__hexHashLengths = array(
'authentihash' => 64,
'md5' => 32,
@ -1200,6 +1252,7 @@ class Attribute extends AppModel
}
break;
case 'email-src':
case 'eppn':
case 'email-dst':
case 'target-email':
case 'whois-registrant-email':
@ -1255,6 +1308,7 @@ class Attribute extends AppModel
case 'stix2-pattern':
case 'sigma':
case 'gene':
case 'kusto-query':
case 'mime-type':
case 'identity-card-number':
case 'cookie':
@ -1338,6 +1392,7 @@ class Attribute extends AppModel
case 'github-organisation':
case 'cpe':
case 'twitter-id':
case 'chrome-extension-id':
case 'mobile-application-id':
// no newline
if (!preg_match("#\n#", $value)) {
@ -2141,6 +2196,50 @@ class Attribute extends AppModel
return $fails;
}
public function ISODatetimeToUTC($data, $alias)
{
// convert into utc and micro sec
if (!empty($data[$alias]['first_seen'])) {
$d = new DateTime($data[$alias]['first_seen']);
$d->setTimezone(new DateTimeZone('GMT'));
$fs_sec = $d->format('U');
$fs_micro = $d->format('u');
$fs_micro = str_pad($fs_micro, 6, "0", STR_PAD_LEFT);
$fs = $fs_sec . $fs_micro;
$data[$alias]['first_seen'] = $fs;
}
if (!empty($data[$alias]['last_seen'])) {
$d = new DateTime($data[$alias]['last_seen']);
$d->setTimezone(new DateTimeZone('GMT'));
$ls_sec = $d->format('U');
$ls_micro = $d->format('u');
$ls_micro = str_pad($ls_micro, 6, "0", STR_PAD_LEFT);
$ls = $ls_sec . $ls_micro;
$data[$alias]['last_seen'] = $ls;
}
return $data;
}
public function UTCToISODatetime($data, $alias)
{
if (!empty($data[$alias]['first_seen'])) {
$fs = $data[$alias]['first_seen'];
$fs_sec = intval($fs / 1000000); // $fs is in micro (10^6)
$fs_micro = $fs % 1000000;
$fs_micro = str_pad($fs_micro, 6, "0", STR_PAD_LEFT);
$fs = $fs_sec . '.' . $fs_micro;
$data[$alias]['first_seen'] = DateTime::createFromFormat('U.u', $fs)->format('Y-m-d\TH:i:s.uP');
}
if (!empty($data[$alias]['last_seen'])) {
$ls = $data[$alias]['last_seen'];
$ls_sec = intval($ls / 1000000); // $ls is in micro (10^6)
$ls_micro = $ls % 1000000;
$ls_micro = str_pad($ls_micro, 6, "0", STR_PAD_LEFT);
$ls = $ls_sec . '.' . $ls_micro;
$data[$alias]['last_seen'] = DateTime::createFromFormat('U.u', $ls)->format('Y-m-d\TH:i:s.uP');
}
return $data;
}
public function hids($user, $type, $tags = '', $from = false, $to = false, $last = false, $jobId = false, $enforceWarninglist = false)
{
@ -2255,7 +2354,7 @@ class Attribute extends AppModel
$rules = array();
foreach ($eventIds as $event) {
$conditions['AND'] = array('Attribute.to_ids' => 1, "Event.published" => 1, 'Attribute.event_id' => $event['Event']['id']);
$valid_types = array('ip-dst', 'ip-src', 'ip-dst|port', 'ip-src|port', 'email-src', 'email-dst', 'email-subject', 'email-attachment', 'domain', 'domain|ip', 'hostname', 'url', 'user-agent', 'snort');
$valid_types = array('ip-dst', 'ip-src', 'ip-dst|port', 'ip-src|port', 'eppn', 'email-src', 'email-dst', 'email-subject', 'email-attachment', 'domain', 'domain|ip', 'hostname', 'url', 'user-agent', 'snort');
$conditions['AND']['Attribute.type'] = $valid_types;
if (!empty($type)) {
$conditions['AND'][] = array('Attribute.type' => $type);
@ -3262,6 +3361,8 @@ class Attribute extends AppModel
if (empty($at['Tag'])) {
unset($results[$k]['AttributeTag'][$k2]);
$tagCulled = true;
} else {
$results[$k]['AttributeTag'][$k2]['Tag']['local'] = $results[$k]['AttributeTag'][$k2]['local'];
}
}
if ($tagCulled) {
@ -3271,6 +3372,7 @@ class Attribute extends AppModel
if (isset($result['Event']['EventTag'])) {
$results[$k]['Event']['Tag'] = array();
foreach ($result['Event']['EventTag'] as $et) {
$et['Tag']['local'] = $et['local'];
$results[$k]['Event']['Tag'][] = $et['Tag'];
}
unset($results[$k]['Event']['EventTag']);
@ -3527,6 +3629,7 @@ class Attribute extends AppModel
$defaultDistribution = Configure::read('MISP.default_attribute_distribution');
}
}
$saveResult = true;
foreach ($attributes as $k => $attribute) {
if (!empty($attribute['encrypt']) && $attribute['encrypt']) {
$attribute = $this->onDemandEncrypt($attribute);
@ -3536,9 +3639,9 @@ class Attribute extends AppModel
}
unset($attribute['Attachment']);
$this->create();
$this->save($attribute);
$saveResult = $saveResult && $this->save($attribute);
}
return true;
return $saveResult;
}
public function onDemandEncrypt($attribute)
@ -3589,11 +3692,6 @@ class Attribute extends AppModel
return true;
}
public function convertToOpenIOC($user, $attributes)
{
return $this->IOCExport->buildAll($user, $event);
}
private function __createTagSubQuery($tag_id, $blocked = false, $scope = 'Event', $limitAttributeHitsTo = 'event')
{
$conditionKey = $blocked ? array('NOT' => array('EventTag.tag_id' => $tag_id)) : array('EventTag.tag_id' => $tag_id);
@ -3661,6 +3759,32 @@ class Attribute extends AppModel
return $conditions;
}
public function setTimestampSeenConditions($timestamp, $conditions, $scope = 'Attribute.first_seen', $returnRaw = false)
{
if (is_array($timestamp)) {
$timestamp[0] = intval($this->Event->resolveTimeDelta($timestamp[0])) * 1000000; // seen in stored in micro-seconds in the DB
$timestamp[1] = intval($this->Event->resolveTimeDelta($timestamp[1])) * 1000000; // seen in stored in micro-seconds in the DB
if ($timestamp[0] > $timestamp[1]) {
$temp = $timestamp[0];
$timestamp[0] = $timestamp[1];
$timestamp[1] = $temp;
}
$conditions['AND'][] = array($scope . ' >=' => $timestamp[0]);
$conditions['AND'][] = array($scope . ' <=' => $timestamp[1]);
} else {
$timestamp = intval($this->Event->resolveTimeDelta($timestamp)) * 1000000; // seen in stored in micro-seconds in the DB
if ($scope == 'Attribute.first_seen') {
$conditions['AND'][] = array($scope . ' >=' => $timestamp);
} else {
$conditions['AND'][] = array($scope . ' <=' => $timestamp);
}
}
if ($returnRaw) {
return $timestamp;
}
return $conditions;
}
public function setToIDSConditions($to_ids, $conditions)
{
if ($to_ids === 'exclude') {
@ -3969,7 +4093,8 @@ class Attribute extends AppModel
if (isset($attribute['uuid'])) {
$existingAttribute = $this->find('first', array(
'conditions' => array('Attribute.uuid' => $attribute['uuid']),
'recursive' => -1
'contain' => array('AttributeTag' => 'Tag'),
'recursive' => -1,
));
$this->Log = ClassRegistry::init('Log');
if (count($existingAttribute)) {
@ -3988,7 +4113,7 @@ class Attribute extends AppModel
return true;
}
// If a field is not set in the request, just reuse the old value
$recoverFields = array('value', 'to_ids', 'distribution', 'category', 'type', 'comment', 'sharing_group_id', 'object_id', 'object_relation');
$recoverFields = array('value', 'to_ids', 'distribution', 'category', 'type', 'comment', 'sharing_group_id', 'object_id', 'object_relation', 'first_seen', 'last_seen');
foreach ($recoverFields as $rF) {
if (!isset($attribute[$rF])) {
$attribute[$rF] = $existingAttribute['Attribute'][$rF];
@ -4058,7 +4183,9 @@ class Attribute extends AppModel
'comment',
'sharing_group_id',
'deleted',
'disable_correlation'
'disable_correlation',
'first_seen',
'last_seen'
);
if ($objectId) {
$fieldList[] = 'object_id';
@ -4080,28 +4207,36 @@ class Attribute extends AppModel
));
return $this->validationErrors;
} else {
if (isset($attribute['Tag']) && $user['Role']['perm_tagger']) {
foreach ($attribute['Tag'] as $tag) {
$tag_id = $this->AttributeTag->Tag->captureTag($tag, $user);
if ($tag_id) {
// fix the IDs here
$this->AttributeTag->attachTagToAttribute($this->id, $attribute['event_id'], $tag_id);
} else {
// If we couldn't attach the tag it is most likely because we couldn't create it - which could have many reasons
// However, if a tag couldn't be added, it could also be that the user is a tagger but not a tag editor
// In which case if no matching tag is found, no tag ID is returned. Logging these is pointless as it is the correct behaviour.
if ($user['Role']['perm_tag_editor']) {
$this->Log->create();
$this->Log->save(array(
'org' => $user['Organisation']['name'],
'model' => 'Attrubute',
'model_id' => $this->id,
'email' => $user['email'],
'action' => 'edit',
'user_id' => $user['id'],
'title' => 'Failed create or attach Tag ' . $tag['name'] . ' to the attribute.',
'change' => ''
));
if ($user['Role']['perm_tagger']) {
/*
We should uncomment the line below in the future once we have tag soft-delete
A solution to still keep the behavior for previous instance could be to not soft-delete the Tag if the remote instance
has a version below x
*/
// $this->AttributeTag->pruneOutdatedAttributeTagsFromSync(isset($attribute['Tag']) ? $attribute['Tag'] : array(), $existingAttribute['AttributeTag']);
if (isset($attribute['Tag'])) {
foreach ($attribute['Tag'] as $tag) {
$tag_id = $this->AttributeTag->Tag->captureTag($tag, $user);
if ($tag_id) {
// fix the IDs here
$this->AttributeTag->attachTagToAttribute($this->id, $attribute['event_id'], $tag_id);
} else {
// If we couldn't attach the tag it is most likely because we couldn't create it - which could have many reasons
// However, if a tag couldn't be added, it could also be that the user is a tagger but not a tag editor
// In which case if no matching tag is found, no tag ID is returned. Logging these is pointless as it is the correct behaviour.
if ($user['Role']['perm_tag_editor']) {
$this->Log->create();
$this->Log->save(array(
'org' => $user['Organisation']['name'],
'model' => 'Attrubute',
'model_id' => $this->id,
'email' => $user['email'],
'action' => 'edit',
'user_id' => $user['id'],
'title' => 'Failed create or attach Tag ' . $tag['name'] . ' to the attribute.',
'change' => ''
));
}
}
}
}
@ -4119,6 +4254,7 @@ class Attribute extends AppModel
$result = $this->fetchAttributes($user, array(
'conditions' => array('Attribute.id' => $id),
'flatten' => 1,
'deleted' => [0,1],
'recursive' => -1,
'contain' => array('Event')
));
@ -4218,6 +4354,9 @@ class Attribute extends AppModel
'uuid' => array('function' => 'set_filter_uuid'),
'deleted' => array('function' => 'set_filter_deleted'),
'timestamp' => array('function' => 'set_filter_timestamp'),
'attribute_timestamp' => array('function' => 'set_filter_timestamp'),
'first_seen' => array('function' => 'set_filter_seen'),
'last_seen' => array('function' => 'set_filter_seen'),
'to_ids' => array('function' => 'set_filter_to_ids'),
'comment' => array('function' => 'set_filter_comment')
),
@ -4292,6 +4431,10 @@ class Attribute extends AppModel
$filters['wildcard'] = $filters['searchall'];
}
}
$subqueryElements = $this->Event->harvestSubqueryElements($filters);
$filters = $this->Event->addFiltersFromSubqueryElements($filters, $subqueryElements);
$conditions = $this->buildFilterConditions($user, $filters);
$params = array(
'conditions' => $conditions,
@ -4306,6 +4449,7 @@ class Attribute extends AppModel
'includeWarninglistHits' => !empty($filters['includeWarninglistHits']) ? $filters['includeWarninglistHits'] : 0,
'includeContext' => !empty($filters['includeContext']) ? $filters['includeContext'] : 0,
'includeSightings' => !empty($filters['includeSightings']) ? $filters['includeSightings'] : 0,
'includeSightingdb' => !empty($filters['includeSightingdb']) ? $filters['includeSightingdb'] : 0,
'includeCorrelations' => !empty($filters['includeCorrelations']) ? $filters['includeCorrelations'] : 0,
'includeDecayScore' => !empty($filters['includeDecayScore']) ? $filters['includeDecayScore'] : 0,
'includeFullModel' => !empty($filters['includeFullModel']) ? $filters['includeFullModel'] : 0,
@ -4366,7 +4510,7 @@ class Attribute extends AppModel
$loop = false;
if (empty($params['limit'])) {
$memory_in_mb = $this->convert_to_memory_limit_to_mb(ini_get('memory_limit'));
$memory_scaling_factor = isset($exportTool->memory_scaling_factor) ? $exportTool->memory_scaling_factor : 100;
$memory_scaling_factor = isset($exportTool->memory_scaling_factor) ? $exportTool->memory_scaling_factor : 80;
$params['limit'] = $memory_in_mb * $memory_scaling_factor;
$loop = true;
$params['page'] = 1;
@ -4389,6 +4533,10 @@ class Attribute extends AppModel
while ($continue) {
$this->Whitelist = ClassRegistry::init('Whitelist');
$results = $this->fetchAttributes($user, $params, $continue);
if ($params['includeSightingdb']) {
$this->Sightingdb = ClassRegistry::init('Sightingdb');
$results = $this->Sightingdb->attachToAttributes($results, $user);
}
$params['page'] += 1;
$results = $this->Whitelist->removeWhitelistedFromArray($results, true);
$results = array_values($results);
@ -4396,8 +4544,9 @@ class Attribute extends AppModel
$temp = '';
foreach ($results as $attribute) {
$elementCounter++;
$temp .= $exportTool->handler($attribute, $exportToolParams);
if ($temp !== '') {
$handlerResult = $exportTool->handler($attribute, $exportToolParams);
$temp .= $handlerResult;
if ($handlerResult !== '') {
if ($i != count($results) -1) {
$temp .= $exportTool->separator($exportToolParams);
}

17
app/Model/AttributeTag.php
View File

@ -103,6 +103,22 @@ class AttributeTag extends AppModel
return true;
}
// This function help mirroring the tags at attribute level. It will delete tags that are not present on the remote attribute
public function pruneOutdatedAttributeTagsFromSync($newerTags, $originalAttributeTags)
{
$newerTagsName = array();
foreach ($newerTags as $tag) {
$newerTagsName[] = strtolower($tag['name']);
}
foreach ($originalAttributeTags as $k => $attributeTag) {
if (!$attributeTag['local']) { //
if (!in_array(strtolower($attributeTag['Tag']['name']), $newerTagsName)) {
$this->softDelete($attributeTag['id']);
}
}
}
}
public function countForTag($tag_id, $user)
{
return $this->find('count', array(
@ -172,6 +188,7 @@ class AttributeTag extends AppModel
$attributes = $this->Attribute->fetchAttributes($user, array(
'conditions' => $conditions,
'flatten' => 1,
'includeAllTags' => 1
));
if (empty($attributes)) {

31
app/Model/Behavior/RegexpBehavior.php
View File

@ -4,37 +4,40 @@ App::uses('Regexp', 'Model');
/**
* Behavior to regexp all string fields in a model
*
*/
class RegexpBehavior extends ModelBehavior
{
private $__allRegexp = array();
private $__allRegexp = null;
public $excluded_types = array('sigma', 'float');
public function setup(Model $model, $config = null)
{
$regexp = new Regexp();
$this->__allRegexp = $regexp->find('all', array('order' => 'id ASC'));
}
/**
* replace the current value according to the regexp rules, or block blacklisted regular expressions
*
* @param Model $Model
* @param unknown_type $array
* @param string $type
* @param string $value
* @return string
*/
public function runRegexp(Model $Model, $type, $value)
{
if (in_array($type, $this->excluded_types)) {
return $value;
}
if ($this->__allRegexp === null) {
$regexp = new Regexp();
$this->__allRegexp = $regexp->find('all', array('order' => 'id ASC'));
}
foreach ($this->__allRegexp as $regexp) {
if (!empty($regexp['Regexp']['replacement']) && !empty($regexp['Regexp']['regexp']) && ($regexp['Regexp']['type'] === 'ALL' || $regexp['Regexp']['type'] === $type)) {
$value = preg_replace($regexp['Regexp']['regexp'], $regexp['Regexp']['replacement'], $value);
}
if (empty($regexp['Regexp']['replacement']) && preg_match($regexp['Regexp']['regexp'], $value) && ($regexp['Regexp']['type'] === 'ALL' || $regexp['Regexp']['type'] === $type)) {
return false;
if ($regexp['Regexp']['type'] === 'ALL' || $regexp['Regexp']['type'] === $type) {
if (!empty($regexp['Regexp']['replacement']) && !empty($regexp['Regexp']['regexp'])) {
$value = preg_replace($regexp['Regexp']['regexp'], $regexp['Regexp']['replacement'], $value);
}
if (empty($regexp['Regexp']['replacement']) && preg_match($regexp['Regexp']['regexp'], $value)) {
return false;
}
}
}
return $value;

921
app/Model/Datasource/Database/MysqlExtendedLogging.php Normal file
View File

@ -0,0 +1,921 @@
<?php
/**
* MySQL layer for DBO
*
* CakePHP(tm) : Rapid Development Framework (https://cakephp.org)
* Copyright (c) Cake Software Foundation, Inc. (https://cakefoundation.org)
*
* Licensed under The MIT License
* For full copyright and license information, please see the LICENSE.txt
* Redistributions of files must retain the above copyright notice.
*
* @copyright Copyright (c) Cake Software Foundation, Inc. (https://cakefoundation.org)
* @link https://cakephp.org CakePHP(tm) Project
* @package Cake.Model.Datasource.Database
* @since CakePHP(tm) v 0.10.5.1790
* @license https://opensource.org/licenses/mit-license.php MIT License
*/
App::uses('DboSource', 'Model/Datasource');
/**
* MySQL DBO driver object
*
* Provides connection and SQL generation for MySQL RDMS
*
* @package Cake.Model.Datasource.Database
*/
class MysqlExtendedLogging extends DboSource {
/**
* Datasource description
*
* @var string
*/
public $description = "MySQL DBO Driver";
/**
* Base configuration settings for MySQL driver
*
* @var array
*/
protected $_baseConfig = array(
'persistent' => true,
'host' => 'localhost',
'login' => 'root',
'password' => '',
'database' => 'cake',
'port' => '3306',
'flags' => array()
);
/**
* Reference to the PDO object connection
*
* @var PDO
*/
protected $_connection = null;
/**
* Start quote
*
* @var string
*/
public $startQuote = "`";
/**
* End quote
*
* @var string
*/
public $endQuote = "`";
/**
* use alias for update and delete. Set to true if version >= 4.1
*
* @var bool
*/
protected $_useAlias = true;
/**
* List of engine specific additional field parameters used on table creating
*
* @var array
*/
public $fieldParameters = array(
'charset' => array('value' => 'CHARACTER SET', 'quote' => false, 'join' => ' ', 'column' => false, 'position' => 'beforeDefault'),
'collate' => array('value' => 'COLLATE', 'quote' => false, 'join' => ' ', 'column' => 'Collation', 'position' => 'beforeDefault'),
'comment' => array('value' => 'COMMENT', 'quote' => true, 'join' => ' ', 'column' => 'Comment', 'position' => 'afterDefault'),
'unsigned' => array(
'value' => 'UNSIGNED',
'quote' => false,
'join' => ' ',
'column' => false,
'position' => 'beforeDefault',
'noVal' => true,
'options' => array(true),
'types' => array('integer', 'smallinteger', 'tinyinteger', 'float', 'decimal', 'biginteger')
)
);
/**
* List of table engine specific parameters used on table creating
*
* @var array
*/
public $tableParameters = array(
'charset' => array('value' => 'DEFAULT CHARSET', 'quote' => false, 'join' => '=', 'column' => 'charset'),
'collate' => array('value' => 'COLLATE', 'quote' => false, 'join' => '=', 'column' => 'Collation'),
'engine' => array('value' => 'ENGINE', 'quote' => false, 'join' => '=', 'column' => 'Engine'),
'comment' => array('value' => 'COMMENT', 'quote' => true, 'join' => '=', 'column' => 'Comment'),
);
/**
* MySQL column definition
*
* @var array
* @link https://dev.mysql.com/doc/refman/5.7/en/data-types.html MySQL Data Types
*/
public $columns = array(
'primary_key' => array('name' => 'NOT NULL AUTO_INCREMENT'),
'string' => array('name' => 'varchar', 'limit' => '255'),
'text' => array('name' => 'text'),
'enum' => array('name' => 'enum'),
'biginteger' => array('name' => 'bigint', 'limit' => '20'),
'integer' => array('name' => 'int', 'limit' => '11', 'formatter' => 'intval'),
'smallinteger' => array('name' => 'smallint', 'limit' => '6', 'formatter' => 'intval'),
'tinyinteger' => array('name' => 'tinyint', 'limit' => '4', 'formatter' => 'intval'),
'float' => array('name' => 'float', 'formatter' => 'floatval'),
'decimal' => array('name' => 'decimal', 'formatter' => 'floatval'),
'datetime' => array('name' => 'datetime', 'format' => 'Y-m-d H:i:s', 'formatter' => 'date'),
'timestamp' => array('name' => 'timestamp', 'format' => 'Y-m-d H:i:s', 'formatter' => 'date'),
'time' => array('name' => 'time', 'format' => 'H:i:s', 'formatter' => 'date'),
'date' => array('name' => 'date', 'format' => 'Y-m-d', 'formatter' => 'date'),
'binary' => array('name' => 'blob'),
'boolean' => array('name' => 'tinyint', 'limit' => '1')
);
/**
* Mapping of collation names to character set names
*
* @var array
*/
protected $_charsets = array();
/**
* Connects to the database using options in the given configuration array.
*
* MySQL supports a few additional options that other drivers do not:
*
* - `unix_socket` Set to the path of the MySQL sock file. Can be used in place
* of host + port.
* - `ssl_key` SSL key file for connecting via SSL. Must be combined with `ssl_cert`.
* - `ssl_cert` The SSL certificate to use when connecting via SSL. Must be
* combined with `ssl_key`.
* - `ssl_ca` The certificate authority for SSL connections.
*
* @return bool True if the database could be connected, else false
* @throws MissingConnectionException
*/
public function connect() {
$config = $this->config;
$this->connected = false;
$flags = $config['flags'] + array(
PDO::ATTR_PERSISTENT => $config['persistent'],
PDO::MYSQL_ATTR_USE_BUFFERED_QUERY => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
);
if (!empty($config['encoding'])) {
$flags[PDO::MYSQL_ATTR_INIT_COMMAND] = 'SET NAMES ' . $config['encoding'];
}
if (!empty($config['ssl_key']) && !empty($config['ssl_cert'])) {
$flags[PDO::MYSQL_ATTR_SSL_KEY] = $config['ssl_key'];
$flags[PDO::MYSQL_ATTR_SSL_CERT] = $config['ssl_cert'];
}
if (!empty($config['ssl_ca'])) {
$flags[PDO::MYSQL_ATTR_SSL_CA] = $config['ssl_ca'];
}
if (empty($config['unix_socket'])) {
$dsn = "mysql:host={$config['host']};port={$config['port']};dbname={$config['database']}";
} else {
$dsn = "mysql:unix_socket={$config['unix_socket']};dbname={$config['database']}";
}
try {
$this->_connection = new PDO(
$dsn,
$config['login'],
$config['password'],
$flags
);
$this->connected = true;
if (!empty($config['settings'])) {
foreach ($config['settings'] as $key => $value) {
$this->_execute("SET $key=$value");
}
}
} catch (PDOException $e) {
throw new MissingConnectionException(array(
'class' => get_class($this),
'message' => $e->getMessage()
));
}
$this->_charsets = array();
$this->_useAlias = (bool)version_compare($this->getVersion(), "4.1", ">=");
return $this->connected;
}
/**
* Check whether the MySQL extension is installed/loaded
*
* @return bool
*/
public function enabled() {
return in_array('mysql', PDO::getAvailableDrivers());
}
/**
* Returns an array of sources (tables) in the database.
*
* @param mixed $data List of tables.
* @return array Array of table names in the database
*/
public function listSources($data = null) {
$cache = parent::listSources();
if ($cache) {
return $cache;
}
$result = $this->_execute('SHOW TABLES FROM ' . $this->name($this->config['database']));
if (!$result) {
$result->closeCursor();
return array();
}
$tables = array();
while ($line = $result->fetch(PDO::FETCH_NUM)) {
$tables[] = $line[0];
}
$result->closeCursor();
parent::listSources($tables);
return $tables;
}
/**
* Builds a map of the columns contained in a result
*
* @param PDOStatement $results The results to format.
* @return void
*/
public function resultSet($results) {
$this->map = array();
$numFields = $results->columnCount();
$index = 0;
while ($numFields-- > 0) {
$column = $results->getColumnMeta($index);
if ($column['len'] === 1 && (empty($column['native_type']) || $column['native_type'] === 'TINY')) {
$type = 'boolean';
} else {
$type = empty($column['native_type']) ? 'string' : $column['native_type'];
}
if (!empty($column['table']) && strpos($column['name'], $this->virtualFieldSeparator) === false) {
$this->map[$index++] = array($column['table'], $column['name'], $type);
} else {
$this->map[$index++] = array(0, $column['name'], $type);
}
}
}
/**
* Fetches the next row from the current result set
*
* @return mixed array with results fetched and mapped to column names or false if there is no results left to fetch
*/
public function fetchResult() {
if ($row = $this->_result->fetch(PDO::FETCH_NUM)) {
$resultRow = array();
foreach ($this->map as $col => $meta) {
list($table, $column, $type) = $meta;
$resultRow[$table][$column] = $row[$col];
if ($type === 'boolean' && $row[$col] !== null) {
$resultRow[$table][$column] = $this->boolean($resultRow[$table][$column]);
}
}
return $resultRow;
}
$this->_result->closeCursor();
return false;
}
/**
* Gets the database encoding
*
* @return string The database encoding
*/
public function getEncoding() {
return $this->_execute('SHOW VARIABLES LIKE ?', array('character_set_client'))->fetchObject()->Value;
}
/**
* Query charset by collation
*
* @param string $name Collation name
* @return string|false Character set name
*/
public function getCharsetName($name) {
if ((bool)version_compare($this->getVersion(), "5", "<")) {
return false;
}
if (isset($this->_charsets[$name])) {
return $this->_charsets[$name];
}
$r = $this->_execute(
'SELECT CHARACTER_SET_NAME FROM INFORMATION_SCHEMA.COLLATIONS WHERE COLLATION_NAME = ?',
array($name)
);
$cols = $r->fetch(PDO::FETCH_ASSOC);
if (isset($cols['CHARACTER_SET_NAME'])) {
$this->_charsets[$name] = $cols['CHARACTER_SET_NAME'];
} else {
$this->_charsets[$name] = false;
}
return $this->_charsets[$name];
}
/**
* Returns an array of the fields in given table name.
*
* @param Model|string $model Name of database table to inspect or model instance
* @return array Fields in table. Keys are name and type
* @throws CakeException
*/
public function describe($model) {
$key = $this->fullTableName($model, false);
$cache = parent::describe($key);
if ($cache) {
return $cache;
}
$table = $this->fullTableName($model);
$fields = false;
$cols = $this->_execute('SHOW FULL COLUMNS FROM ' . $table);
if (!$cols) {
throw new CakeException(__d('cake_dev', 'Could not describe table for %s', $table));
}
while ($column = $cols->fetch(PDO::FETCH_OBJ)) {
$fields[$column->Field] = array(
'type' => $this->column($column->Type),
'null' => ($column->Null === 'YES' ? true : false),
'default' => $column->Default,
'length' => $this->length($column->Type)
);
if (in_array($fields[$column->Field]['type'], $this->fieldParameters['unsigned']['types'], true)) {
$fields[$column->Field]['unsigned'] = $this->_unsigned($column->Type);
}
if (in_array($fields[$column->Field]['type'], array('timestamp', 'datetime')) &&
in_array(strtoupper($column->Default), array('CURRENT_TIMESTAMP', 'CURRENT_TIMESTAMP()'))
) {
$fields[$column->Field]['default'] = null;
}
if (!empty($column->Key) && isset($this->index[$column->Key])) {
$fields[$column->Field]['key'] = $this->index[$column->Key];
}
foreach ($this->fieldParameters as $name => $value) {
if (!empty($column->{$value['column']})) {
$fields[$column->Field][$name] = $column->{$value['column']};
}
}
if (isset($fields[$column->Field]['collate'])) {
$charset = $this->getCharsetName($fields[$column->Field]['collate']);
if ($charset) {
$fields[$column->Field]['charset'] = $charset;
}
}
}
$this->_cacheDescription($key, $fields);
$cols->closeCursor();
return $fields;
}
/**
* Generates and executes an SQL UPDATE statement for given model, fields, and values.
*
* @param Model $model The model to update.
* @param array $fields The fields to update.
* @param array $values The values to set.
* @param mixed $conditions The conditions to use.
* @return bool
*/
public function update(Model $model, $fields = array(), $values = null, $conditions = null) {
if (!$this->_useAlias) {
return parent::update($model, $fields, $values, $conditions);
}
if (!$values) {
$combined = $fields;
} else {
$combined = array_combine($fields, $values);
}
$alias = $joins = false;
$fields = $this->_prepareUpdateFields($model, $combined, empty($conditions), !empty($conditions));
$fields = implode(', ', $fields);
$table = $this->fullTableName($model);
if (!empty($conditions)) {
$alias = $this->name($model->alias);
if ($model->name === $model->alias) {
$joins = implode(' ', $this->_getJoins($model));
}
}
$conditions = $this->conditions($this->defaultConditions($model, $conditions, $alias), true, true, $model);
if ($conditions === false) {
return false;
}
if (!$this->execute($this->renderStatement('update', compact('table', 'alias', 'joins', 'fields', 'conditions')))) {
$model->onError();
return false;
}
return true;
}
/**
* Generates and executes an SQL DELETE statement for given id/conditions on given model.
*
* @param Model $model The model to delete from.
* @param mixed $conditions The conditions to use.
* @return bool Success
*/
public function delete(Model $model, $conditions = null) {
if (!$this->_useAlias) {
return parent::delete($model, $conditions);
}
$alias = $this->name($model->alias);
$table = $this->fullTableName($model);
$joins = implode(' ', $this->_getJoins($model));
if (empty($conditions)) {
$alias = $joins = false;
}
$complexConditions = $this->_deleteNeedsComplexConditions($model, $conditions);
if (!$complexConditions) {
$joins = false;
}
$conditions = $this->conditions($this->defaultConditions($model, $conditions, $alias), true, true, $model);
if ($conditions === false) {
return false;
}
if ($this->execute($this->renderStatement('delete', compact('alias', 'table', 'joins', 'conditions'))) === false) {
$model->onError();
return false;
}
return true;
}
/**
* Checks whether complex conditions are needed for a delete with the given conditions.
*
* @param Model $model The model to delete from.
* @param mixed $conditions The conditions to use.
* @return bool Whether or not complex conditions are needed
*/
protected function _deleteNeedsComplexConditions(Model $model, $conditions) {
$fields = array_keys($this->describe($model));
foreach ((array)$conditions as $key => $value) {
if (in_array(strtolower(trim($key)), $this->_sqlBoolOps, true)) {
if ($this->_deleteNeedsComplexConditions($model, $value)) {
return true;
}
} elseif (strpos($key, $model->alias) === false && !in_array($key, $fields, true)) {
return true;
}
}
return false;
}
/**
* Sets the database encoding
*
* @param string $enc Database encoding
* @return bool
*/
public function setEncoding($enc) {
return $this->_execute('SET NAMES ' . $enc) !== false;
}
/**
* Returns an array of the indexes in given datasource name.
*
* @param string $model Name of model to inspect
* @return array Fields in table. Keys are column and unique
*/
public function index($model) {
$index = array();
$table = $this->fullTableName($model);
$old = version_compare($this->getVersion(), '4.1', '<=');
if ($table) {
$indexes = $this->_execute('SHOW INDEX FROM ' . $table);
// @codingStandardsIgnoreStart
// MySQL columns don't match the cakephp conventions.
while ($idx = $indexes->fetch(PDO::FETCH_OBJ)) {
if ($old) {
$idx = (object)current((array)$idx);
}
if (!isset($index[$idx->Key_name]['column'])) {
$col = array();
$index[$idx->Key_name]['column'] = $idx->Column_name;
if ($idx->Index_type === 'FULLTEXT') {
$index[$idx->Key_name]['type'] = strtolower($idx->Index_type);
} else {
$index[$idx->Key_name]['unique'] = (int)($idx->Non_unique == 0);
}
} else {
if (!empty($index[$idx->Key_name]['column']) && !is_array($index[$idx->Key_name]['column'])) {
$col[] = $index[$idx->Key_name]['column'];
}
$col[] = $idx->Column_name;
$index[$idx->Key_name]['column'] = $col;
}
if (!empty($idx->Sub_part)) {
if (!isset($index[$idx->Key_name]['length'])) {
$index[$idx->Key_name]['length'] = array();
}
$index[$idx->Key_name]['length'][$idx->Column_name] = $idx->Sub_part;
}
}
// @codingStandardsIgnoreEnd
$indexes->closeCursor();
}
return $index;
}
/**
* Generate a MySQL Alter Table syntax for the given Schema comparison
*
* @param array $compare Result of a CakeSchema::compare()
* @param string $table The table name.
* @return string|false String of alter statements to make.
*/
public function alterSchema($compare, $table = null) {
if (!is_array($compare)) {
return false;
}
$out = '';
$colList = array();
foreach ($compare as $curTable => $types) {
$indexes = $tableParameters = $colList = array();
if (!$table || $table === $curTable) {
$out .= 'ALTER TABLE ' . $this->fullTableName($curTable) . " \n";
foreach ($types as $type => $column) {
if (isset($column['indexes'])) {
$indexes[$type] = $column['indexes'];
unset($column['indexes']);
}
if (isset($column['tableParameters'])) {
$tableParameters[$type] = $column['tableParameters'];
unset($column['tableParameters']);
}
switch ($type) {
case 'add':
foreach ($column as $field => $col) {
$col['name'] = $field;
$alter = 'ADD ' . $this->buildColumn($col);
if (isset($col['after'])) {
$alter .= ' AFTER ' . $this->name($col['after']);
}
$colList[] = $alter;
}
break;
case 'drop':
foreach ($column as $field => $col) {
$col['name'] = $field;
$colList[] = 'DROP ' . $this->name($field);
}
break;
case 'change':
foreach ($column as $field => $col) {
if (!isset($col['name'])) {
$col['name'] = $field;
}
$alter = 'CHANGE ' . $this->name($field) . ' ' . $this->buildColumn($col);
if (isset($col['after'])) {
$alter .= ' AFTER ' . $this->name($col['after']);
}
$colList[] = $alter;
}
break;
}
}
$colList = array_merge($colList, $this->_alterIndexes($curTable, $indexes));
$colList = array_merge($colList, $this->_alterTableParameters($curTable, $tableParameters));
$out .= "\t" . implode(",\n\t", $colList) . ";\n\n";
}
}
return $out;
}
/**
* Generate a "drop table" statement for the given table
*
* @param type $table Name of the table to drop
* @return string Drop table SQL statement
*/
protected function _dropTable($table) {
return 'DROP TABLE IF EXISTS ' . $this->fullTableName($table) . ";";
}
/**
* Generate MySQL table parameter alteration statements for a table.
*
* @param string $table Table to alter parameters for.
* @param array $parameters Parameters to add & drop.
* @return array Array of table property alteration statements.
*/
protected function _alterTableParameters($table, $parameters) {
if (isset($parameters['change'])) {
return $this->buildTableParameters($parameters['change']);
}
return array();
}
/**
* Format indexes for create table
*
* @param array $indexes An array of indexes to generate SQL from
* @param string $table Optional table name, not used
* @return array An array of SQL statements for indexes
* @see DboSource::buildIndex()
*/
public function buildIndex($indexes, $table = null) {
$join = array();
foreach ($indexes as $name => $value) {
$out = '';
if ($name === 'PRIMARY') {
$out .= 'PRIMARY ';
$name = null;
} else {
if (!empty($value['unique'])) {
$out .= 'UNIQUE ';
}
$name = $this->startQuote . $name . $this->endQuote;
}
if (isset($value['type']) && strtolower($value['type']) === 'fulltext') {
$out .= 'FULLTEXT ';
}
$out .= 'KEY ' . $name . ' (';
if (is_array($value['column'])) {
if (isset($value['length'])) {
$vals = array();
foreach ($value['column'] as $column) {
$name = $this->name($column);
if (isset($value['length'])) {
$name .= $this->_buildIndexSubPart($value['length'], $column);
}
$vals[] = $name;
}
$out .= implode(', ', $vals);
} else {
$out .= implode(', ', array_map(array(&$this, 'name'), $value['column']));
}
} else {
$out .= $this->name($value['column']);
if (isset($value['length'])) {
$out .= $this->_buildIndexSubPart($value['length'], $value['column']);
}
}
$out .= ')';
$join[] = $out;
}
return $join;
}
/**
* Generate MySQL index alteration statements for a table.
*
* @param string $table Table to alter indexes for
* @param array $indexes Indexes to add and drop
* @return array Index alteration statements
*/
protected function _alterIndexes($table, $indexes) {
$alter = array();
if (isset($indexes['drop'])) {
foreach ($indexes['drop'] as $name => $value) {
$out = 'DROP ';
if ($name === 'PRIMARY') {
$out .= 'PRIMARY KEY';
} else {
$out .= 'KEY ' . $this->startQuote . $name . $this->endQuote;
}
$alter[] = $out;
}
}
if (isset($indexes['add'])) {
$add = $this->buildIndex($indexes['add']);
foreach ($add as $index) {
$alter[] = 'ADD ' . $index;
}
}
return $alter;
}
/**
* Format length for text indexes
*
* @param array $lengths An array of lengths for a single index
* @param string $column The column for which to generate the index length
* @return string Formatted length part of an index field
*/
protected function _buildIndexSubPart($lengths, $column) {
if ($lengths === null) {
return '';
}
if (!isset($lengths[$column])) {
return '';
}
return '(' . $lengths[$column] . ')';
}
/**
* Returns a detailed array of sources (tables) in the database.
*
* @param string $name Table name to get parameters
* @return array Array of table names in the database
*/
public function listDetailedSources($name = null) {
$condition = '';
if (is_string($name)) {
$condition = ' WHERE name = ' . $this->value($name);
}
$result = $this->_connection->query('SHOW TABLE STATUS ' . $condition, PDO::FETCH_ASSOC);
if (!$result) {
$result->closeCursor();
return array();
}
$tables = array();
foreach ($result as $row) {
$tables[$row['Name']] = (array)$row;
unset($tables[$row['Name']]['queryString']);
if (!empty($row['Collation'])) {
$charset = $this->getCharsetName($row['Collation']);
if ($charset) {
$tables[$row['Name']]['charset'] = $charset;
}
}
}
$result->closeCursor();
if (is_string($name) && isset($tables[$name])) {
return $tables[$name];
}
return $tables;
}
/**
* Converts database-layer column types to basic types
*
* @param string $real Real database-layer column type (i.e. "varchar(255)")
* @return string Abstract column type (i.e. "string")
*/
public function column($real) {
if (is_array($real)) {
$col = $real['name'];
if (isset($real['limit'])) {
$col .= '(' . $real['limit'] . ')';
}
return $col;
}
$col = str_replace(')', '', $real);
$limit = $this->length($real);
if (strpos($col, '(') !== false) {
list($col, $vals) = explode('(', $col);
}
if (in_array($col, array('date', 'time', 'datetime', 'timestamp'))) {
return $col;
}
if (($col === 'tinyint' && $limit === 1) || $col === 'boolean') {
return 'boolean';
}
if (strpos($col, 'bigint') !== false || $col === 'bigint') {
return 'biginteger';
}
if (strpos($col, 'tinyint') !== false) {
return 'tinyinteger';
}
if (strpos($col, 'smallint') !== false) {
return 'smallinteger';
}
if (strpos($col, 'int') !== false) {
return 'integer';
}
if (strpos($col, 'char') !== false || $col === 'tinytext') {
return 'string';
}
if (strpos($col, 'text') !== false) {
return 'text';
}
if (strpos($col, 'blob') !== false || $col === 'binary') {
return 'binary';
}
if (strpos($col, 'float') !== false || strpos($col, 'double') !== false) {
return 'float';
}
if (strpos($col, 'decimal') !== false || strpos($col, 'numeric') !== false) {
return 'decimal';
}
if (strpos($col, 'enum') !== false) {
return "enum($vals)";
}
if (strpos($col, 'set') !== false) {
return "set($vals)";
}
return 'text';
}
/**
* {@inheritDoc}
*/
public function value($data, $column = null, $null = true) {
$value = parent::value($data, $column, $null);
if (is_numeric($value) && substr($column, 0, 3) === 'set') {
return $this->_connection->quote($value);
}
return $value;
}
/**
* Gets the schema name
*
* @return string The schema name
*/
public function getSchemaName() {
return $this->config['database'];
}
/**
* Check if the server support nested transactions
*
* @return bool
*/
public function nestedTransactionSupported() {
return $this->useNestedTransactions && version_compare($this->getVersion(), '4.1', '>=');
}
/**
* Check if column type is unsigned
*
* @param string $real Real database-layer column type (i.e. "varchar(255)")
* @return bool True if column is unsigned, false otherwise
*/
protected function _unsigned($real) {
return strpos(strtolower($real), 'unsigned') !== false;
}
/**
* Inserts multiple values into a table. Uses a single query in order to insert
* multiple rows.
*
* @param string $table The table being inserted into.
* @param array $fields The array of field/column names being inserted.
* @param array $values The array of values to insert. The values should
* be an array of rows. Each row should have values keyed by the column name.
* Each row must have the values in the same order as $fields.
* @return bool
*/
public function insertMulti($table, $fields, $values) {
$table = $this->fullTableName($table);
$holder = implode(', ', array_fill(0, count($fields), '?'));
$fields = implode(', ', array_map(array($this, 'name'), $fields));
$pdoMap = array(
'integer' => PDO::PARAM_INT,
'float' => PDO::PARAM_STR,
'boolean' => PDO::PARAM_BOOL,
'string' => PDO::PARAM_STR,
'text' => PDO::PARAM_STR
);
$columnMap = array();
$rowHolder = "({$holder})";
$sql = "INSERT INTO {$table} ({$fields}) VALUES ";
$countRows = count($values);
for ($i = 0; $i < $countRows; $i++) {
if ($i !== 0) {
$sql .= ',';
}
$sql .= " $rowHolder";
}
$statement = $this->_connection->prepare($sql);
foreach ($values[key($values)] as $key => $val) {
$type = $this->introspectType($val);
$columnMap[$key] = $pdoMap[$type];
}
$valuesList = array();
$i = 1;
foreach ($values as $value) {
foreach ($value as $col => $val) {
$valuesList[] = $val;
$statement->bindValue($i, $val, $columnMap[$col]);
$i++;
}
}
$result = $statement->execute();
$statement->closeCursor();
if ($this->fullDebug) {
$this->logQuery($sql, $valuesList);
}
return $result;
}
}

48
app/Model/DecayingModelsFormulas/Base.php
View File

@ -6,8 +6,30 @@ abstract class DecayingModelBase
return 'BONFIRE LIT';
}
protected function __extractTagBasename($tagName) {
$pieces = array();
if (preg_match('/^[^:="]+:[^:="]+="[^:="]+"$/i', $tagName)) {
$temp = explode(':', $tagName);
$pieces = array_merge(array($temp[0]), explode('=', $temp[1]));
$pieces['complete'] = $tagName;
$pieces['namespace'] = $pieces[0];
$pieces['predicate'] = $pieces[1];
$pieces['2tag'] = sprintf('%s:%s', $pieces[0], $pieces[1]);
$pieces['base'] = sprintf('%s:%s', $pieces[0], $pieces[1]);
} elseif (preg_match('/^[^:="]+:[^:="]+$/i', $tagName)) {
$pieces = explode(':', $tagName);
$pieces['complete'] = $tagName;
$pieces['namespace'] = $pieces[0];
$pieces['predicate'] = $pieces[1];
$pieces['2tag'] = sprintf('%s:%s', $pieces[0], $pieces[1]);
$pieces['base'] = $pieces[0];
}
return $pieces;
}
// Get effective taxonomy ratio based on taxonomies attached to the attribute
// Basically, it adapts the ratio defined in the model to fit the actual attached tags
// Basically, it adapts the ratio defined in the model to fit the actual attached tags
protected function __getRatioScore($model, $tags)
{
$ratioScore = array();
@ -17,21 +39,21 @@ abstract class DecayingModelBase
}
$total_score = 0.0;
foreach ($tags as $tag) {
$namespace_predicate = explode('=', $tag['Tag']['name'])[0];
if (isset($taxonomy_base_ratio[$namespace_predicate]) && is_numeric($tag['Tag']['numerical_value'])) {
$total_score += floatval($taxonomy_base_ratio[$namespace_predicate]);
$tagBaseName = $this->__extractTagBasename($tag['Tag']['name'])['base'];
if (isset($taxonomy_base_ratio[$tagBaseName]) && is_numeric($tag['Tag']['numerical_value'])) {
$total_score += floatval($taxonomy_base_ratio[$tagBaseName]);
}
}
foreach ($tags as $i => $tag) {
$namespace_predicate = explode('=', $tag['Tag']['name'])[0];
if (isset($taxonomy_base_ratio[$namespace_predicate]) && is_numeric($tag['Tag']['numerical_value'])) {
$ratioScore[$namespace_predicate] = floatval($taxonomy_base_ratio[$namespace_predicate]) / $total_score;
$tagBaseName = $this->__extractTagBasename($tag['Tag']['name'])['base'];
if (isset($taxonomy_base_ratio[$tagBaseName]) && is_numeric($tag['Tag']['numerical_value'])) {
$ratioScore[$tagBaseName] = floatval($taxonomy_base_ratio[$tagBaseName]) / $total_score;
}
}
return $ratioScore;
}
// return attribute tag with event tag matching the namespace+predicate overridden
// return attribute tag with event tag matching the tag basename overridden
protected function __getPrioritisedTag($attribute)
{
$tags = array();
@ -40,15 +62,15 @@ abstract class DecayingModelBase
if (isset($attribute['EventTag'])) {
foreach ($attribute['EventTag'] as $i => $tag) {
$tags[] = $tag;
$namespace_predicate = explode('=', $tag['Tag']['name'])[0];
$temp_mapping[$namespace_predicate][] = $i;
$tagBaseName = $this->__extractTagBasename($tag['Tag']['name'])['base'];
$temp_mapping[$tagBaseName][] = $i;
}
}
if (isset($attribute['AttributeTag'])) {
foreach ($attribute['AttributeTag'] as $tag) {
$namespace_predicate = explode('=', $tag['Tag']['name'])[0];
if (!empty($temp_mapping[$namespace_predicate])) { // need to override event tag
foreach ($temp_mapping[$namespace_predicate] as $i => $eventtag_index) {
$tagBaseName = $this->__extractTagBasename($tag['Tag']['name'])['base'];
if (!empty($temp_mapping[$tagBaseName])) { // need to override event tag
foreach ($temp_mapping[$tagBaseName] as $i => $eventtag_index) {
$overridden_tags[] = array(
'EventTag' => $tags[$eventtag_index],
'AttributeTag' => $tag

581
app/Model/Event.php
View File

@ -2,7 +2,6 @@
App::uses('AppModel', 'Model');
App::uses('CakeEmail', 'Network/Email');
App::uses('RandomTool', 'Tools');
Configure::load('config'); // This is needed to load GnuPG.bodyonlyencrypted
class Event extends AppModel
{
@ -121,7 +120,15 @@ class Event extends AppModel
'scope' => 'Event',
'requiresPublished' => 1,
'params' => array('returnFormat' => 'stix', 'includeAttachments' => 1),
'description' => 'Click this to download an a STIX document containing the STIX version of all events and attributes that you have access to.'
'description' => 'Click this to download a STIX document containing the STIX version of all events and attributes that you have access to.'
),
'stix-json' => array(
'extension' => '.json',
'type' => 'STIX',
'scope' => 'Event',
'requiresPublished' => 1,
'params' => array('returnFormat' => 'stix', 'includeAttachments' => 1),
'description' => 'Click this to download a STIX document containing the STIX version of all events and attributes that you have access to.'
),
'stix2' => array(
'extension' => '.json',
@ -129,7 +136,7 @@ class Event extends AppModel
'scope' => 'Event',
'requiresPublished' => 1,
'params' => array('returnFormat' => 'stix2', 'includeAttachments' => 1),
'description' => 'Click this to download an a STIX2 document containing the STIX2 version of all events and attributes that you have access to.'
'description' => 'Click this to download a STIX2 document containing the STIX2 version of all events and attributes that you have access to.'
),
'rpz' => array(
'extension' => '.txt',
@ -173,8 +180,10 @@ class Event extends AppModel
'snort' => array('txt', 'NidsSnortExport', 'rules'),
'rpz' => array('txt', 'RPZExport', 'rpz'),
'text' => array('text', 'TextExport', 'txt'),
'hashes' => array('txt', 'HashesExport', 'txt'),
'csv' => array('csv', 'CsvExport', 'csv'),
'stix' => array('xml', 'Stix1Export', 'xml'),
'stix-json' => array('json', 'Stix1Export', 'json'),
'stix2' => array('json', 'Stix2Export', 'json'),
'yara' => array('txt', 'YaraExport', 'yara'),
'yara-json' => array('json', 'YaraExport', 'json'),
@ -1055,9 +1064,9 @@ class Event extends AppModel
return $error;
}
private function __executeRestfulEventToServer($event, $server, $resourceId, &$newLocation, &$newTextBody, $HttpSocket)
private function __executeRestfulEventToServer($event, $server, $resourceId, &$newLocation, &$newTextBody, $HttpSocket, $scope)
{
$result = $this->restfulEventToServer($event, $server, $resourceId, $newLocation, $newTextBody, $HttpSocket);
$result = $this->restfulEventToServer($event, $server, $resourceId, $newLocation, $newTextBody, $HttpSocket, $scope);
if (is_numeric($result)) {
$error = $this->__resolveErrorCode($result, $event, $server);
if ($error) {
@ -1067,24 +1076,51 @@ class Event extends AppModel
return true;
}
public function uploadEventToServer($event, $server, $HttpSocket = null)
public function uploadSightingsToServer($sightings, $server, $event_uuid, $HttpSocket = null)
{
$HttpSocket = $this->setupHttpSocket($server, $HttpSocket);
$request = $this->setupSyncRequest($server);
$uri = $server['Server']['url'] . '/sightings/bulkSaveSightings/' . $event_uuid;
foreach ($sightings as &$sighting) {
if (!isset($sighting['org_id'])) {
$sighting['org_id'] = '0';
}
}
$data = json_encode($sightings);
if (!empty(Configure::read('Security.sync_audit'))) {
$pushLogEntry = sprintf(
"==============================================================\n\n[%s] Pushing Sightings for Event #%s to Server #%d:\n\n%s\n\n",
date("Y-m-d H:i:s"),
$event_uuid,
$server['Server']['id'],
$data
);
file_put_contents(APP . 'files/scripts/tmp/debug_server_' . $server['Server']['id'] . '.log', $pushLogEntry, FILE_APPEND);
}
$response = $HttpSocket->post($uri, $data, $request);
return $this->__handleRestfulEventToServerResponse($response, $newLocation, $newTextBody);
}
public function uploadEventToServer($event, $server, $HttpSocket = null, $scope = 'events')
{
$this->Server = ClassRegistry::init('Server');
$push = $this->Server->checkVersionCompatibility($server['Server']['id'], false, $HttpSocket);
if (empty($push['canPush'])) {
if ($scope === 'events' && empty($push['canPush'])) {
return 'The remote user is not a sync user - the upload of the event has been blocked.';
} elseif ($scope === 'sightings' && empty($push['canPush']) && empty($push['canSight'])) {
return 'The remote user is not a sightings user - the upload of the sightings has been blocked.';
}
if (!empty($server['Server']['unpublish_event'])) {
$event['Event']['published'] = 0;
}
$updated = null;
$newLocation = $newTextBody = '';
$result = $this->__executeRestfulEventToServer($event, $server, null, $newLocation, $newTextBody, $HttpSocket);
$result = $this->__executeRestfulEventToServer($event, $server, null, $newLocation, $newTextBody, $HttpSocket, $scope);
if ($result !== true) {
return $result;
}
if (strlen($newLocation)) { // HTTP/1.1 302 Found and Location: http://<newLocation>
$result = $this->__executeRestfulEventToServer($event, $server, $newLocation, $newLocation, $newTextBody, $HttpSocket);
$result = $this->__executeRestfulEventToServer($event, $server, $newLocation, $newLocation, $newTextBody, $HttpSocket, $scope);
if ($result !== true) {
return $result;
}
@ -1173,7 +1209,7 @@ class Event extends AppModel
}
// Uploads the event and the associated Attributes to another Server
public function restfulEventToServer($event, $server, $urlPath, &$newLocation, &$newTextBody, $HttpSocket = null)
public function restfulEventToServer($event, $server, $urlPath, &$newLocation, &$newTextBody, $HttpSocket = null, $scope)
{
$event = $this->__prepareForPushToServer($event, $server);
if (is_numeric($event)) {
@ -1182,7 +1218,11 @@ class Event extends AppModel
$url = $server['Server']['url'];
$HttpSocket = $this->setupHttpSocket($server, $HttpSocket);
$request = $this->setupSyncRequest($server);
$uri = $url . '/events' . $this->__getLastUrlPathComponent($urlPath);
if ($scope === 'sightings') {
$scope .= '/bulkSaveSightings';
$urlPath = $event['Event']['uuid'];
}
$uri = $url . '/' . $scope . $this->__getLastUrlPathComponent($urlPath);
$data = json_encode($event);
if (!empty(Configure::read('Security.sync_audit'))) {
$pushLogEntry = sprintf(
@ -1818,7 +1858,8 @@ class Event extends AppModel
'excludeGalaxy',
'includeRelatedTags',
'excludeLocalTags',
'includeDecayScore'
'includeDecayScore',
'includeSightingdb'
);
if (!isset($options['excludeLocalTags']) && !empty($user['Role']['perm_sync']) && empty($user['Role']['perm_site_admin'])) {
$options['excludeLocalTags'] = 1;
@ -2023,9 +2064,9 @@ class Event extends AppModel
// do not expose all the data ...
$fields = array('Event.id', 'Event.orgc_id', 'Event.org_id', 'Event.date', 'Event.threat_level_id', 'Event.info', 'Event.published', 'Event.uuid', 'Event.attribute_count', 'Event.analysis', 'Event.timestamp', 'Event.distribution', 'Event.proposal_email_lock', 'Event.user_id', 'Event.locked', 'Event.publish_timestamp', 'Event.sharing_group_id', 'Event.disable_correlation', 'Event.extends_uuid');
$fieldsAtt = array('Attribute.id', 'Attribute.type', 'Attribute.category', 'Attribute.value', 'Attribute.to_ids', 'Attribute.uuid', 'Attribute.event_id', 'Attribute.distribution', 'Attribute.timestamp', 'Attribute.comment', 'Attribute.sharing_group_id', 'Attribute.deleted', 'Attribute.disable_correlation', 'Attribute.object_id', 'Attribute.object_relation');
$fieldsAtt = array('Attribute.id', 'Attribute.type', 'Attribute.category', 'Attribute.value', 'Attribute.to_ids', 'Attribute.uuid', 'Attribute.event_id', 'Attribute.distribution', 'Attribute.timestamp', 'Attribute.comment', 'Attribute.sharing_group_id', 'Attribute.deleted', 'Attribute.disable_correlation', 'Attribute.object_id', 'Attribute.object_relation', 'Attribute.first_seen', 'Attribute.last_seen');
$fieldsObj = array('*');
$fieldsShadowAtt = array('ShadowAttribute.id', 'ShadowAttribute.type', 'ShadowAttribute.category', 'ShadowAttribute.value', 'ShadowAttribute.to_ids', 'ShadowAttribute.uuid', 'ShadowAttribute.event_uuid', 'ShadowAttribute.event_id', 'ShadowAttribute.old_id', 'ShadowAttribute.comment', 'ShadowAttribute.org_id', 'ShadowAttribute.proposal_to_delete', 'ShadowAttribute.timestamp');
$fieldsShadowAtt = array('ShadowAttribute.id', 'ShadowAttribute.type', 'ShadowAttribute.category', 'ShadowAttribute.value', 'ShadowAttribute.to_ids', 'ShadowAttribute.uuid', 'ShadowAttribute.event_uuid', 'ShadowAttribute.event_id', 'ShadowAttribute.old_id', 'ShadowAttribute.comment', 'ShadowAttribute.org_id', 'ShadowAttribute.proposal_to_delete', 'ShadowAttribute.timestamp', 'ShadowAttribute.first_seen', 'ShadowAttribute.last_seen');
$fieldsOrg = array('id', 'name', 'uuid', 'local');
$fieldsServer = array('id', 'url', 'name');
if (!$options['includeAllTags']) {
@ -2086,6 +2127,7 @@ class Event extends AppModel
if ($options['metadata']) {
unset($params['contain']['Attribute']);
unset($params['contain']['ShadowAttribute']);
unset($params['contain']['Object']);
}
if ($user['Role']['perm_site_admin']) {
$params['contain']['User'] = array('fields' => 'email');
@ -2095,7 +2137,6 @@ class Event extends AppModel
return array();
}
// Do some refactoring with the event
$this->Sighting = ClassRegistry::init('Sighting');
$userEmails = array();
$fields = array(
'common' => array('distribution', 'sharing_group_id', 'uuid'),
@ -2148,7 +2189,7 @@ class Event extends AppModel
}
$event['Attribute'] = $this->Feed->attachFeedCorrelations($event['Attribute'], $user, $event['Event'], $overrideLimit);
}
if (!empty($options['includeServerCorrelations']) && $user['org_id'] == Configure::read('MISP.host_org_id')) {
if (!empty($options['includeServerCorrelations']) && ($user['Role']['perm_site_admin'] || $user['org_id'] == Configure::read('MISP.host_org_id'))) {
$this->Feed = ClassRegistry::init('Feed');
if (!empty($options['overrideLimit'])) {
$overrideLimit = true;
@ -2260,7 +2301,14 @@ class Event extends AppModel
}
$event['ShadowAttribute'] = $this->Feed->attachFeedCorrelations($event['ShadowAttribute'], $user, $event['Event'], $overrideLimit, 'Server');
}
$event['Sighting'] = $this->Sighting->attachToEvent($event, $user);
if (empty($options['metadata'])) {
$this->Sighting = ClassRegistry::init('Sighting');
$event['Sighting'] = $this->Sighting->attachToEvent($event, $user);
}
if ($options['includeSightingdb']) {
$this->Sightingdb = ClassRegistry::init('Sightingdb');
$event = $this->Sightingdb->attachToEvent($event, $user);
}
// remove proposals to attributes that we cannot see
// if the shadow attribute wasn't moved within an attribute before, this is the case
if (isset($event['ShadowAttribute'])) {
@ -2654,6 +2702,13 @@ class Event extends AppModel
return $conditions;
}
public function set_filter_seen(&$params, $conditions, $options)
{
$f = $options['scope'] . '.' . $options['filter'];
$conditions = $this->Attribute->setTimestampSeenConditions($params[$options['filter']], $conditions, $f);
return $conditions;
}
public function set_filter_timestamp(&$params, $conditions, $options)
{
if ($options['filter'] == 'from') {
@ -2678,7 +2733,10 @@ class Event extends AppModel
),
'event_timestamp' => array(
'Event.timestamp'
)
),
'attribute_timestamp' => array(
'Attribute.timestamp'
),
);
foreach ($filters[$options['filter']] as $f) {
$conditions = $this->Attribute->setTimestampConditions($params[$options['filter']], $conditions, $f);
@ -2699,140 +2757,6 @@ class Event extends AppModel
return $conditions;
}
public function csv($user, $params, $search = false, &$continue = true)
{
$conditions = array();
$simple_params = array(
'eventid' => array('function' => 'set_filter_eventid'),
'ignore' => array('function' => 'set_filter_ignore'),
'tags' => array('function' => 'set_filter_tags'),
'category' => array('function' => 'set_filter_simple_attribute'),
'type' => array('function' => 'set_filter_simple_attribute'),
'object_relation' => array('function' => 'set_filter_simple_attribute'),
'from' => array('function' => 'set_filter_timestamp'),
'to' => array('function' => 'set_filter_timestamp'),
'last' => array('function' => 'set_filter_timestamp'),
'value' => array('function' => 'set_filter_value'),
'timestamp' => array('function' => 'set_filter_timestamp'),
'attributeIDList' => array('functon' => 'set_filter_attribute_id')
);
foreach ($params as $param => $paramData) {
if (isset($simple_params[$param]) && $params[$param] !== false) {
$options = array(
'filter' => $param,
'scope' => 'Event',
'pop' => !empty($simple_param_scoped[$param]['pop'])
);
$conditions = $this->{$simple_params[$param]['function']}($params, $conditions, $options);
}
}
//$attributeIDList = array(), $includeContext = false, $enforceWarninglist = false
$this->recursive = -1;
if (!empty($params['eventid']) && $params['eventid'] === 'search') {
foreach ($params['attributeIDList'] as $aID) {
$conditions['AND']['OR'][] = array('Attribute.id' => $aID);
}
}
$csv_params = array(
'conditions' => $conditions, //array of conditions
'fields' => array('Attribute.event_id', 'Attribute.distribution', 'Attribute.category', 'Attribute.type', 'Attribute.value', 'Attribute.comment', 'Attribute.uuid', 'Attribute.to_ids', 'Attribute.timestamp', 'Attribute.id', 'Attribute.object_relation'),
'order' => array('Attribute.uuid ASC'),
'flatten' => true
);
// copy over the parameters that have to deal with pagination or additional functionality to be executed
$control_params = array(
'limit', 'page', 'enforceWarninglist'
);
foreach ($control_params as $control_param) {
if (!empty($params[$control_param])) {
$csv_params[$control_param] = $params[$control_param];
}
}
$csv_params = $this->__appendIncludesCSV($csv_params, !empty($params['includeContext']));
$attributes = $this->Attribute->fetchAttributes($user, $csv_params, $continue);
$attributes = $this->__sanitiseCSVAttributes($attributes, !empty($params['includeContext']), !empty($params['ignore']));
return $attributes;
}
private function __appendIncludesCSV($params, $includeContext)
{
if ($includeContext) {
$params['contain'] = array(
'Event' => array(
'fields' => array('id', 'info', 'org_id', 'orgc_id', 'date', 'distribution', 'analysis'),
'SharingGroup' => array('fields' => array('id', 'name')),
'Org' => array('id', 'name'),
'Orgc' => array('id', 'name'),
'ThreatLevel' => array(
'fields' => array('id', 'name'),
),
'EventTag' => array(
'Tag' => array(
'fields' => array('id', 'name')
)
)
),
);
}
$params['contain']['Object'] = array('fields' => array('id', 'uuid', 'name', 'meta-category'));
return $params;
}
private function __sanitiseCSVAttributes($attributes, $includeContext, $ignore)
{
if (!empty($ignore)) {
$this->Whitelist = ClassRegistry::init('Whitelist');
$attributes = $this->Whitelist->removeWhitelistedFromArray($attributes, true);
}
foreach ($attributes as &$attribute) {
$this->__escapeCSVField($attribute['Attribute']['value']);
$this->__escapeCSVField($attribute['Attribute']['comment']);
$this->__escapeCSVField($attribute['Attribute']['object_relation']);
$this->__escapeCSVField($attribute['Attribute']['uuid']);
$this->__escapeCSVField($attribute['Attribute']['category']);
$this->__escapeCSVField($attribute['Attribute']['type']);
$attribute['Attribute']['timestamp'] = date('Ymd', $attribute['Attribute']['timestamp']);
if (empty($attribute['Object'])) {
$attribute['Object']['uuid'] = '""';
$attribute['Object']['name'] = '';
$attribute['Object']['meta-category'] = '';
}
$this->__escapeCSVField($attribute['Object']['name']);
$this->__escapeCSVField($attribute['Object']['uuid']);
$this->__escapeCSVField($attribute['Object']['meta-category']);
if ($includeContext) {
$this->__escapeCSVField($attribute['Event']['info']);
$this->__escapeCSVField($attribute['Event']['uuid']);
$this->__escapeCSVField($attribute['Org']['name']);
$this->__escapeCSVField($attribute['Orgc']['name']);
$attribute['Event']['Tag']['name'] = '';
$attribute['attribute_tag'] = '';
if (!empty($attribute['AttributeTag'])) {
$tags = array();
foreach ($attribute['AttributeTag'] as $attributeTag) {
if (!empty($attributeTag['Tag']['name'])) {
$tags[] = $attributeTag['Tag']['name'];
}
}
$attribute['Attribute']['attribute_tag'] = implode(',', $tags);
}
$this->__escapeCSVField($attribute['Attribute']['attribute_tag']);
if (!empty($attribute['Event']['EventTag'])) {
$tags = array();
foreach ($attribute['Event']['EventTag'] as $eventTag) {
if (!empty($eventTag['Tag']['name'])) {
$tags[] = $eventTag['Tag']['name'];
}
}
$attribute['Event']['Tag']['name'] = implode(',', $tags);
}
$this->__escapeCSVField($attribute['Event']['Tag']['name']);
}
}
return $attributes;
}
public function sendAlertEmailRouter($id, $user, $oldpublish = null)
{
if (Configure::read('MISP.block_old_event_alert')) {
@ -3164,7 +3088,7 @@ class Event extends AppModel
$bodyevent = $temp[0];
$body = $temp[1];
$result = true;
$tplColorString = !empty(Configure::read('MISP.email_subject_TLP_string')) ? Configure::read('MISP.email_subject_TLP_string') : "TLP Amber";
$tplColorString = !empty(Configure::read('MISP.email_subject_TLP_string')) ? Configure::read('MISP.email_subject_TLP_string') : "tlp:amber";
$subject = "[" . Configure::read('MISP.org') . " MISP] Need info about event " . $id . " - ".$tplColorString;
$result = $this->User->sendEmail($reporter, $bodyevent, $body, $subject, $user) && $result;
}
@ -4031,8 +3955,8 @@ class Event extends AppModel
return true;
}
// Uploads this specific event to all remote servers
public function uploadEventToServersRouter($id, $passAlong = null)
// Uploads this specific event or sightings to all remote servers
public function uploadEventToServersRouter($id, $passAlong = null, $scope = 'events')
{
$eventOrgcId = $this->find('first', array(
'conditions' => array('Event.id' => $id),
@ -4060,7 +3984,11 @@ class Event extends AppModel
$event['Event']['locked'] = 1;
// get a list of the servers
$this->Server = ClassRegistry::init('Server');
$conditions = array('push' => 1);
if ($scope === 'sightings') {
$conditions = array('push_sightings' => 1);
} else {
$conditions = array('push' => 1);
}
if ($passAlong) {
$conditions[] = array('Server.id !=' => $passAlong);
}
@ -4076,13 +4004,18 @@ class Event extends AppModel
$failedServers = array();
App::uses('SyncTool', 'Tools');
foreach ($servers as &$server) {
if ((!isset($server['Server']['internal']) || !$server['Server']['internal']) && $event['Event']['distribution'] < 2) {
if (
($scope === 'events' &&
(!isset($server['Server']['internal']) || !$server['Server']['internal']) && $event['Event']['distribution'] < 2) ||
($scope === 'sightings' &&
(!isset($server['Server']['push_sightings']) || !$server['Server']['push_sightings']))
) {
continue;
}
$syncTool = new SyncTool();
$HttpSocket = $syncTool->setupHttpSocket($server);
// Skip servers where the event has come from.
if (($passAlong != $server)) {
if (($passAlong != $server['Server']['id'])) {
$params = array();
if (!empty($server['Server']['push_rules'])) {
$push_rules = json_decode($server['Server']['push_rules'], true);
@ -4100,14 +4033,31 @@ class Event extends AppModel
$event = $this->fetchEvent($elevatedUser, $params);
$event = $event[0];
$event['Event']['locked'] = 1;
$thisUploaded = $this->uploadEventToServer($event, $server, $HttpSocket);
// attach sightings if needed
if ($scope === 'sightings') {
$this->Sighting = ClassRegistry::init('Sighting');
$fakeSyncUser = array(
'org_id' => $server['Server']['remote_org_id'],
'Role' => array(
'perm_site_admin' => 0
)
);
$sightings = $this->Sighting->attachToEvent($event, $fakeSyncUser);
if (!empty($sightings)) {
$thisUploaded = $this->uploadSightingsToServer($sightings, $server, $event['Event']['uuid'], $HttpSocket);
} else {
$thisUploaded = true;
}
} else {
$thisUploaded = $this->uploadEventToServer($event, $server, $HttpSocket, $scope);
if (isset($this->data['ShadowAttribute'])) {
$this->Server->syncProposals($HttpSocket, $server, null, $id, $this);
}
}
if (!$thisUploaded) {
$uploaded = !$uploaded ? $uploaded : $thisUploaded;
$failedServers[] = $server['Server']['url'];
}
if (isset($this->data['ShadowAttribute'])) {
$this->Server->syncProposals($HttpSocket, $server, null, $id, $this);
}
}
}
if (!$uploaded) {
@ -4133,9 +4083,16 @@ class Event extends AppModel
return $workerType;
}
public function publishRouter($id, $passAlong = null, $user)
public function publishRouter($id, $passAlong = null, $user, $scope = 'events')
{
if (Configure::read('MISP.background_jobs')) {
$job_type = 'publish_' . $scope;
$function = 'publish';
$message = 'Publishing.';
if ($scope === 'sightings') {
$message = 'Publishing sightings.';
$function = 'publish_sightings';
}
$job = ClassRegistry::init('Job');
$job->create();
$data = array(
@ -4146,24 +4103,56 @@ class Event extends AppModel
'retries' => 0,
'org_id' => $user['org_id'],
'org' => $user['Organisation']['name'],
'message' => 'Publishing.',
'message' => $message
);
$job->save($data);
$jobId = $job->id;
$process_id = CakeResque::enqueue(
'prio',
'EventShell',
array('publish', $id, $passAlong, $jobId, $user['id']),
array($function, $id, $passAlong, $jobId, $user['id']),
true
);
$job->saveField('process_id', $process_id);
return $process_id;
} elseif ($scope === 'sightings') {
$result = $this->publish_sightings($id, $passAlong);
return $result;
} else {
$result = $this->publish($id, $passAlong);
return $result;
}
}
public function publish_sightings($id, $passAlong = null, $jobId = null)
{
if (is_numeric($id)) {
$condition = array('Event.id' => $id);
} else {
$condition = array('Event.uuid' => $id);
}
$event = $this->find('first', array(
'recursive' => -1,
'conditions' => $condition
));
if (empty($event)) {
return false;
}
if ($jobId) {
$this->Behaviors->unload('SysLogLogable.SysLogLogable');
} else {
// update the DB to set the sightings timestamp
// for background jobs, this should be done already
$fieldList = array('id', 'info', 'sighting_timestamp');
$event['Event']['sighting_timestamp'] = time();
$event['Event']['skip_zmq'] = 1;
$event['Event']['skip_kafka'] = 1;
$this->save($event, array('fieldList' => $fieldList));
}
$uploaded = $this->uploadEventToServersRouter($id, $passAlong, 'sightings');
return $uploaded;
}
// Performs all the actions required to publish an event
public function publish($id, $passAlong = null, $jobId = null)
{
@ -4434,186 +4423,37 @@ class Event extends AppModel
return false;
}
public function removeOlder(&$eventArray)
public function removeOlder(&$eventArray, $scope = 'events')
{
if ($scope === 'sightings' ) {
$field = 'sighting_timestamp';
} else {
$field = 'timestamp';
}
$uuidsToCheck = array();
foreach ($eventArray as $k => &$event) {
$uuidsToCheck[$event['uuid']] = $k;
}
$localEvents = array();
$temp = $this->find('all', array('recursive' => -1, 'fields' => array('Event.uuid', 'Event.timestamp', 'Event.locked')));
$temp = $this->find('all', array('recursive' => -1, 'fields' => array('Event.uuid', 'Event.' . $field, 'Event.locked')));
foreach ($temp as $e) {
$localEvents[$e['Event']['uuid']] = array('timestamp' => $e['Event']['timestamp'], 'locked' => $e['Event']['locked']);
$localEvents[$e['Event']['uuid']] = array($field => $e['Event'][$field], 'locked' => $e['Event']['locked']);
}
foreach ($uuidsToCheck as $uuid => $eventArrayId) {
if (isset($localEvents[$uuid]) && ($localEvents[$uuid]['timestamp'] >= $eventArray[$eventArrayId]['timestamp'] || !$localEvents[$uuid]['locked'])) {
// remove all events for the sighting sync if the remote is not aware of the new field yet
if (!isset($eventArray[$eventArrayId][$field])) {
unset($eventArray[$eventArrayId]);
} else {
if (isset($localEvents[$uuid])
&& ($localEvents[$uuid][$field] >= $eventArray[$eventArrayId][$field]
|| ($scope === 'events' && !$localEvents[$uuid]['locked'])))
{
unset($eventArray[$eventArrayId]);
}
}
}
}
public function stix2($id, $tags, $attachments, $user, $returnType = 'json', $from = false, $to = false, $last = false, $jobId = false, $returnFile = false)
{
$eventIDs = $this->Attribute->dissectArgs($id);
$tagIDs = $this->Attribute->dissectArgs($tags);
$idList = $this->getAccessibleEventIds($eventIDs[0], $eventIDs[1], $tagIDs[0], $tagIDs[1]);
if (!empty($idList)) {
$event_ids = $this->fetchEventIds($user, $from, $to, $last, true);
$event_ids = array_intersect($event_ids, $idList);
}
$randomFileName = $this->generateRandomFileName();
$scriptDir = APP . "files/scripts/";
$stix2_framing_cmd = $this->getPythonVersion() . ' ' . $scriptDir . 'misp_framing.py stix2 ' . escapeshellarg(CakeText::uuid()) . ' 2>' . APP . 'tmp/logs/exec-errors.log';
$stix2_framing = json_decode(shell_exec($stix2_framing_cmd), true);
if (empty($stix2_framing)) {
return array('success' => 0, 'message' => 'There was an issue generating the STIX 2.0 export.');
}
$separator = $stix2_framing['separator'];
$tmpDir = $scriptDir . "tmp/";
$stixFile = new File($tmpDir . $randomFileName . ".stix");
$stixFile->write($stix2_framing['header']);
if ($jobId) {
$this->Job = ClassRegistry::init('Job');
$this->Job->id = $jobId;
if (!$this->Job->exists()) {
$jobId = false;
}
}
$i = 0;
$eventCount = count($event_ids);
if ($event_ids) {
foreach ($event_ids as $event_id) {
$tempFile = new File($tmpDir . $randomFileName, true, 0644);
$event = $this->fetchEvent($user, array('eventid' => $event_id, 'includeAttachments' => $attachments));
if (empty($event)) {
continue;
}
$event[0]['Tag'] = array();
foreach ($event[0]['EventTag'] as $tag) {
$event[0]['Tag'][] = $tag['Tag'];
}
App::uses('JSONConverterTool', 'Tools');
$converter = new JSONConverterTool();
$event = $converter->convert($event[0]);
$tempFile->write($event);
unset($event);
$scriptFile = $scriptDir . "stix2/misp2stix2.py ";
$result = shell_exec($this->getPythonVersion() . ' ' . $scriptFile . $tempFile->path . ' 2>' . APP . 'tmp/logs/exec-errors.log');
$decoded = json_decode($result, true);
if (isset($decoded['success']) && $decoded['success'] == 1) {
$file = new File($tmpDir . $randomFileName . '.out', true, 0644);
$result = substr($file->read(), 1, -1);
$file->delete();
$stixFile->append($result . (($i + 1) != $eventCount ? $separator : ''));
} else {
return false;
}
$i++;
if ($jobId) {
$this->Job->saveField('message', 'Event ' . $i . '/' . $eventCount);
if ($i % 10 == 0) {
$this->Job->saveField('progress', $i * 80 / $eventCount);
}
}
$tempFile->close();
}
}
$stixFile->append($stix2_framing['footer']);
if ($tempFile) {
$tempFile->delete();
}
if (!$returnFile) {
$data2return = $stixFile->read();
$stixFile->delete();
}
return array('success' => 1, 'data' => $returnFile ? $stixFile->path : $data2return);
}
public function stix($id, $tags, $attachments, $user, $returnType = 'xml', $from = false, $to = false, $last = false, $jobId = false, $returnFile = false)
{
$eventIDs = $this->Attribute->dissectArgs($id);
$tagIDs = $this->Attribute->dissectArgs($tags);
$idList = $this->getAccessibleEventIds($eventIDs[0], $eventIDs[1], $tagIDs[0], $tagIDs[1]);
if (!empty($idList)) {
$event_ids = $this->fetchEventIds($user, $from, $to, $last, true);
$event_ids = array_intersect($event_ids, $idList);
}
$randomFileName = $this->generateRandomFileName();
$tmpDir = APP . "files" . DS . "scripts";
$stix_framing_cmd = $this->getPythonVersion() . ' ' . $tmpDir . DS . 'misp_framing.py stix ' . $this->__getAnnounceBaseurl() . ' ' . escapeshellarg(Configure::read('MISP.org')) . ' ' . escapeshellarg($returnType) . ' 2>' . APP . 'tmp/logs/exec-errors.log';
$stix_framing = json_decode(shell_exec($stix_framing_cmd), true);
if (empty($stix_framing)) {
return array('success' => 0, 'message' => 'There was an issue generating the STIX export.');
}
$separator = $stix_framing['separator'];
$tmpDir = $tmpDir . DS . "tmp";
$stixFile = new File($tmpDir . DS . $randomFileName . ".stix");
$stixFile->write($stix_framing['header']);
$result = array();
if ($jobId) {
$this->Job = ClassRegistry::init('Job');
$this->Job->id = $jobId;
if (!$this->Job->exists()) {
$jobId = false;
}
}
$i = 0;
$eventCount = count($event_ids);
if ($event_ids) {
foreach ($event_ids as $event_id) {
$tempFile = new File($tmpDir . DS . $randomFileName, true, 0644);
$event = $this->fetchEvent($user, array('eventid' => $event_id, 'includeAttachments' => $attachments));
if (empty($event)) {
continue;
}
$event[0]['Tag'] = array();
foreach ($event[0]['EventTag'] as $tag) {
$event[0]['Tag'][] = $tag['Tag'];
}
App::uses('JSONConverterTool', 'Tools');
$converter = new JSONConverterTool();
$event = $converter->convert($event[0]);
$tempFile->write($event);
unset($event);
$scriptFile = APP . "files" . DS . "scripts" . DS . "misp2stix.py";
$result = shell_exec($this->getPythonVersion() . ' ' . $scriptFile . ' ' . $randomFileName . ' ' . escapeshellarg($returnType) . ' ' . $this->__getAnnounceBaseurl() . ' ' . escapeshellarg(Configure::read('MISP.org')) . ' 2>' . APP . 'tmp/logs/exec-errors.log');
// The result of the script will be a returned JSON object with 2 variables: success (boolean) and message
// If success = 1 then the temporary output file was successfully written, otherwise an error message is passed along
$decoded = json_decode($result, true);
if (!isset($decoded['success']) || !$decoded['success']) {
$tempFile->delete();
$stixFile->delete();
return array('success' => 0, 'message' => $decoded['message']);
}
$file = new File(APP . "files" . DS . "scripts" . DS . "tmp" . DS . $randomFileName . ".out");
$stix_event = $file->read();
if (($i + 1) != $eventCount) {
$stix_event .= $separator;
}
$stixFile->append($stix_event);
$file->close();
$file->delete();
$i++;
if ($jobId) {
$this->Job->saveField('message', 'Event ' . $i . '/' . $eventCount);
if ($i % 10 == 0) {
$this->Job->saveField('progress', $i * 80 / $eventCount);
}
}
$tempFile->close();
}
}
$stixFile->append($stix_framing['footer']);
if ($tempFile) {
$tempFile->delete();
}
if (!$returnFile) {
$data = $stixFile->read();
$stixFile->delete();
}
return array('success' => 1, 'data' => $returnFile ? $stixFile->path : $data);
}
public function getAccessibleEventIds($include, $exclude, $includedTags, $excludedTags)
{
$conditions = array();
@ -5436,11 +5276,26 @@ class Event extends AppModel
private function __fillAttribute($attribute, $defaultDistribution)
{
if (!isset($attribute['category'])) {
$attribute['category'] = $this->Event->Attribute->typeDefinitions[$attribute['type']]['default_category'];
if (is_array($attribute['type'])) {
$attribute_type = $attribute['type'][0];
if (empty($attribute['category'])) {
$categories = array();
foreach ($attribute['type'] as $type) {
$category = $this->Attribute->typeDefinitions[$type]['default_category'];
if (!in_array($category, $categories)) {
$categories[] = $category;
}
}
$attribute['category'] = count($categories) === 1 ? $categories[0] : $categories;
}
} else {
$attribute_type = $attribute['type'];
if (empty($attribute['category'])) {
$attribute['category'] = $this->Attribute->typedefinitions[$attribute_type]['default_category'];
}
}
if (!isset($attribute['to_ids'])) {
$attribute['to_ids'] = $this->Event->Attribute->typeDefinitions[$attribute['type']]['to_ids'];
$attribute['to_ids'] = $this->Attribute->typeDefinitions[$attribute_type]['to_ids'];
}
$attribute['value'] = $this->Attribute->runRegexp($attribute['type'], $attribute['value']);
$attribute['distribution'] = (isset($attribute['distribution']) ? (int)$attribute['distribution'] : $defaultDistribution);
@ -6011,6 +5866,7 @@ class Event extends AppModel
unset($data[$dataType . 'Tag'][$k]);
continue;
}
$dataTag['Tag']['local'] = empty($dataTag['local']) ? 0 : 1;
if (!isset($excludeGalaxy) || !$excludeGalaxy) {
if (substr($dataTag['Tag']['name'], 0, strlen('misp-galaxy:')) === 'misp-galaxy:') {
$cluster = $this->GalaxyCluster->getCluster($dataTag['Tag']['name']);
@ -6733,6 +6589,10 @@ class Event extends AppModel
$filters['wildcard'] = $filters['searchall'];
}
}
$subqueryElements = $this->harvestSubqueryElements($filters);
$filters = $this->addFiltersFromSubqueryElements($filters, $subqueryElements);
$filters['include_attribute_count'] = 1;
$eventid = $this->filterEventIds($user, $filters, $elementCounter);
$eventCount = count($eventid);
@ -6935,4 +6795,57 @@ class Event extends AppModel
}
return true;
}
public function harvestSubqueryElements($options)
{
$acceptedRules = array(
'galaxy' => 1,
'org' => array('sector', 'local', 'nationality')
);
$subqueryElement = array(
'galaxy' => array(),
'org' => array(),
);
foreach($options as $rule => $value) {
$split = explode(".", $rule, 2);
if (count($split) > 1) {
$scope = $split[0];
$element = $split[1];
if (isset($acceptedRules[$scope])) {
if (is_array($acceptedRules[$scope]) && !in_array($element, $acceptedRules[$scope])) {
continue;
} else {
$subqueryElement[$scope][$element] = $value;
}
}
}
}
return $subqueryElement;
}
public function addFiltersFromSubqueryElements($filters, $subqueryElements)
{
if (!empty($subqueryElements['galaxy'])) {
$this->GalaxyCluster = ClassRegistry::init('GalaxyCluster');
$tagsFromGalaxyMeta = $this->GalaxyCluster->getClusterTagsFromMeta($subqueryElements['galaxy']);
if (empty($tagsFromGalaxyMeta)) {
$filters['eventid'] = -1;
}
if (!empty($filters['tags'])) {
$filters['tags'][] = $tagsFromGalaxyMeta;
} else {
$filters['tags'] = $tagsFromGalaxyMeta;
}
}
if (!empty($subqueryElements['org'])) {
$Organisation = ClassRegistry::init('Organisation');
$orgcIdsFromMeta = $Organisation->getOrgIdsFromMeta($subqueryElements['org']);
if (!empty($filters['org'])) {
$filters['org'][] = $orgcIdsFromMeta;
} else {
$filters['org'] = $orgcIdsFromMeta;
}
}
return $filters;
}
}

11
app/Model/Feed.php
View File

@ -365,19 +365,12 @@ class Feed extends AppModel
if ($scope === 'Server' || $source[$scope]['source_format'] == 'misp') {
$pipe = $redis->multi(Redis::PIPELINE);
$eventUuidHitPosition = array();
$i = 0;
foreach ($objects as $k => $object) {
if (isset($object[$scope])) {
foreach ($object[$scope] as $currentFeed) {
if ($source[$scope]['id'] == $currentFeed['id']) {
$eventUuidHitPosition[$i] = $k;
$i++;
if (in_array($object['type'], $compositeTypes)) {
$value = explode('|', $object['value']);
$redis->smembers($cachePrefix . 'event_uuid_lookup:' . md5($value[0]));
} else {
$redis->smembers($cachePrefix . 'event_uuid_lookup:' . md5($object['value']));
}
$eventUuidHitPosition[] = $k;
$redis->smembers($cachePrefix . 'event_uuid_lookup:' . $hashTable[$k]);
}
}
}

8
app/Model/Galaxy.php
View File

@ -378,8 +378,12 @@ class Galaxy extends AppModel
{
$galaxy = $this->find('first', array(
'recursive' => -1,
'fields' => 'id',
'conditions' => array('Galaxy.type' => $type, 'Galaxy.namespace' => $namespace),
'fields' => array('MAX(Galaxy.version) as latest_version', 'id'),
'conditions' => array(
'Galaxy.type' => $type,
'Galaxy.namespace' => $namespace
),
'group' => array('name', 'id')
));
return empty($galaxy) ? 0 : $galaxy['Galaxy']['id'];
}

145
app/Model/GalaxyCluster.php
View File

@ -142,32 +142,98 @@ class GalaxyCluster extends AppModel
*/
public function getCluster($name)
{
$conditions = array('LOWER(GalaxyCluster.tag_name)' => strtolower($name));
if (is_numeric($name)) {
$conditions = array('GalaxyCluster.id' => $name);
}
if (isset($this->__clusterCache[$name])) {
return $this->__clusterCache[$name];
}
$objects = array('Galaxy', 'GalaxyElement');
if (is_numeric($name)) {
$conditions = array('GalaxyCluster.id' => $name);
} else {
$conditions = array('LOWER(GalaxyCluster.tag_name)' => strtolower($name));
}
$cluster = $this->find('first', array(
'conditions' => $conditions,
'contain' => array('Galaxy', 'GalaxyElement')
));
if (!empty($cluster)) {
if (isset($cluster['Galaxy'])) {
$cluster['GalaxyCluster']['Galaxy'] = $cluster['Galaxy'];
unset($cluster['Galaxy']);
}
$elements = array();
foreach ($cluster['GalaxyElement'] as $element) {
if (!isset($elements[$element['key']])) {
$elements[$element['key']] = array($element['value']);
} else {
$elements[$element['key']][] = $element['value'];
$cluster = $this->postprocess($cluster);
}
$this->__clusterCache[$name] = $cluster;
return $cluster;
}
/**
* @param array $events
* @param bool $replace
* @return array
*/
public function attachClustersToEventIndex(array $events, $replace = false)
{
$clusterTagNames = array();
foreach ($events as $event) {
foreach ($event['EventTag'] as $k2 => $eventTag) {
if (substr($eventTag['Tag']['name'], 0, strlen('misp-galaxy:')) === 'misp-galaxy:') {
$clusterTagNames[] = strtolower($eventTag['Tag']['name']);
}
}
unset($cluster['GalaxyElement']);
}
$clusters = $this->find('all', array(
'conditions' => array('LOWER(GalaxyCluster.tag_name)' => $clusterTagNames),
'contain' => array('Galaxy', 'GalaxyElement'),
));
$clustersByTagName = array();
foreach ($clusters as $cluster) {
$clustersByTagName[strtolower($cluster['GalaxyCluster']['tag_name'])] = $cluster;
}
foreach ($events as $k => $event) {
foreach ($event['EventTag'] as $k2 => $eventTag) {
$tagName = strtolower($eventTag['Tag']['name']);
if (isset($clustersByTagName[$tagName])) {
$cluster = $this->postprocess($clustersByTagName[$tagName], $eventTag['Tag']['id']);
$cluster['GalaxyCluster']['tag_id'] = $eventTag['Tag']['id'];
$cluster['GalaxyCluster']['local'] = $eventTag['local'];
$events[$k]['GalaxyCluster'][] = $cluster['GalaxyCluster'];
if ($replace) {
unset($events[$k]['EventTag'][$k2]);
}
}
}
}
return $events;
}
/**
* @param array $cluster
* @param int|null $tagId
* @return array
*/
private function postprocess(array $cluster, $tagId = null)
{
if (isset($cluster['Galaxy'])) {
$cluster['GalaxyCluster']['Galaxy'] = $cluster['Galaxy'];
unset($cluster['Galaxy']);
}
$elements = array();
foreach ($cluster['GalaxyElement'] as $element) {
if (!isset($elements[$element['key']])) {
$elements[$element['key']] = array($element['value']);
} else {
$elements[$element['key']][] = $element['value'];
}
}
unset($cluster['GalaxyElement']);
$cluster['GalaxyCluster']['meta'] = $elements;
if ($tagId) {
$cluster['GalaxyCluster']['tag_id'] = $tagId;
} else {
$this->Tag = ClassRegistry::init('Tag');
$tag_id = $this->Tag->find(
'first',
@ -182,29 +248,40 @@ class GalaxyCluster extends AppModel
if (!empty($tag_id)) {
$cluster['GalaxyCluster']['tag_id'] = $tag_id['Tag']['id'];
}
$cluster['GalaxyCluster']['meta'] = $elements;
}
$this->__clusterCache[$name] = $cluster;
return $cluster;
}
public function attachClustersToEventIndex($events, $replace = false)
public function getClusterTagsFromMeta($galaxyElements)
{
foreach ($events as $k => $event) {
foreach ($event['EventTag'] as $k2 => $eventTag) {
if (substr($eventTag['Tag']['name'], 0, strlen('misp-galaxy:')) === 'misp-galaxy:') {
$cluster = $this->getCluster($eventTag['Tag']['name']);
if ($cluster) {
$cluster['GalaxyCluster']['tag_id'] = $eventTag['Tag']['id'];
$cluster['GalaxyCluster']['local'] = $eventTag['local'];
$events[$k]['GalaxyCluster'][] = $cluster['GalaxyCluster'];
if ($replace) {
unset($events[$k]['EventTag'][$k2]);
}
}
}
}
// AND operator between cluster metas
$tmpResults = array();
foreach ($galaxyElements as $galaxyElementKey => $galaxyElementValue) {
$tmpResults[] = array_values($this->GalaxyElement->find('list', array(
'conditions' => array(
'key' => $galaxyElementKey,
'value' => $galaxyElementValue,
),
'fields' => array('galaxy_cluster_id'),
'recursive' => -1
)));
}
return $events;
$clusterTags = array();
if (!empty($tmpResults)) {
// Get all Clusters matching all conditions
$matchingClusters = $tmpResults[0];
array_shift($tmpResults);
foreach ($tmpResults as $tmpResult) {
$matchingClusters = array_intersect($matchingClusters, $tmpResult);
}
$clusterTags = $this->find('list', array(
'conditions' => array('id' => $matchingClusters),
'fields' => array('GalaxyCluster.tag_name'),
'recursive' => -1
));
}
return array_values($clusterTags);
}
}

98
app/Model/Log.php
View File

@ -16,51 +16,56 @@ class Log extends AppModel
);
public $validate = array(
'action' => array(
'rule' => array('inList', array(
'accept',
'accept_delegation',
'add',
'admin_email',
'auth',
'auth_fail',
'blacklisted',
'change_pw',
'delete',
'disable',
'discard',
'edit',
'email',
'enable',
'error',
'export',
'file_upload',
'galaxy',
'include_formula',
'login',
'login_fail',
'logout',
'merge',
'pruneUpdateLogs',
'publish',
'publish alert',
'pull',
'purge_events',
'push',
'remove_dead_workers',
'request',
'request_delegation',
'reset_auth_key',
'security',
'serverSettingsEdit',
'tag',
'undelete',
'update',
'update_database',
'upgrade_24',
'upload_sample',
'version_warning',
'warning'
)),
'rule' => array(
'inList',
array( // ensure that the length of the rules is < 20 in length
'accept',
'accept_delegation',
'add',
'admin_email',
'auth',
'auth_fail',
'blacklisted',
'change_pw',
'delete',
'disable',
'discard',
'edit',
'email',
'enable',
'error',
'export',
'file_upload',
'galaxy',
'include_formula',
'login',
'login_fail',
'logout',
'merge',
'pruneUpdateLogs',
'publish',
'publish_sightings',
'publish alert',
'pull',
'purge_events',
'push',
'remove_dead_workers',
'request',
'request_delegation',
'reset_auth_key',
'security',
'serverSettingsEdit',
'tag',
'undelete',
'update',
'update_database',
'update_db_worker',
'upgrade_24',
'upload_sample',
'version_warning',
'warning'
)
),
'message' => 'Options : ...'
)
);
@ -184,6 +189,7 @@ class Log extends AppModel
* @param int $modelId
* @param string $title
* @param string|array $change
* @return array
* @throws Exception
*/
public function createLogEntry($user, $action, $model, $modelId = 0, $title = '', $change = '')
@ -222,6 +228,8 @@ class Log extends AppModel
if (!$result) {
throw new Exception("Cannot save log because of validation errors: " . json_encode($this->validationErrors));
}
return $result;
}
// to combat a certain bug that causes the upgrade scripts to loop without being able to set the correct version

413
app/Model/MispObject.php
View File

@ -46,6 +46,8 @@ class MispObject extends AppModel
),
);
public $shortDist = array(0 => 'Organisation', 1 => 'Community', 2 => 'Connected', 3 => 'All', 4 => ' Sharing Group', 5 => 'Inherit');
public $validate = array(
'uuid' => array(
'uuid' => array(
@ -60,6 +62,18 @@ class MispObject extends AppModel
)
);
public function afterFind($results, $primary = false)
{
foreach ($results as $k => $v) {
$results[$k] = $this->Attribute->UTCToISODatetime($results[$k], $this->alias);
}
return $results;
}
public function beforeSave($options = array()) {
$this->data = $this->Attribute->ISODatetimeToUTC($this->data, $this->alias);
}
public function beforeValidate($options = array())
{
parent::beforeValidate();
@ -75,6 +89,14 @@ class MispObject extends AppModel
$date = new DateTime();
$this->data[$this->alias]['timestamp'] = $date->getTimestamp();
}
// parse first_seen different formats
if (isset($this->data[$this->alias]['first_seen'])) {
$this->data[$this->alias]['first_seen'] = $this->data[$this->alias]['first_seen'] === '' ? null : $this->data[$this->alias]['first_seen'];
}
// parse last_seen different formats
if (isset($this->data[$this->alias]['last_seen'])) {
$this->data[$this->alias]['last_seen'] = $this->data[$this->alias]['last_seen'] === '' ? null : $this->data[$this->alias]['last_seen'];
}
if (empty($this->data[$this->alias]['template_version'])) {
$this->data[$this->alias]['template_version'] = 1;
}
@ -184,6 +206,18 @@ class MispObject extends AppModel
$result = $this->id;
foreach ($object['Attribute'] as $k => $attribute) {
$object['Attribute'][$k]['object_id'] = $this->id;
if (
(!array_key_exists('first_seen', $object['Attribute'][$k]) || is_null($object['Attribute'][$k]['first_seen'])) &&
(array_key_exists('first_seen', $object['Object']) && !is_null($object['Object']['first_seen']))
) {
$object['Attribute'][$k]['first_seen'] = $object['Object']['first_seen'];
}
if (
(!array_key_exists('last_seen', $object['Attribute'][$k]) || is_null($object['Attribute'][$k]['last_seen'])) &&
(array_key_exists('last_seen', $object['Object']) && !is_null($object['Object']['last_seen']))
) {
$object['Attribute'][$k]['last_seen'] = $object['Object']['last_seen'];
}
}
$this->Attribute->saveAttributes($object['Attribute']);
} else {
@ -254,6 +288,28 @@ class MispObject extends AppModel
return $conditions;
}
public function fetchObjectSimple($user, $options = array())
{
$params = array(
'conditions' => $this->buildConditions($user),
'fields' => array(),
'recursive' => -1
);
if (isset($options['conditions'])) {
$params['conditions']['AND'][] = $options['conditions'];
}
if (isset($options['fields'])) {
$params['fields'] = $options['fields'];
}
$results = $this->find('all', array(
'conditions' => $params['conditions'],
'recursive' => -1,
'fields' => $params['fields'],
'contain' => array('Event' => array('distribution', 'id', 'user_id', 'orgc_id', 'org_id')),
'sort' => false
));
return $results;
}
// Method that fetches all objects
// very flexible, it's basically a replacement for find, with the addition that it restricts access based on user
@ -489,12 +545,57 @@ class MispObject extends AppModel
$tmpfile->delete();
$tmpfile->close();
}
if (!isset($attributes['Attribute'][$k]['first_seen'])) {
$attributes['Attribute'][$k]['first_seen'] = null;
}
if (!isset($attributes['Attribute'][$k]['last_seen'])) {
$attributes['Attribute'][$k]['last_seen'] = null;
}
unset($attributes['Attribute'][$k]['save']);
}
return $attributes;
}
public function deltaMerge($object, $objectToSave)
// Set Object's *-seen (and ObjectAttribute's *-seen and ObjectAttribute's value if requested) to the provided *-seen value
// Therefore, synchronizing the 3 values
public function syncObjectAndAttributeSeen($object, $forcedSeenOnElements, $applyOnAttribute=True) {
if (empty($forcedSeenOnElements)) {
return $object;
}
if (isset($forcedSeenOnElements['first_seen'])) {
$object['Object']['first_seen'] = $forcedSeenOnElements['first_seen'];
}
if (isset($forcedSeenOnElements['last_seen'])) {
$object['Object']['last_seen'] = $forcedSeenOnElements['last_seen'];
}
if ($applyOnAttribute) {
if (isset($object['Attribute'])) {
$attributes = $object['Attribute'];
} else {
$attributes = $this->find('first', array(
'conditions' => array('id' => $object['Object']['id']),
'contain' => array('Attribute')
))['Attribute'];
}
foreach($attributes as $i => $attribute) {
if (isset($forcedSeenOnElements['first_seen'])) {
$attributes[$i]['first_seen'] = $forcedSeenOnElements['first_seen'];
if ($attribute['object_relation'] == 'first-seen') {
$attributes[$i]['value'] = $forcedSeenOnElements['first_seen'];
}
} elseif (isset($forcedSeenOnElements['last_seen'])) {
$attributes[$i]['last_seen'] = $forcedSeenOnElements['last_seen'];
if ($attribute['object_relation'] == 'last-seen') {
$attributes[$i]['value'] = $forcedSeenOnElements['last_seen'];
}
}
}
$object['Attribute'] = $attributes;
}
return $object;
}
public function deltaMerge($object, $objectToSave, $onlyAddNewAttribute=false)
{
if (!isset($objectToSave['Object'])) {
$dataToBackup = array('ObjectReferences', 'Attribute', 'ShadowAttribute');
@ -513,69 +614,141 @@ class MispObject extends AppModel
}
unset($dataToBackup);
}
$object['Object']['comment'] = $objectToSave['Object']['comment'];
$object['Object']['distribution'] = $objectToSave['Object']['distribution'];
if ($object['Object']['distribution'] == 4) {
$object['Object']['sharing_group_id'] = $objectToSave['Object']['sharing_group_id'];
if (isset($objectToSave['Object']['comment'])) {
$object['Object']['comment'] = $objectToSave['Object']['comment'];
}
if (isset($objectToSave['Object']['distribution'])) {
$object['Object']['distribution'] = $objectToSave['Object']['distribution'];
if ($object['Object']['distribution'] == 4) {
$object['Object']['sharing_group_id'] = $objectToSave['Object']['sharing_group_id'];
}
}
$date = new DateTime();
$object['Object']['timestamp'] = $date->getTimestamp();
$forcedSeenOnElements = array();
if (isset($objectToSave['Object']['first_seen'])) {
$forcedSeenOnElements['first_seen'] = $objectToSave['Object']['first_seen'];
}
if (isset($objectToSave['Object']['last_seen'])) {
$forcedSeenOnElements['last_seen'] = $objectToSave['Object']['last_seen'];
}
$object = $this->syncObjectAndAttributeSeen($object, $forcedSeenOnElements, false);
$this->save($object);
$checkFields = array('category', 'value', 'to_ids', 'distribution', 'sharing_group_id', 'comment', 'disable_correlation');
if (!empty($objectToSave['Attribute'])) {
foreach ($objectToSave['Attribute'] as $newKey => $newAttribute) {
foreach ($object['Attribute'] as $origKey => $originalAttribute) {
if (!empty($newAttribute['uuid'])) {
if ($newAttribute['uuid'] == $originalAttribute['uuid']) {
$different = false;
foreach ($checkFields as $f) {
if ($f == 'sharing_group_id' && empty($newAttribute[$f])) {
$newAttribute[$f] = 0;
if (!$onlyAddNewAttribute) {
$checkFields = array('category', 'value', 'to_ids', 'distribution', 'sharing_group_id', 'comment', 'disable_correlation', 'first_seen', 'last_seen');
if (!empty($objectToSave['Attribute'])) {
foreach ($objectToSave['Attribute'] as $newKey => $newAttribute) {
foreach ($object['Attribute'] as $origKey => $originalAttribute) {
if (!empty($newAttribute['uuid'])) {
if ($newAttribute['uuid'] == $originalAttribute['uuid']) {
$different = false;
foreach ($checkFields as $f) {
if ($f == 'sharing_group_id' && empty($newAttribute[$f])) {
$newAttribute[$f] = 0;
}
if ($newAttribute[$f] != $originalAttribute[$f]) {
$different = true;
}
// Set seen of object at attribute level
if (isset($forcedSeenOnElements['first_seen'])) {
$newAttribute['first_seen'] = $forcedSeenOnElements['first_seen'];
if ($newAttribute['object_relation'] == 'first-seen') {
// $newAttribute['value'] = $forcedSeenOnElements['first_seen'];
}
$different = true;
}
if (isset($forcedSeenOnElements['last_seen'])) {
$newAttribute['last_seen'] = $forcedSeenOnElements['last_seen'];
if ($newAttribute['object_relation'] == 'last-seen') {
// $newAttribute['value'] = $forcedSeenOnElements['last_seen'];
}
$different = true;
}
}
if ($newAttribute[$f] != $originalAttribute[$f]) {
$different = true;
if ($different) {
$newAttribute['id'] = $originalAttribute['id'];
$newAttribute['event_id'] = $object['Object']['event_id'];
$newAttribute['object_id'] = $object['Object']['id'];
$newAttribute['timestamp'] = $date->getTimestamp();
$result = $this->Event->Attribute->save(array('Attribute' => $newAttribute), array(
'category',
'value',
'to_ids',
'distribution',
'sharing_group_id',
'comment',
'timestamp',
'object_id',
'event_id',
'disable_correlation',
'first_seen',
'last_seen'
));
}
unset($object['Attribute'][$origKey]);
continue 2;
}
if ($different) {
$newAttribute['id'] = $originalAttribute['id'];
$newAttribute['event_id'] = $object['Object']['event_id'];
$newAttribute['object_id'] = $object['Object']['id'];
$newAttribute['timestamp'] = $date->getTimestamp();
$result = $this->Event->Attribute->save(array('Attribute' => $newAttribute), array(
'category',
'value',
'to_ids',
'distribution',
'sharing_group_id',
'comment',
'timestamp',
'object_id',
'event_id',
'disable_correlation'
));
}
unset($object['Attribute'][$origKey]);
continue 2;
}
}
}
$this->Event->Attribute->create();
$newAttribute['event_id'] = $object['Object']['event_id'];
$newAttribute['object_id'] = $object['Object']['id'];
if (!isset($newAttribute['timestamp'])) {
$newAttribute['distribution'] = Configure::read('MISP.default_attribute_distribution');
if ($newAttribute['distribution'] == 'event') {
$newAttribute['distribution'] = 5;
$this->Event->Attribute->create();
$newAttribute['event_id'] = $object['Object']['event_id'];
$newAttribute['object_id'] = $object['Object']['id'];
// Set seen of object at attribute level
if (isset($forcedSeenOnElements['first_seen'])) {
$newAttribute['first_seen'] = $forcedSeenOnElements['first_seen'];
if ($newAttribute['object_relation'] == 'first-seen') {
$newAttribute['value'] = $forcedSeenOnElements['first_seen'];
}
}
if (isset($forcedSeenOnElements['last_seen'])) {
$newAttribute['last_seen'] = $forcedSeenOnElements['last_seen'];
if ($newAttribute['object_relation'] == 'last-seen') {
$newAttribute['value'] = $forcedSeenOnElements['last_seen'];
}
}
if (!isset($newAttribute['timestamp'])) {
$newAttribute['distribution'] = Configure::read('MISP.default_attribute_distribution');
if ($newAttribute['distribution'] == 'event') {
$newAttribute['distribution'] = 5;
}
}
$this->Event->Attribute->save($newAttribute);
$attributeArrays['add'][] = $newAttribute;
unset($objectToSave['Attribute'][$newKey]);
}
foreach ($object['Attribute'] as $origKey => $originalAttribute) {
$originalAttribute['deleted'] = 1;
$this->Event->Attribute->save($originalAttribute);
}
$this->Event->Attribute->save($newAttribute);
$attributeArrays['add'][] = $newAttribute;
unset($objectToSave['Attribute'][$newKey]);
}
foreach ($object['Attribute'] as $origKey => $originalAttribute) {
$originalAttribute['deleted'] = 1;
$this->Event->Attribute->save($originalAttribute);
} else { // we only add the new attribute
$newAttribute = $objectToSave['Attribute'][0];
$this->Event->Attribute->create();
$newAttribute['event_id'] = $object['Object']['event_id'];
$newAttribute['object_id'] = $object['Object']['id'];
// Set seen of object at attribute level
if (
(!array_key_exists('first_seen', $newAttribute) || is_null($newAttribute['first_seen'])) &&
(!array_key_exists('first_seen', $object['Object']) && !is_null($object['Object']['first_seen']))
) {
$newAttribute['first_seen'] = $object['Object']['first_seen'];
}
if (
(!array_key_exists('last_seen', $newAttribute) || is_null($newAttribute['last_seen'])) &&
(!array_key_exists('last_seen', $object['Object']) && !is_null($object['Object']['last_seen']))
) {
$newAttribute['last_seen'] = $object['Object']['last_seen'];
$different = true;
}
if (!isset($newAttribute['timestamp'])) {
$newAttribute['distribution'] = Configure::read('MISP.default_attribute_distribution');
if ($newAttribute['distribution'] == 'event') {
$newAttribute['distribution'] = 5;
}
}
$saveAttributeResult = $this->Attribute->saveAttributes(array($newAttribute));
return $saveAttributeResult ? $this->id : $this->validationErrors;
}
return $this->id;
}
@ -597,8 +770,10 @@ class MispObject extends AppModel
$this->Event->unpublishEvent($eventId);
$objectId = $this->id;
$partialFails = array();
foreach ($object['Object']['Attribute'] as $attribute) {
$this->Attribute->captureAttribute($attribute, $eventId, $user, $objectId, $log);
if (!empty($object['Object']['Attribute'])) {
foreach ($object['Object']['Attribute'] as $attribute) {
$this->Attribute->captureAttribute($attribute, $eventId, $user, $objectId, $log);
}
}
return true;
} else {
@ -690,8 +865,6 @@ class MispObject extends AppModel
'change' => 'Validation errors: ' . json_encode($this->validationErrors) . ' Full Object: ' . json_encode($attribute),
));
return $this->validationErrors;
} else {
$this->Event->unpublishEvent($eventId);
}
if (!empty($object['Attribute'])) {
foreach ($object['Attribute'] as $attribute) {
@ -701,14 +874,14 @@ class MispObject extends AppModel
return true;
}
public function updateTimestamp($id)
public function updateTimestamp($id, $timestamp = false)
{
$date = new DateTime();
$object = $this->find('first', array(
'recursive' => -1,
'conditions' => array('Object.id' => $id)
));
$object['Object']['timestamp'] = $date->getTimestamp();
$object['Object']['timestamp'] = $timestamp == false ? $date->getTimestamp() : $timestamp;
$object['Object']['skip_zmq'] = 1;
$object['Object']['skip_kafka'] = 1;
$result = $this->save($object);
@ -837,4 +1010,130 @@ class MispObject extends AppModel
return $saved_object['Object']['id'];
}
public function resolveUpdatedTemplate($template, $object, $update_template_available = false)
{
$toReturn = array(
'updateable_attribute' => false,
'not_updateable_attribute' => false,
'newer_template_version' => false,
'template' => $template
);
if (!empty($template)) {
$newer_template = $this->ObjectTemplate->find('first', array(
'conditions' => array(
'ObjectTemplate.uuid' => $object['Object']['template_uuid'],
'ObjectTemplate.version >' => $object['Object']['template_version'],
),
'recursive' => -1,
'contain' => array(
'ObjectTemplateElement'
),
'order' => array('ObjectTemplate.version DESC')
));
if (!empty($newer_template)) {
$toReturn['newer_template_version'] = $newer_template['ObjectTemplate']['version'];
// ignore IDs for comparison
$cur_template_temp = Hash::remove(Hash::remove($template['ObjectTemplateElement'], '{n}.id'), '{n}.object_template_id');
$newer_template_temp = Hash::remove(Hash::remove($newer_template['ObjectTemplateElement'], '{n}.id'), '{n}.object_template_id');
$template_difference = array();
// check how current template is included in the newer
foreach ($cur_template_temp as $cur_obj_rel) {
$flag_sim = false;
foreach ($newer_template_temp as $newer_obj_rel) {
$tmp = Hash::diff($cur_obj_rel, $newer_obj_rel);
if (count($tmp) == 0) {
$flag_sim = true;
break;
}
}
if (!$flag_sim) {
$template_difference[] = $cur_obj_rel;
}
}
$toReturn['updateable_attribute'] = $object['Attribute'];
$toReturn['not_updateable_attribute'] = array();
} else {
$toReturn['newer_template_version'] = false;
}
if (!empty($template_difference)) { // older template not completely embeded in newer
foreach ($template_difference as $temp_diff_element) {
foreach ($object['Attribute'] as $i => $attribute) {
if (
$attribute['object_relation'] == $temp_diff_element['object_relation']
&& $attribute['type'] == $temp_diff_element['type']
) { // This attribute cannot be merged automatically
$attribute['merge-possible'] = false;
$toReturn['not_updateable_attribute'][] = $attribute;
unset($toReturn['updateable_attribute'][$i]);
}
}
}
}
if ($update_template_available) { // template version bump requested
$toReturn['template'] = $newer_template; // bump the template version
}
}
return $toReturn;
}
public function reviseObject($revised_object, $object) {
$revised_object = json_decode(base64_decode($revised_object), true);
$revised_object_both = array('mergeable' => array(), 'notMergeable' => array());
// Loop through attributes to inject and perform the correct action
// (inject, duplicate, add warnings, ...) when applicable
foreach ($revised_object['Attribute'] as $attribute_to_inject) {
$flag_no_collision = true;
foreach ($object['Attribute'] as $attribute) {
if (
$attribute['object_relation'] == $attribute_to_inject['object_relation']
&& $attribute['type'] == $attribute_to_inject['type']
&& $attribute['value'] !== $attribute_to_inject['value']
) { // Collision on value
$multiple = !empty(Hash::extract($template['ObjectTemplateElement'], sprintf('{n}[object_relation=%s][type=%s][multiple=true]', $attribute['object_relation'], $attribute['type'])));
if ($multiple) { // if multiple is set, check if an entry exists already
$flag_entry_exists = false;
foreach ($object['Attribute'] as $attr) {
if (
$attr['object_relation'] == $attribute_to_inject['object_relation']
&& $attr['type'] == $attribute_to_inject['type']
&& $attr['value'] === $attribute_to_inject['value']
) {
$flag_entry_exists = true;
break;
}
}
if (!$flag_entry_exists) { // entry does no exists, can be duplicated
$attribute_to_inject['is_multiple'] = true;
$revised_object_both['mergeable'][] = $attribute_to_inject;
$object['Attribute'][] = $attribute_to_inject;
}
} else { // Collision on value, multiple not set => propose overwrite
$attribute_to_inject['current_value'] = $attribute['value'];
$attribute_to_inject['merge-possible'] = true; // the user can still swap value
$revised_object_both['notMergeable'][] = $attribute_to_inject;
}
$flag_no_collision = false;
} else if (
$attribute['object_relation'] == $attribute_to_inject['object_relation']
&& $attribute['type'] == $attribute_to_inject['type']
&& $attribute['value'] === $attribute_to_inject['value']
) { // all good, they are basically the same, do nothing
$revised_object_both['mergeable'][] = $attribute_to_inject;
$flag_no_collision = false;
}
}
if ($flag_no_collision) { // no collision, nor equalities => inject it straight away
$revised_object_both['mergeable'][] = $attribute_to_inject;
$object['Attribute'][] = $attribute_to_inject;
}
}
return array(
'object' => $object,
'revised_object_both' => $revised_object_both
);
}
}

13
app/Model/Organisation.php
View File

@ -415,4 +415,17 @@ class Organisation extends AppModel
$event['Org'] = $this->__orgCache[$event['Event']['org_id']];
return $event;
}
public function getOrgIdsFromMeta($metaConditions)
{
$orgIds = $this->find('list', array(
'conditions' => $metaConditions,
'fields' => array('id'),
'recursive' => -1
));
if (empty($orgIds)) {
return array(-1);
}
return array_values($orgIds);
}
}

2
app/Model/Post.php
View File

@ -118,7 +118,7 @@ class Post extends AppModel
$bodyDetail .= "The following message was added: \n";
$bodyDetail .= "\n";
$bodyDetail .= $message . "\n";
$tplColorString = !empty(Configure::read('MISP.email_subject_TLP_string')) ? Configure::read('MISP.email_subject_TLP_string') : "TLP Amber";
$tplColorString = !empty(Configure::read('MISP.email_subject_TLP_string')) ? Configure::read('MISP.email_subject_TLP_string') : "tlp:amber";
$subject = "[" . Configure::read('MISP.org') . " MISP] New post in discussion " . $post['Post']['thread_id'] . " - ".$tplColorString;
foreach ($orgMembers as $recipient) {
$this->User->sendEmail($recipient, $bodyDetail, $body, $subject);

3
app/Model/Role.php
View File

@ -220,6 +220,9 @@ class Role extends AppModel
$this->data['Role']['memory_limit'] = '';
}
}
if (empty($this->data['Role']['rate_limit_count'])) {
$this->data['Role']['rate_limit_count'] = 0;
}
return true;
}

722
app/Model/Server.php
View File

@ -81,6 +81,16 @@ class Server extends AppModel
//'on' => 'create', // Limit validation to 'create' or 'update' operations
),
),
'push_sightings' => array(
'boolean' => array(
'rule' => array('boolean'),
//'message' => 'Your custom message here',
'allowEmpty' => true,
'required' => false,
//'last' => false, // Stop validation after this rule
//'on' => 'create', // Limit validation to 'create' or 'update' operations
),
),
'lastpushedid' => array(
'numeric' => array(
'rule' => array('numeric'),
@ -134,7 +144,9 @@ class Server extends AppModel
'Push' => 'MISP/app/Console/cake Server push [user_id] [server_id]',
'Cache feeds for quick lookups' => 'MISP/app/Console/cake Server cacheFeed [user_id] [feed_id|all|csv|text|misp]',
'Fetch feeds as local data' => 'MISP/app/Console/cake Server fetchFeed [user_id] [feed_id|all|csv|text|misp]',
'Run enrichment' => 'MISP/app/Console/cake Event enrichEvent [user_id] [event_id] [json_encoded_module_list]'
'Run enrichment' => 'MISP/app/Console/cake Event enrichEvent [user_id] [event_id] [json_encoded_module_list]',
'Test' => 'MISP/app/Console/cake Server test [server_id]',
'List' => 'MISP/app/Console/cake Server list'
),
'description' => __('If you would like to automate tasks such as caching feeds or pulling from server instances, you can do it using the following command line tools. Simply execute the given commands via the command line / create cron jobs easily out of them.'),
'header' => __('Automating certain console tasks')
@ -428,7 +440,7 @@ class Server extends AppModel
'email_subject_TLP_string' => array(
'level' => 2,
'description' => __('This is the TLP string for e-mails when email_subject_tag is not found.'),
'value' => 'TLP Amber',
'value' => 'tlp:amber',
'errorMessage' => '',
'test' => 'testForEmpty',
'type' => 'string',
@ -1164,6 +1176,15 @@ class Server extends AppModel
),
'Security' => array(
'branch' => 1,
'disable_form_security' => array(
'level' => 0,
'description' => __('Disabling this setting will remove all form tampering protection. Do not set this setting pretty much ever. You were warned.'),
'value' => false,
'errorMessage' => 'This setting leaves your users open to CSRF attacks. Do not please consider disabling this setting.',
'test' => 'testBoolFalse',
'type' => 'boolean',
'null' => true
),
'salt' => array(
'level' => 0,
'description' => __('The salt used for the hashed passwords. You cannot reset this from the GUI, only manually from the settings.php file. Keep in mind, this will invalidate all passwords in the database.'),
@ -1857,6 +1878,14 @@ class Server extends AppModel
'test' => 'testForNumeric',
'type' => 'numeric'
),
'Sightings_sighting_db_enable' => array(
'level' => 1,
'description' => __('Enable SightingDB integration.'),
'value' => false,
'errorMessage' => '',
'test' => 'testBool',
'type' => 'boolean'
),
'CustomAuth_enable' => array(
'level' => 2,
'description' => __('Enable this functionality if you would like to handle the authentication via an external tool and authenticate with MISP using a custom header.'),
@ -2485,6 +2514,11 @@ class Server extends AppModel
$job->saveField('message', 'Pulling proposals.');
}
$pulledProposals = $eventModel->ShadowAttribute->pullProposals($user, $server);
if ($jobId) {
$job->saveField('progress', 75);
$job->saveField('message', 'Pulling sightings.');
}
$pulledSightings = $eventModel->Sighting->pullSightings($user, $server);
if ($jobId) {
$job->saveField('progress', 100);
$job->saveField('message', 'Pull completed.');
@ -2501,13 +2535,14 @@ class Server extends AppModel
'user_id' => $user['id'],
'title' => 'Pull from ' . $server['Server']['url'] . ' initiated by ' . $email,
'change' => sprintf(
'%s events and %s proposals pulled or updated. %s events failed or didn\'t need an update.',
'%s events, %s proposals and %s sightings pulled or updated. %s events failed or didn\'t need an update.',
count($successes),
$pulledProposals,
$pulledSightings,
count($fails)
)
));
return array($successes, $fails, $pulledProposals);
return array($successes, $fails, $pulledProposals, $pulledSightings);
}
public function filterRuleToParameter($filter_rules)
@ -2517,29 +2552,35 @@ class Server extends AppModel
return $final;
}
$filter_rules = json_decode($filter_rules, true);
$url_params = null;
foreach ($filter_rules as $field => $rules) {
$temp = array();
foreach ($rules as $operator => $elements) {
foreach ($elements as $k => $element) {
if ($operator === 'NOT') {
$element = '!' . $element;
}
if (!empty($element)) {
$temp[] = $element;
if ($field === 'url_params') {
$url_params = json_decode($rules, true);
} else {
foreach ($rules as $operator => $elements) {
foreach ($elements as $k => $element) {
if ($operator === 'NOT') {
$element = '!' . $element;
}
if (!empty($element)) {
$temp[] = $element;
}
}
}
}
if (!empty($temp)) {
$temp = implode('|', $temp);
$final[substr($field, 0, strlen($field) -1)] = $temp;
if (!empty($temp)) {
$temp = implode('|', $temp);
$final[substr($field, 0, strlen($field) -1)] = $temp;
}
}
}
$final = array_merge_recursive($final, $url_params);
return $final;
}
// Get an array of event_ids that are present on the remote server
public function getEventIdsFromServer($server, $all = false, $HttpSocket=null, $force_uuid=false, $ignoreFilterRules = false)
public function getEventIdsFromServer($server, $all = false, $HttpSocket=null, $force_uuid=false, $ignoreFilterRules = false, $scope = 'events')
{
$url = $server['Server']['url'];
if ($ignoreFilterRules) {
@ -2566,8 +2607,21 @@ class Server extends AppModel
$eventIds = array();
if ($all) {
if (!empty($eventArray)) {
foreach ($eventArray as $event) {
$eventIds[] = $event['uuid'];
if ($scope === 'sightings') {
foreach ($eventArray as $event) {
$localEvent = $this->Event->find('first', array(
'recursive' => -1,
'fields' => array('Event.uuid', 'Event.sighting_timestamp'),
'conditions' => array('Event.uuid' => $event['uuid'])
));
if (!empty($localEvent) && $localEvent['Event']['sighting_timestamp'] > $event['sighting_timestamp']) {
$eventIds[] = $event['uuid'];
}
}
} else {
foreach ($eventArray as $event) {
$eventIds[] = $event['uuid'];
}
}
}
} else {
@ -2607,7 +2661,7 @@ class Server extends AppModel
}
}
}
$this->Event->removeOlder($eventArray);
$this->Event->removeOlder($eventArray, $scope);
if (!empty($eventArray)) {
foreach ($eventArray as $event) {
if ($force_uuid) {
@ -2640,7 +2694,7 @@ class Server extends AppModel
$this->read(null, $id);
$url = $this->data['Server']['url'];
$push = $this->checkVersionCompatibility($id, $user);
if (isset($push['canPush']) && !$push['canPush']) {
if (is_array($push) && !$push['canPush'] && !$push['canSight']) {
$push = 'Remote instance is outdated or no permission to push.';
}
if (!is_array($push)) {
@ -2665,113 +2719,124 @@ class Server extends AppModel
}
return $push;
}
if ("full" == $technique) {
$eventid_conditions_key = 'Event.id >';
$eventid_conditions_value = 0;
} elseif ("incremental" == $technique) {
$eventid_conditions_key = 'Event.id >';
$eventid_conditions_value = $this->data['Server']['lastpushedid'];
} elseif (intval($technique) !== 0) {
$eventid_conditions_key = 'Event.id';
$eventid_conditions_value = intval($technique);
} else {
throw new InvalidArgumentException("Technique parameter must be 'full', 'incremental' or event ID.");
}
$sgs = $this->Event->SharingGroup->find('all', array(
'recursive' => -1,
'contain' => array('Organisation', 'SharingGroupOrg' => array('Organisation'), 'SharingGroupServer')
));
$sgIds = array();
foreach ($sgs as $k => $sg) {
if ($this->Event->SharingGroup->checkIfServerInSG($sg, $this->data)) {
$sgIds[] = $sg['SharingGroup']['id'];
// sync events if user is capable
if ($push['canPush']) {
if ("full" == $technique) {
$eventid_conditions_key = 'Event.id >';
$eventid_conditions_value = 0;
} elseif ("incremental" == $technique) {
$eventid_conditions_key = 'Event.id >';
$eventid_conditions_value = $this->data['Server']['lastpushedid'];
} elseif (intval($technique) !== 0) {
$eventid_conditions_key = 'Event.id';
$eventid_conditions_value = intval($technique);
} else {
throw new InvalidArgumentException("Technique parameter must be 'full', 'incremental' or event ID.");
}
}
if (empty($sgIds)) {
$sgIds = array(-1);
}
$findParams = array(
'conditions' => array(
$eventid_conditions_key => $eventid_conditions_value,
'Event.published' => 1,
'Event.attribute_count >' => 0,
'OR' => array(
array(
'AND' => array(
array('Event.distribution >' => 0),
array('Event.distribution <' => 4),
),
),
array(
'AND' => array(
'Event.distribution' => 4,
'Event.sharing_group_id' => $sgIds
$sgs = $this->Event->SharingGroup->find('all', array(
'recursive' => -1,
'contain' => array('Organisation', 'SharingGroupOrg' => array('Organisation'), 'SharingGroupServer')
));
$sgIds = array();
foreach ($sgs as $k => $sg) {
if ($this->Event->SharingGroup->checkIfServerInSG($sg, $this->data)) {
$sgIds[] = $sg['SharingGroup']['id'];
}
}
if (empty($sgIds)) {
$sgIds = array(-1);
}
$findParams = array(
'conditions' => array(
$eventid_conditions_key => $eventid_conditions_value,
'Event.published' => 1,
'Event.attribute_count >' => 0,
'OR' => array(
array(
'AND' => array(
array('Event.distribution >' => 0),
array('Event.distribution <' => 4),
),
),
array(
'AND' => array(
'Event.distribution' => 4,
'Event.sharing_group_id' => $sgIds
),
)
)
)
), // array of conditions
'recursive' => -1, //int
'contain' => array('EventTag' => array('fields' => array('EventTag.tag_id'))),
'fields' => array('Event.id', 'Event.timestamp', 'Event.uuid', 'Event.orgc_id'), // array of field names
);
$eventIds = $this->Event->find('all', $findParams);
$eventUUIDsFiltered = $this->getEventIdsForPush($id, $HttpSocket, $eventIds, $user);
if ($eventUUIDsFiltered === false || empty($eventUUIDsFiltered)) {
$pushFailed = true;
}
if (!empty($eventUUIDsFiltered)) {
$eventCount = count($eventUUIDsFiltered);
// now process the $eventIds to pull each of the events sequentially
), // array of conditions
'recursive' => -1, //int
'contain' => array('EventTag' => array('fields' => array('EventTag.tag_id'))),
'fields' => array('Event.id', 'Event.timestamp', 'Event.sighting_timestamp', 'Event.uuid', 'Event.orgc_id'), // array of field names
);
$eventIds = $this->Event->find('all', $findParams);
$eventUUIDsFiltered = $this->getEventIdsForPush($id, $HttpSocket, $eventIds, $user);
if ($eventUUIDsFiltered === false || empty($eventUUIDsFiltered)) {
$pushFailed = true;
}
if (!empty($eventUUIDsFiltered)) {
$successes = array();
$fails = array();
$lowestfailedid = null;
foreach ($eventUUIDsFiltered as $k => $eventUuid) {
$params = array();
if (!empty($this->data['Server']['push_rules'])) {
$push_rules = json_decode($this->data['Server']['push_rules'], true);
if (!empty($push_rules['tags']['NOT'])) {
$params['blockedAttributeTags'] = $push_rules['tags']['NOT'];
$eventCount = count($eventUUIDsFiltered);
// now process the $eventIds to push each of the events sequentially
if (!empty($eventUUIDsFiltered)) {
$successes = array();
$fails = array();
$lowestfailedid = null;
foreach ($eventUUIDsFiltered as $k => $eventUuid) {
$params = array();
if (!empty($this->data['Server']['push_rules'])) {
$push_rules = json_decode($this->data['Server']['push_rules'], true);
if (!empty($push_rules['tags']['NOT'])) {
$params['blockedAttributeTags'] = $push_rules['tags']['NOT'];
}
}
$params = array_merge($params, array(
'event_uuid' => $eventUuid,
'includeAttachments' => true,
'includeAllTags' => true,
'deleted' => array(0,1),
'excludeGalaxy' => 1
));
$event = $this->Event->fetchEvent($user, $params);
$event = $event[0];
$event['Event']['locked'] = 1;
$result = $this->Event->uploadEventToServer($event, $this->data, $HttpSocket);
if ('Success' === $result) {
$successes[] = $event['Event']['id'];
} else {
$fails[$event['Event']['id']] = $result;
}
if ($jobId && $k%10 == 0) {
$job->saveField('progress', 100 * $k / $eventCount);
}
}
$params = array_merge($params, array(
'event_uuid' => $eventUuid,
'includeAttachments' => true,
'includeAllTags' => true,
'deleted' => array(0,1),
'excludeGalaxy' => 1
));
$event = $this->Event->fetchEvent($user, $params);
$event = $event[0];
$event['Event']['locked'] = 1;
$result = $this->Event->uploadEventToServer($event, $this->data, $HttpSocket);
if ('Success' === $result) {
$successes[] = $event['Event']['id'];
if (count($fails) > 0) {
// there are fails, take the lowest fail
$lastpushedid = min(array_keys($fails));
} else {
$fails[$event['Event']['id']] = $result;
}
if ($jobId && $k%10 == 0) {
$job->saveField('progress', 100 * $k / $eventCount);
// no fails, take the highest success
$lastpushedid = max($successes);
}
// increment lastid based on the highest ID seen
// Save the entire Server data instead of just a single field, so that the logger can be fed with the extra fields.
$this->data['Server']['lastpushedid'] = $lastpushedid;
$this->save($this->data);
}
if (count($fails) > 0) {
// there are fails, take the lowest fail
$lastpushedid = min(array_keys($fails));
} else {
// no fails, take the highest success
$lastpushedid = max($successes);
}
// increment lastid based on the highest ID seen
// Save the entire Server data instead of just a single field, so that the logger can be fed with the extra fields.
$this->data['Server']['lastpushedid'] = $lastpushedid;
$this->save($this->data);
}
$this->syncProposals($HttpSocket, $this->data, null, null, $this->Event);
}
$this->syncProposals($HttpSocket, $this->data, null, null, $this->Event);
if ($push['canPush'] || $push['canSight']) {
$sightingSuccesses = $this->syncSightings($HttpSocket, $this->data, $user, $this->Event);
} else {
$sightingSuccesses = array();
}
if (!isset($successes)) {
$successes = array();
$successes = $sightingSuccesses;
} else {
$successes = array_merge($successes, $sightingSuccesses);
}
if (!isset($fails)) {
$fails = array();
@ -2824,6 +2889,33 @@ class Server extends AppModel
return $uuidList;
}
public function syncSightings($HttpSocket, $server, $user, $eventModel)
{
$successes = array();
if (!$server['Server']['push_sightings']) {
return $successes;
}
$this->Sighting = ClassRegistry::init('Sighting');
$HttpSocket = $this->setupHttpSocket($server, $HttpSocket);
$eventIds = $this->getEventIdsFromServer($server, true, $HttpSocket, false, true, 'sightings');
// now process the $eventIds to push each of the events sequentially
if (!empty($eventIds)) {
// check each event and push sightings when needed
foreach ($eventIds as $k => $eventId) {
$event = $eventModel->fetchEvent($user, $options = array('event_uuid' => $eventId, 'metadata' => true));
if (!empty($event)) {
$event = $event[0];
$event['Sighting'] = $this->Sighting->attachToEvent($event, $user);
$result = $eventModel->uploadEventToServer($event, $server, $HttpSocket, 'sightings');
if ($result === 'Success') {
$successes[] = 'Sightings for event ' . $event['Event']['id'];
}
}
}
}
return $successes;
}
public function syncProposals($HttpSocket, $server, $sa_id = null, $event_id = null, $eventModel)
{
$saModel = ClassRegistry::init('ShadowAttribute');
@ -4104,6 +4196,7 @@ class Server extends AppModel
}
$remoteVersion = json_decode($response->body, true);
$canPush = isset($remoteVersion['perm_sync']) ? $remoteVersion['perm_sync'] : false;
$canSight = isset($remoteVersion['perm_sighting']) ? $remoteVersion['perm_sighting'] : false;
$remoteVersion = explode('.', $remoteVersion['version']);
if (!isset($remoteVersion[0])) {
$this->Log = ClassRegistry::init('Log');
@ -4165,7 +4258,7 @@ class Server extends AppModel
'title' => ucfirst($issueLevel) . ': ' . $response,
));
}
return array('success' => $success, 'response' => $response, 'canPush' => $canPush, 'version' => $remoteVersion);
return array('success' => $success, 'response' => $response, 'canPush' => $canPush, 'canSight' => $canSight, 'version' => $remoteVersion);
}
public function isJson($string)
@ -4198,7 +4291,7 @@ class Server extends AppModel
$dataSource = $this->getDataSource()->config['datasource'];
if ($dataSource == 'Database/Mysql') {
$sql = sprintf(
'select table_name, sum((data_length+index_length)/1024/1024) AS used, sum(data_free)/1024/1024 reclaimable from information_schema.tables where table_schema = %s group by table_name;',
'select TABLE_NAME, sum((DATA_LENGTH+INDEX_LENGTH)/1024/1024) AS used, sum(DATA_FREE)/1024/1024 AS reclaimable from information_schema.tables where table_schema = %s group by TABLE_NAME;',
"'" . $this->getDataSource()->config['database'] . "'"
);
$sqlResult = $this->query($sql);
@ -4207,14 +4300,14 @@ class Server extends AppModel
foreach ($temp[0] as $k => $v) {
$temp[0][$k] = round($v, 2) . 'MB';
}
$temp[0]['table'] = $temp['tables']['table_name'];
$temp[0]['table'] = $temp['tables']['TABLE_NAME'];
$result[] = $temp[0];
}
return $result;
}
else if ($dataSource == 'Database/Postgres') {
$sql = sprintf(
'select table_name as table, pg_total_relation_size(%s||%s||table_name) as used from information_schema.tables where table_schema = %s group by table_name;',
'select TABLE_NAME as table, pg_total_relation_size(%s||%s||TABLE_NAME) as used from information_schema.tables where table_schema = %s group by TABLE_NAME;',
"'" . $this->getDataSource()->config['database'] . "'",
"'.'",
"'" . $this->getDataSource()->config['database'] . "'"
@ -4253,6 +4346,341 @@ class Server extends AppModel
return $output;
}
public function dbSchemaDiagnostic()
{
$actualDbVersion = $this->AdminSetting->find('first', array(
'conditions' => array('setting' => 'db_version')
))['AdminSetting']['value'];
$dataSource = $this->getDataSource()->config['datasource'];
$schemaDiagnostic = array(
'dataSource' => $dataSource,
'actual_db_version' => $actualDbVersion,
'checked_table_column' => array(),
'diagnostic' => array(),
'diagnostic_index' => array(),
'expected_db_version' => '?',
'error' => '',
'update_locked' => $this->isUpdateLocked(),
'remaining_lock_time' => $this->getLockRemainingTime(),
'update_fail_number_reached' => $this->UpdateFailNumberReached(),
'indexes' => array()
);
if ($dataSource == 'Database/Mysql') {
$dbActualSchema = $this->getActualDBSchema();
$dbExpectedSchema = $this->getExpectedDBSchema();
if ($dbExpectedSchema !== false) {
$db_schema_comparison = $this->compareDBSchema($dbActualSchema['schema'], $dbExpectedSchema['schema']);
$db_indexes_comparison = $this->compareDBIndexes($dbActualSchema['indexes'], $dbExpectedSchema['indexes']);
$schemaDiagnostic['checked_table_column'] = $dbActualSchema['column'];
$schemaDiagnostic['diagnostic'] = $db_schema_comparison;
$schemaDiagnostic['diagnostic_index'] = $db_indexes_comparison;
$schemaDiagnostic['expected_db_version'] = $dbExpectedSchema['db_version'];
foreach($dbActualSchema['schema'] as $tableName => $tableMetas) {
foreach($tableMetas as $tableMeta) {
$schemaDiagnostic['columnPerTable'][$tableName][] = $tableMeta['column_name'];
}
}
$schemaDiagnostic['indexes'] = $dbActualSchema['indexes'];
} else {
$schemaDiagnostic['error'] = sprintf('Diagnostic not available as the expected schema file could not be loaded');
}
} else {
$schemaDiagnostic['error'] = sprintf('Diagnostic not available for DataSource `%s`', $dataSource);
}
if (!empty($schemaDiagnostic['diagnostic'])) {
foreach ($schemaDiagnostic['diagnostic'] as $table => &$fields) {
foreach ($fields as &$field) {
$field = $this->__attachRecoveryQuery($field, $table);
}
}
}
return $schemaDiagnostic;
}
/*
* Work in progress, still needs DEFAULT in the schema for it to work correctly
* Currently only works for missing_column and column_different
* Only currently supported field types are: int, tinyint, varchar, text
*/
private function __attachRecoveryQuery($field, $table)
{
if (isset($field['error_type'])) {
$length = false;
if (in_array($field['error_type'], array('missing_column', 'column_different'))) {
if ($field['expected']['data_type'] === 'int') {
$length = 11;
} elseif ($field['expected']['data_type'] === 'tinyint') {
$length = 1;
} elseif ($field['expected']['data_type'] === 'varchar') {
$length = $field['expected']['character_maximum_length'];
} elseif ($field['expected']['data_type'] === 'text') {
$length = null;
}
}
if ($length !== false) {
switch($field['error_type']) {
case 'missing_column':
$field['sql'] = sprintf(
'ALTER TABLE `%s` ADD COLUMN `%s` %s%s %s %s %s;',
$table,
$field['column_name'],
$field['expected']['data_type'],
$length !== null ? sprintf('(%d)', $length) : '',
isset($field['expected']['column_default']) ? $field['expected']['column_default'] . '"' : '',
$field['expected']['is_nullable'] === 'NO' ? 'NOT NULL' : 'NULL',
empty($field['expected']['collation_name']) ? '' : 'COLLATE ' . $field['expected']['collation_name']
);
break;
case 'column_different':
$field['sql'] = sprintf(
'ALTER TABLE `%s` MODIFY COLUMN `%s` %s%s %s %s %s;',
$table,
$field['column_name'],
$field['expected']['data_type'],
$length !== null ? sprintf('(%d)', $length) : '',
isset($field['expected']['column_default']) ? 'DEFAULT "' . $field['expected']['column_default'] . '"' : '',
$field['expected']['is_nullable'] === 'NO' ? 'NOT NULL' : 'NULL',
empty($field['expected']['collation_name']) ? '' : 'COLLATE ' . $field['expected']['collation_name']
);
break;
}
} elseif($field['error_type'] == 'missing_table') {
$allFields = array();
foreach ($field['expected_table'] as $expectedField) {
$length = false;
if ($expectedField['data_type'] === 'int') {
$length = 11;
} elseif ($expectedField['data_type'] === 'tinyint') {
$length = 1;
} elseif ($expectedField['data_type'] === 'varchar') {
$length = $expectedField['character_maximum_length'];
} elseif ($expectedField['data_type'] === 'text') {
$length = null;
}
$fieldSql = sprintf('`%s` %s%s %s %s %s',
$expectedField['column_name'],
$expectedField['data_type'],
$length !== null ? sprintf('(%d)', $length) : '',
isset($expectedField['column_default']) ? 'DEFAULT "' . $expectedField['column_default'] . '"' : '',
$expectedField['is_nullable'] === 'NO' ? 'NOT NULL' : 'NULL',
empty($expectedField['collation_name']) ? '' : 'COLLATE ' . $expectedField['collation_name']
);
$allFields[] = $fieldSql;
}
$field['sql'] = __("% The command below is a suggestion and might be incorrect. Please ask if you are not sure what you are doing.") . "</br></br>" . sprintf(
"CREATE TABLE IF NOT EXISTS `%s` ( %s ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;",
$table,
implode(', ', $allFields)
);
}
}
return $field;
}
public function getExpectedDBSchema()
{
App::uses('Folder', 'Utility');
$file = new File(ROOT . DS . 'db_schema.json', true);
$dbExpectedSchema = json_decode($file->read(), true);
$file->close();
if (!is_null($dbExpectedSchema)) {
return $dbExpectedSchema;
} else {
return false;
}
}
// TODO: Use CakePHP 3.X's Schema System
/*
$db = ConnectionManager::get('default');
// Create a schema collection.
$collection = $db->schemaCollection();
// Get the table names
$tables = $collection->listTables();
// Get a single table (instance of Schema\TableSchema)
$tableSchema = $collection->describe('posts');
*/
public function getActualDBSchema(
$tableColumnNames = array(
'column_name',
'is_nullable',
'data_type',
'character_maximum_length',
'numeric_precision',
// 'datetime_precision', -- Only available on MySQL 5.6+
'collation_name',
'column_default'
)
){
$dbActualSchema = array();
$dbActualIndexes = array();
$dataSource = $this->getDataSource()->config['datasource'];
if ($dataSource == 'Database/Mysql') {
$sqlGetTable = sprintf('SELECT TABLE_NAME FROM information_schema.tables WHERE table_schema = %s;', "'" . $this->getDataSource()->config['database'] . "'");
$sqlResult = $this->query($sqlGetTable);
$tables = HASH::extract($sqlResult, '{n}.tables.TABLE_NAME');
foreach ($tables as $table) {
$sqlSchema = sprintf(
"SELECT %s
FROM information_schema.columns
WHERE table_schema = '%s' AND TABLE_NAME = '%s'", implode(',', $tableColumnNames), $this->getDataSource()->config['database'], $table);
$sqlResult = $this->query($sqlSchema);
foreach ($sqlResult as $column_schema) {
$dbActualSchema[$table][] = $column_schema['columns'];
}
$dbActualIndexes[$table] = $this->getDatabaseIndexes($this->getDataSource()->config['database'], $table);
}
}
else if ($dataSource == 'Database/Postgres') {
return array('Database/Postgres' => array('description' => __('Can\'t check database schema for Postgres database type')));
}
return array('schema' => $dbActualSchema, 'column' => $tableColumnNames, 'indexes' => $dbActualIndexes);
}
public function compareDBSchema($dbActualSchema, $dbExpectedSchema)
{
// Column that should be ignored while performing the comparison
$whiteListFields = array(
'users' => array('external_auth_required', 'external_auth_key'),
);
$nonCriticalColumnElements = array('is_nullable', 'collation_name');
$dbDiff = array();
// perform schema comparison for tables
foreach($dbExpectedSchema as $tableName => $columns) {
if (!array_key_exists($tableName, $dbActualSchema)) {
$dbDiff[$tableName][] = array(
'description' => sprintf(__('Table `%s` does not exist'), $tableName),
'error_type' => 'missing_table',
'expected_table' => $columns,
'column_name' => $tableName,
'is_critical' => true
);
} else {
// perform schema comparison for table's columns
$expectedColumnKeys = array();
$keyedExpectedColumn = array();
foreach($columns as $column) {
$expectedColumnKeys[] = $column['column_name'];
$keyedExpectedColumn[$column['column_name']] = $column;
}
$existingColumnKeys = array();
$keyedActualColumn = array();
foreach($dbActualSchema[$tableName] as $column) {
$existingColumnKeys[] = $column['column_name'];
$keyedActualColumn[$column['column_name']] = $column;
}
$additionalKeysInActualSchema = array_diff($existingColumnKeys, $expectedColumnKeys);
foreach($additionalKeysInActualSchema as $additionalKeys) {
if (isset($whiteListFields[$tableName]) && in_array($additionalKeys, $whiteListFields[$tableName])) {
continue; // column is whitelisted
}
$dbDiff[$tableName][] = array(
'description' => sprintf(__('Column `%s` exists but should not'), $additionalKeys),
'error_type' => 'additional_column',
'column_name' => $additionalKeys,
'is_critical' => false
);
}
foreach ($keyedExpectedColumn as $columnName => $column) {
if (isset($whiteListFields[$tableName]) && in_array($columnName, $whiteListFields[$tableName])) {
continue; // column is whitelisted
}
if (isset($keyedActualColumn[$columnName])) {
$colDiff = array_diff_assoc($column, $keyedActualColumn[$columnName]);
if (count($colDiff) > 0) {
$colElementDiffs = array_keys(array_diff_assoc($column, $keyedActualColumn[$columnName]));
$isCritical = false;
foreach($colElementDiffs as $colElementDiff) {
if(!in_array($colElementDiff, $nonCriticalColumnElements)) {
if ($colElementDiff == 'column_default') {
$expectedValue = $column['column_default'];
$actualValue = $keyedActualColumn[$columnName]['column_default'];
if (preg_match(sprintf('@(\'|")+%s(\1)+@', $expectedValue), $actualValue) || (empty($expectedValue) && $actualValue === 'NULL')) { // some version of mysql quote the default value
continue;
} else {
$isCritical = true;
break;
}
} else {
$isCritical = true;
break;
}
}
}
$dbDiff[$tableName][] = array(
'description' => sprintf(__('Column `%s` is different'), $columnName),
'column_name' => $column['column_name'],
'error_type' => 'column_different',
'actual' => $keyedActualColumn[$columnName],
'expected' => $column,
'is_critical' => $isCritical
);
}
} else {
$dbDiff[$tableName][] = array(
'description' => sprintf(__('Column `%s` does not exist but should'), $columnName),
'column_name' => $columnName,
'error_type' => 'missing_column',
'actual' => array(),
'expected' => $column,
'is_critical' => true
);
}
}
}
}
foreach(array_diff(array_keys($dbActualSchema), array_keys($dbExpectedSchema)) as $additionalTable) {
$dbDiff[$additionalTable][] = array(
'description' => sprintf(__('Table `%s` is an additional table'), $additionalTable),
'column_name' => $additionalTable,
'error_type' => 'additional_table',
'is_critical' => false
);
}
return $dbDiff;
}
public function compareDBIndexes($actualIndex, $expectedIndex)
{
$indexDiff = array();
foreach($expectedIndex as $tableName => $indexes) {
if (!array_key_exists($tableName, $actualIndex)) {
// If table does not exists, it is covered by the schema diagnostic
} else {
$tableIndexDiff = array_diff($indexes, $actualIndex[$tableName]); // check for missing indexes
if (count($tableIndexDiff) > 0) {
foreach($tableIndexDiff as $columnDiff) {
$indexDiff[$tableName][$columnDiff] = sprintf(__('Column `%s` should be indexed'), $columnDiff);
}
}
$tableIndexDiff = array_diff($actualIndex[$tableName], $indexes); // check for additional indexes
if (count($tableIndexDiff) > 0) {
foreach($tableIndexDiff as $columnDiff) {
$indexDiff[$tableName][$columnDiff] = sprintf(__('Column `%s` is indexed but should not'), $columnDiff);
}
}
}
}
return $indexDiff;
}
public function getDatabaseIndexes($database, $table)
{
$sqlTableIndex = sprintf(
"SELECT DISTINCT TABLE_NAME, COLUMN_NAME FROM information_schema.statistics WHERE TABLE_SCHEMA = '%s' AND TABLE_NAME = '%s';",
$database,
$table
);
$sqlTableIndexResult = $this->query($sqlTableIndex);
$tableIndex = Hash::extract($sqlTableIndexResult, '{n}.statistics.COLUMN_NAME');
return $tableIndex;
}
public function writeableDirsDiagnostics(&$diagnostic_errors)
{
App::uses('File', 'Utility');
@ -4336,7 +4764,7 @@ class Server extends AppModel
public function stixDiagnostics(&$diagnostic_errors, &$stixVersion, &$cyboxVersion, &$mixboxVersion, &$maecVersion, &$stix2Version, &$pymispVersion)
{
$result = array();
$expected = array('stix' => '>1.2.0.6', 'cybox' => '>2.1.0.18.dev0', 'mixbox' => '1.0.3', 'maec' => '>4.1.0.14', 'stix2' => '1.1.3', 'pymisp' => '>2.4.93');
$expected = array('stix' => '>1.2.0.6', 'cybox' => '>2.1.0.18.dev0', 'mixbox' => '1.0.3', 'maec' => '>4.1.0.14', 'stix2' => '>1.2.0', 'pymisp' => '>2.4.93');
// check if the STIX and Cybox libraries are working using the test script stixtest.py
$scriptResult = shell_exec($this->getPythonVersion() . ' ' . APP . 'files' . DS . 'scripts' . DS . 'stixtest.py');
$scriptResult = json_decode($scriptResult, true);
@ -4384,6 +4812,7 @@ class Server extends AppModel
'binary' => Configure::read('GnuPG.binary') ?: '/usr/bin/gpg'
));
} catch (Exception $e) {
$this->logException("Error during initializing GPG.", $e, LOG_NOTICE);
$gpgStatus = 2;
$continue = false;
}
@ -4391,6 +4820,7 @@ class Server extends AppModel
try {
$key = $gpg->addSignKey(Configure::read('GnuPG.email'), Configure::read('GnuPG.password'));
} catch (Exception $e) {
$this->logException("Error during adding GPG signing key.", $e, LOG_NOTICE);
$gpgStatus = 3;
$continue = false;
}
@ -4400,6 +4830,7 @@ class Server extends AppModel
$gpgStatus = 0;
$signed = $gpg->sign('test', Crypt_GPG::SIGN_MODE_CLEAR);
} catch (Exception $e) {
$this->logException("Error during GPG signing.", $e, LOG_NOTICE);
$gpgStatus = 4;
}
}
@ -4506,6 +4937,7 @@ class Server extends AppModel
'default' => array('ok' => false),
'email' => array('ok' => false),
'prio' => array('ok' => false),
'update' => array('ok' => false),
'scheduler' => array('ok' => false)
);
}
@ -4521,6 +4953,7 @@ class Server extends AppModel
'default' => array('ok' => true),
'email' => array('ok' => true),
'prio' => array('ok' => true),
'update' => array('ok' => true),
'scheduler' => array('ok' => true)
);
$procAccessible = file_exists('/proc');
@ -4539,11 +4972,17 @@ class Server extends AppModel
if (!$alive || !$correct_user) {
$ok = false;
$workerIssueCount++;
$worker_array[$entry]['ok'] = false;
}
$worker_array[$entry]['workers'][] = array('pid' => $pid, 'user' => $worker['user'], 'alive' => $alive, 'correct_user' => $correct_user, 'ok' => $ok);
}
foreach ($worker_array as $k => $queue) {
if (isset($worker_array[$k]['workers'])) {
foreach($worker_array[$k]['workers'] as $worker) {
if ($worker['ok']) {
$worker_array[$k]['ok'] = true; // If at least one worker is up, the queue can be considered working
}
}
}
if ($k != 'scheduler') {
$worker_array[$k]['jobCount'] = CakeResque::getQueueSize($k);
}
@ -5132,7 +5571,7 @@ class Server extends AppModel
public function startWorker($queue)
{
$validTypes = array('default', 'email', 'scheduler', 'cache', 'prio');
$validTypes = array('default', 'email', 'scheduler', 'cache', 'prio', 'update');
if (!in_array($queue, $validTypes)) {
return __('Invalid worker type.');
}
@ -5349,4 +5788,47 @@ class Server extends AppModel
}
return $success;
}
public function getRemoteUser($id)
{
$server = $this->find('first', array(
'conditions' => array('Server.id' => $id),
'recursive' => -1
));
$HttpSocket = $this->setupHttpSocket($server);
$request = $this->setupSyncRequest($server);
$uri = $server['Server']['url'] . '/users/view/me.json';
try {
$response = $HttpSocket->get($uri, false, $request);
} catch (Exception $e) {
$this->Log = ClassRegistry::init('Log');
$this->Log->create();
$message = __('Could not reset fetch remote user account.');
$this->Log->save(array(
'org' => 'SYSTEM',
'model' => 'Server',
'model_id' => $id,
'email' => 'SYSTEM',
'action' => 'error',
'user_id' => 0,
'title' => 'Error: ' . $message,
));
return $message;
}
if ($response->isOk()) {
$user = json_decode($response->body, true);
if (!empty($user['User'])) {
$result = array(
'Email' => $user['User']['email'],
'Role name' => isset($user['Role']['name']) ? $user['Role']['name'] : 'Unknown, outdated instance',
'Sync flag' => isset($user['Role']['perm_sync']) ? ($user['Role']['perm_sync'] ? 1 : 0) : 'Unknown, outdated instance'
);
return $result;
} else {
return __('No user object received in response.');
}
} else {
return $response->code;
}
}
}

35
app/Model/ShadowAttribute.php
View File

@ -133,6 +133,16 @@ class ShadowAttribute extends AppModel
'rule' => array('boolean'),
),
),
'first_seen' => array(
'rule' => array('datetimeOrNull'),
'required' => false,
'message' => array('Invalid ISO 8601 format')
),
'last_seen' => array(
'rule' => array('datetimeOrNull'),
'required' => false,
'message' => array('Invalid ISO 8601 format')
)
);
public function __construct($id = false, $table = null, $ds = null)
@ -177,6 +187,9 @@ class ShadowAttribute extends AppModel
if ($this->data['ShadowAttribute']['deleted']) {
$this->__beforeDeleteCorrelation($this->data['ShadowAttribute']);
}
// convert into utc and micro sec
$this->data = $this->Attribute->ISODatetimeToUTC($this->data, $this->alias);
return true;
}
@ -358,6 +371,14 @@ class ShadowAttribute extends AppModel
return true;
}
public function afterFind($results, $primary = false)
{
foreach ($results as $k => $v) {
$results[$k] = $this->Attribute->UTCToISODatetime($results[$k], $this->alias);
}
return $results;
}
public function validateTypeValue($fields)
{
$category = $this->data['ShadowAttribute']['category'];
@ -474,6 +495,20 @@ class ShadowAttribute extends AppModel
return $fails;
}
// check whether the variable is null or datetime
public function datetimeOrNull($fields)
{
$k = array_keys($fields)[0];
$seen = $fields[$k];
try {
new DateTime($seen);
$returnValue = true;
} catch (Exception $e) {
$returnValue = false;
}
return $returnValue || is_null($seen);
}
public function setDeleted($id)
{
$this->Behaviors->detach('SysLogLogable.SysLogLogable');

24
app/Model/SharingGroup.php
View File

@ -181,14 +181,17 @@ class SharingGroup extends AppModel
'fields' => $fieldsSharingGroup[$permissionTree]['fields'],
'order' => 'SharingGroup.name ASC'
));
foreach ($sgs as &$sg) {
if (!isset($this->__sgoCache[$sg['SharingGroup']['org_id']])) {
$this->__sgoCache[$sg['SharingGroup']['org_id']] = $this->Organisation->find('first', array(
'recursive' => -1,
'fields' => $fieldsOrg,
'conditions' => array('id' => $sg['SharingGroup']['org_id'])
));
if (empty($this->__sgoCache)) {
$temp = $this->Organisation->find('all', array(
'recursive' => -1,
'fields' => $fieldsOrg
));
$this->__sgoCache = array();
foreach ($temp as $o) {
$this->__sgoCache[$o['Organisation']['id']] = $o;
}
}
foreach ($sgs as &$sg) {
if(isset($this->__sgoCache[$sg['SharingGroup']['org_id']]['Organisation'])) {
$sg['Organisation'] = $this->__sgoCache[$sg['SharingGroup']['org_id']]['Organisation'];
} else {
@ -196,13 +199,6 @@ class SharingGroup extends AppModel
}
if (!empty($sg['SharingGroupOrg'])) {
foreach ($sg['SharingGroupOrg'] as &$sgo) {
if (!isset($this->__sgoCache[$sgo['org_id']])) {
$this->__sgoCache[$sgo['org_id']] = $this->Organisation->find('first', array(
'recursive' => -1,
'fields' => $fieldsOrg,
'conditions' => array('id' => $sgo['org_id'])
));
}
if (!empty($this->__sgoCache[$sgo['org_id']]['Organisation'])) {
$sgo['Organisation'] = $this->__sgoCache[$sgo['org_id']]['Organisation'];
}

163
app/Model/Sighting.php
View File

@ -248,10 +248,9 @@ class Sighting extends AppModel
return array();
}
$anonymise = Configure::read('Plugin.Sightings_anonymise');
foreach ($sightings as $k => $sighting) {
if (
$sighting['Sighting']['org_id'] == 0 && !empty($sighting['Organisation']) ||
($sighting['Sighting']['org_id'] == 0 && !empty($sighting['Organisation'])) ||
$anonymise
) {
if ($sighting['Sighting']['org_id'] != $user['org_id']) {
@ -272,7 +271,72 @@ class Sighting extends AppModel
return $sightings;
}
public function saveSightings($id, $values, $timestamp, $user, $type = false, $source = false, $sighting_uuid = false)
/*
* Loop through all attributes of an event, including those in objects
* and pass each value to SightingDB. If there's a hit, append the data
* directly to the attributes
*/
public function attachSightingDB($event, $user)
{
if (!empty(Configure::read('Plugin.Sightings_sighting_db_enable'))) {
$host = empty(Configure::read('Plugin.Sightings_sighting_db_host')) ? 'localhost' : Configure::read('Plugin.Sightings_sighting_db_host');
$port = empty(Configure::read('Plugin.Sightings_sighting_db_port')) ? 9999 : Configure::read('Plugin.Sightings_sighting_db_port');
App::uses('SyncTool', 'Tools');
$syncTool = new SyncTool();
$params = array(
'ssl_verify_peer' => false,
'ssl_verify_peer_name' => false,
'ssl_verify_host' => false
);
$HttpSocket = $syncTool->createHttpSocket($params);
if (!empty($event['Attribute'])) {
$event['Attribute'] = $this->__attachSightingDBToAttribute($event['Attribute'], $user, $host, $port, $HttpSocket);
}
if (!empty($event['Object'])) {
foreach ($event['Object'] as &$object) {
if (!empty($object['Attribute'])) {
$object['Attribute'] = $this->__attachSightingDBToAttribute($object['Attribute'], $user, $host, $port, $HttpSocket);
}
}
}
}
return $event;
}
private function __attachSightingDBToAttribute($attributes, $user, $host, $port, $HttpSocket = false)
{
if (empty($HttpSocket)) {
App::uses('SyncTool', 'Tools');
$syncTool = new SyncTool();
$params = array(
'ssl_allow_self_signed' => true,
'ssl_verify_peer' => false
);
$HttpSocket = $syncTool->createHttpSocket($params);
}
foreach($attributes as &$attribute) {
$response = $HttpSocket->get(
sprintf(
'%s:%s/r/all/%s?val=%s',
$host,
$port,
$attribute['type'],
rtrim(str_replace('/', '_', str_replace('+', '-', base64_encode($attribute['value']))), '=')
)
);
if ($response->code == 200) {
$responseData = json_decode($response->body, true);
if ($responseData !== false) {
if (!isset($responseData['error'])) {
$attribute['SightingDB'] = $responseData;
}
}
}
}
return $attributes;
}
public function saveSightings($id, $values, $timestamp, $user, $type = false, $source = false, $sighting_uuid = false, $publish = false, $saveOnBehalfOf = false)
{
$conditions = array();
if ($id && $id !== 'stix') {
@ -314,7 +378,7 @@ class Sighting extends AppModel
$sighting = array(
'attribute_id' => $attribute['Attribute']['id'],
'event_id' => $attribute['Attribute']['event_id'],
'org_id' => $user['org_id'],
'org_id' => ($saveOnBehalfOf === false) ? $user['org_id'] : $saveOnBehalfOf,
'date_sighting' => $timestamp,
'type' => $type,
'source' => $source
@ -337,6 +401,9 @@ class Sighting extends AppModel
return json_encode($this->validationErrors);
}
$sightingsAdded += $result ? 1 : 0;
if ($publish) {
$this->Event->publishRouter($sighting['event_id'], null, $user, 'sightings');
}
}
if ($sightingsAdded == 0) {
return 'There was nothing to add.';
@ -562,6 +629,16 @@ class Sighting extends AppModel
public function restSearch($user, $returnFormat, $filters)
{
$allowedContext = array('event', 'attribute');
// validate context
if (isset($filters['context']) && !in_array($filters['context'], $allowedContext, true)) {
throw new MethodNotAllowedException(_('Invalid context.'));
}
// ensure that an id is provided if context is set
if (!empty($filters['context']) && !isset($filters['id'])) {
throw new MethodNotAllowedException(_('An id must be provided if the context is set.'));
}
if (!isset($this->validFormats[$returnFormat][1])) {
throw new NotFoundException('Invalid output format.');
}
@ -607,10 +684,12 @@ class Sighting extends AppModel
$conditions['Sighting.source'] = $filters['source'];
}
if ($filters['context'] === 'attribute') {
$conditions['Sighting.attribute_id'] = $filters['id'];
} elseif ($filters['context'] === 'event') {
$conditions['Sighting.event_id'] = $filters['id'];
if (!empty($filters['id'])) {
if ($filters['context'] === 'attribute') {
$conditions['Sighting.attribute_id'] = $filters['id'];
} elseif ($filters['context'] === 'event') {
$conditions['Sighting.event_id'] = $filters['id'];
}
}
// fetch sightings matching the query
@ -645,7 +724,6 @@ class Sighting extends AppModel
$filters['requested_attributes'] = array_merge($filters['requested_attributes'], array('event_uuid', 'event_orgc_id', 'event_org_id', 'event_info', 'event_Orgc_name'));
$additional_event_added = true;
}
if (!empty($sight)) {
array_push($allowedSightings, $sight);
}
@ -691,4 +769,71 @@ class Sighting extends AppModel
fclose($tmpfile);
return $final;
}
// Bulk save sightings
public function bulkSaveSightings($eventId, $sightings, $user, $passAlong = null)
{
if (!is_numeric($eventId)) {
$eventId = $this->Event->field('id', array('uuid' => $eventId));
}
$event = $this->Event->fetchEvent($user, array(
'eventid' => $eventId,
'metadata' => 1,
'flatten' => true
));
if (empty($event)) {
return 'Event not found or not accesible by this user.';
}
$saved = 0;
foreach ($sightings as $s) {
$saveOnBehalfOf = false;
if ($user['Role']['perm_sync']) {
if (isset($s['org_id'])) {
if ($s['org_id'] != 0 && !empty($s['Organisation'])) {
$saveOnBehalfOf = $this->Event->Orgc->captureOrg($s['Organisation'], $user);
} else {
$saveOnBehalfOf = 0;
}
}
}
$result = $this->saveSightings($s['attribute_uuid'], false, $s['date_sighting'], $user, $s['type'], $s['source'], $s['uuid'], false, $saveOnBehalfOf);
if (is_numeric($result)) {
$saved += $result;
}
}
if ($saved > 0) {
$this->Event->publishRouter($eventId, $passAlong, $user, 'sightings');
}
return $saved;
}
public function pullSightings($user, $server)
{
$HttpSocket = $this->setupHttpSocket($server);
$this->Server = ClassRegistry::init('Server');
$eventIds = $this->Server->getEventIdsFromServer($server, false, $HttpSocket, false, false, 'sightings');
$saved = 0;
// now process the $eventIds to pull each of the events sequentially
if (!empty($eventIds)) {
// download each event and save sightings
foreach ($eventIds as $k => $eventId) {
$event = $this->Event->downloadEventFromServer($eventId, $server);
$sightings = array();
if(!empty($event) && !empty($event['Event']['Attribute'])) {
foreach($event['Event']['Attribute'] as $attribute) {
if(!empty($attribute['Sighting'])) {
$sightings = array_merge($sightings, $attribute['Sighting']);
}
}
}
if(!empty($event) && !empty($sightings)) {
$result = $this->bulkSaveSightings($event['Event']['uuid'], $sightings, $user, $server['Server']['id']);
if (is_numeric($result)) {
$saved += $result;
}
}
}
}
return $saved;
}
}

396
app/Model/Sightingdb.php Normal file
View File

@ -0,0 +1,396 @@
<?php
App::uses('AppModel', 'Model');
App::uses('ConnectionManager', 'Model');
App::uses('Sanitize', 'Utility');
class Sightingdb extends AppModel
{
public $errors = array(
1 => 'Invalid SightingDB.',
2 => 'No response from SightingDB.',
3 => 'Invalid or unexpected response from SightingDB.',
4 => 'DNS error - name resolution error.',
5 => 'Could not connect to the SightingDB. Error unspecified.'
);
public $actsAs = array(
'SysLogLogable.SysLogLogable' => array(
'userModel' => 'User',
'userKey' => 'user_id',
'change' => 'full'),
'Trim',
'Containable',
);
public $validate = array(
'name' => array(
'notBlank' => array(
'rule' => array('notBlank'),
'required' => array('create'),
'message' => 'Name not set.'
)
),
'host' => array(
'notBlank' => array(
'rule' => array('notBlank'),
'required' => array('create'),
'message' => 'Host not set.'
)
),
'port' => array(
'numeric' => array(
'rule' => array('numeric'),
'message' => 'Port needs to be numeric.'
)
),
'owner' => array(
'notBlank' => array(
'rule' => array('notBlank'),
'required' => array('create'),
'message' => 'Owner not set.'
)
)
);
public $hasMany = array(
'SightingdbOrg' => array(
'className' => 'SightingdbOrg',
'foreignKey' => 'sightingdb_id',
'dependent' => true
)
);
private $__sightingdbs = false;
private $__connectionStatus = array();
/*
* Load all sightingDBs into a persistent array
* Helps with repeated lookups
*/
private function __loadSightingdbs($user)
{
$this->__sightingdbs = $this->find('all', array(
'recursive' => -1,
'contain' => array('SightingdbOrg'),
'conditions' => array('Sightingdb.enabled' => 1)
));
$this->__sightingdbs = $this->extractOrgIdsFromList($this->__sightingdbs);
foreach ($this->__sightingdbs as $k => $sightingdb) {
if (
empty($user['Role']['perm_site_admin']) &&
!empty($sightingdb['Sightingdb']['org_id']) &&
!in_array($user['org_id'], $sightingdb['Sightingdb']['org_id'])
) {
unset($this->__sightingdbs[$k]);
}
if (empty($this->__connectionStatus[$sightingdb['Sightingdb']['id']])) {
$this->__connectionStatus[$sightingdb['Sightingdb']['id']] = $this->requestStatus($sightingdb);
}
if (!is_array($this->__connectionStatus[$sightingdb['Sightingdb']['id']])) {
unset($this->__sightingdbs[$k]);
}
}
$this->__sightingdbs = array_values($this->__sightingdbs);
}
/*
* Loop through a list of attributes, and pass each value to SightingDB.
* If there's a hit, append the data directly to the attributes
*/
public function attachToAttributes($attributes, $user)
{
if (!empty(Configure::read('Plugin.Sightings_sighting_db_enable'))) {
if ($this->__sightingdbs === false) {
$this->__loadSightingdbs($user);
}
if (!empty($this->__sightingdbs)) {
$values = array();
foreach ($attributes as $attribute) {
$values[$attribute['Attribute']['value']] = array();
}
foreach ($this->__sightingdbs as $sightingdb) {
$values = $this->queryValues($values, $sightingdb);
}
foreach ($attributes as &$attribute) {
if (!empty($values[$attribute['Attribute']['value']])) {
$attribute['Attribute']['Sightingdb'] = array_values($values[$attribute['Attribute']['value']]);
}
}
}
}
return $attributes;
}
/*
* Loop through all attributes of an event, including those in objects
* and pass each value to SightingDB. If there's a hit, append the data
* directly to the attributes
*/
public function attachToEvent($event, $user)
{
if (!empty(Configure::read('Plugin.Sightings_sighting_db_enable'))) {
if ($this->__sightingdbs === false) {
$this->__loadSightingdbs($user);
}
if (!empty($this->__sightingdbs)) {
$values = $this->__collectValues($event);
foreach ($this->__sightingdbs as $sightingdb) {
$values = $this->queryValues($values, $sightingdb);
}
$event = $this->__attachValuesToEvent($event, $values);
}
}
return $event;
}
/*
* Extract all attribute values from an event.
* Also accepts the meta format from after pagination
*/
private function __collectValues($event)
{
$values = array();
if (!empty($event['Attribute'])) {
foreach ($event['Attribute'] as $attribute) {
$values[$attribute['value']] = array();
}
}
if (!empty($event['Object'])) {
foreach ($event['Object'] as $object) {
if (!empty($object['Attribute'])) {
foreach ($object['Attribute'] as $attribute) {
$values[$attribute['value']] = array();
}
}
}
}
if (!empty($event['objects'])) {
foreach ($event['objects'] as $object) {
if ($object['objectType'] === 'attribute') {
$values[$object['value']] = array();
} else if ($object['objectType'] === 'object') {
foreach ($object['Attribute'] as $attribute) {
$values[$attribute['value']] = array();
}
}
}
}
return $values;
}
/*
* Reattach the sightingDB results where applicable to all attriutes in an event
*/
private function __attachValuesToEvent($event, $values)
{
if (!empty($event['Attribute'])) {
foreach ($event['Attribute'] as &$attribute) {
if (!empty($values[$attribute['value']])) {
$attribute['Sightingdb'] = array_values($values[$attribute['value']]);
}
}
}
if (!empty($event['Object'])) {
foreach ($event['Object'] as &$object) {
if (!empty($object['Attribute'])) {
foreach ($object['Attribute'] as &$attribute) {
if (!empty($values[$attribute['value']])) {
$attribute['Sightingdb'] = array_values($values[$attribute['value']]);
}
}
}
}
}
if (!empty($event['objects'])) {
foreach ($event['objects'] as &$object) {
if ($object['objectType'] === 'attribute') {
if (!empty($values[$object['value']])) {
$object['Sightingdb'] = array_values($values[$object['value']]);
}
} else if ($object['objectType'] === 'object') {
foreach ($object['Attribute'] as &$attribute) {
if (!empty($values[$attribute['value']])) {
$attribute['Sightingdb'] = array_values($values[$attribute['value']]);
}
}
}
}
}
return $event;
}
/*
* Query the sightingDB for each value extracted
*/
public function queryValues($values, $sightingdb)
{
$host = $sightingdb['Sightingdb']['host'];
$port = $sightingdb['Sightingdb']['port'];
App::uses('SyncTool', 'Tools');
$syncTool = new SyncTool();
$params = array(
'ssl_verify_peer' => empty($sightingdb['Sightingdb']['ssl_skip_verification']),
'ssl_verify_peer_name' => empty($sightingdb['Sightingdb']['ssl_skip_verification']),
'ssl_verify_host' => empty($sightingdb['Sightingdb']['ssl_skip_verification']),
'skip_proxy' => !empty($sightingdb['Sightingdb']['skip_proxy'])
);
$HttpSocket = $syncTool->createHttpSocket($params);
$payload = array('items' => array());
$namespace = empty($sightingdb['Sightingdb']['namespace']) ? 'all' : $sightingdb['Sightingdb']['namespace'];
$valueLookup = array();
foreach ($values as $k => $value) {
$hashedValue = hash('sha256', $k);
$payload['items'][] = array(
'namespace' => $namespace,
'value' => $hashedValue
);
$valueLookup[$hashedValue] = $k;
}
$request = array(
'header' => array(
'Accept' => 'application/json',
'Content-Type' => 'application/json'
)
);
try {
$response = $HttpSocket->post(
sprintf(
'%s:%s/rb',
$host,
$port
),
json_encode($payload),
$request
);
} catch (Exception $e) {
return $values;
}
if ($response->code == 200) {
$responseData = json_decode($response->body, true);
if ($responseData !== false && empty($responseData['error'])) {
foreach ($responseData['items'] as $k => $item) {
if (empty($item['error'])) {
$values[$valueLookup[$item['value']]][$sightingdb['Sightingdb']['id']] = array(
'first_seen' => $item['first_seen'],
'last_seen' => $item['last_seen'],
'count' => $item['count'],
'sightingdb_id' => $sightingdb['Sightingdb']['id']
);
}
}
}
}
return $values;
}
/*
* Extract the org IDs from the sightingdbOrg objects and attach them to a simple list
*/
public function extractOrgIdsFromList($data)
{
foreach ($data as &$element) {
$element = $this->extractOrgIds($element);
}
return $data;
}
public function extractOrgIds($element)
{
if (isset($element['SightingdbOrg'])) {
$element['Sightingdb']['org_id'] = Hash::extract($element['SightingdbOrg'], '{n}.org_id');
unset($element['SightingdbOrg']);
}
return $element;
}
/*
* Query the SightingDB, returning:
* - implementation
* - version
* - vendor
* - author
* - measured response time
*/
public function requestStatus($sightingdb)
{
if (!is_array($sightingdb)) {
$sightingdb = $this->find('first', array(
'conditions' => array('Sightingdb.id' => $sightingdb),
'recursive' => -1
));
}
if (empty($sightingdb)) {
return __('Invalid SightingDB entry.');
}
App::uses('SyncTool', 'Tools');
$syncTool = new SyncTool();
$params = array(
'ssl_allow_self_signed' => true,
'ssl_verify_peer' => false,
'ssl_verify_peer_name' => false
);
$HttpSocket = $syncTool->createHttpSocket($params);
$start = microtime(true);
try {
$response = $HttpSocket->get(
sprintf(
'%s:%s/i',
$sightingdb['Sightingdb']['host'],
$sightingdb['Sightingdb']['port']
)
);
} catch (Exception $e) {
if (strpos($e->getMessage(), 'php_network_getaddresses') !== false) {
return __('Could not resolve Sightingdb address.');
} else {
return __('Something went wrong. Could not contact the SightingDB server.');
}
}
$response_time = round(1000*(microtime(true) - $start));
if ($response->code == 200) {
$responseData = json_decode($response->body, true);
if (!empty($responseData['implementation'])) {
$result = array();
$fields = array('implementation', 'version', 'vendor', 'author');
foreach ($fields as $field) {
$result[$field] = $responseData[$field];
}
$result['response_time'] = $response_time . 'ms';
return $result;
} else {
return __('The SightingDB returned an invalid response.');
}
} else {
return __('No response from the SightingDB server.');
}
}
/*
* Get a list of all valid sightingDBs for the user
*/
public function getSightingdbList($user)
{
$sightingdbs = $this->find('all', array(
'recursive' => -1,
'contain' => 'SightingdbOrg',
'conditions' => array('Sightingdb.enabled' => 1)
));
if (empty($sightingdbs)) {
return array();
}
$sightingdbs = $this->extractOrgIdsFromList($sightingdbs);
$toReturn = array();
foreach ($sightingdbs as $sightingdb) {
if (
!empty($user['Role']['perm_site_admin']) ||
empty($sightingdb['Sightingdb']['org_id']) ||
in_array($user['org_id'], $sightingdb['Sightingdb']['org_id']
)) {
$toReturn[$sightingdb['Sightingdb']['id']] = $sightingdb;
}
}
return $toReturn;
}
}

57
app/Model/SightingdbOrg.php Normal file
View File

@ -0,0 +1,57 @@
<?php
App::uses('AppModel', 'Model');
App::uses('ConnectionManager', 'Model');
App::uses('Sanitize', 'Utility');
class SightingdbOrg extends AppModel
{
public $belongsTo = array(
'Sightingdb' => array(
'className' => 'Sightingdb',
'foreignKey' => 'sightingdb_id'
),
'Organisation' => array(
'className' => 'Organisation',
'foreignKey' => 'org_id'
)
);
public $actsAs = array(
'SysLogLogable.SysLogLogable' => array(
'userModel' => 'User',
'userKey' => 'user_id',
'change' => 'full'),
'Trim',
'Containable',
);
public function resetOrgs($sightingdbId, $org_ids)
{
$sightingdb = $this->Sightingdb->find('first', array(
'conditions' => array('Sightingdb.id' => $sightingdbId),
'recursive' => -1
));
if (empty($sightingdb)) {
return false;
}
$this->deleteAll(array('SightingdbOrg.sightingdb_id' => $sightingdbId));
if (!empty($org_ids)) {
if (!is_array($org_ids)) {
$org_ids = explode(',', $org_ids);
}
foreach ($org_ids as $org_id) {
debug($org_id);
$this->create();
$this->save(
array(
'SightingdbOrg' => array(
'sightingdb_id' => $sightingdbId,
'org_id' => $org_id
)
)
);
}
}
return true;
}
}

35
app/Model/Tag.php
View File

@ -327,10 +327,16 @@ class Tag extends AppModel
} else {
if (
!$user['Role']['perm_site_admin'] &&
$existingTag['Tag']['org_id'] != 0 &&
$existingTag['Tag']['org_id'] != $user['org_id'] &&
$existingTag['Tag']['user_id'] != 0 &&
$existingTag['Tag']['user_id'] != $user['id']
(
(
$existingTag['Tag']['org_id'] != 0 &&
$existingTag['Tag']['org_id'] != $user['org_id']
) ||
(
$existingTag['Tag']['user_id'] != 0 &&
$existingTag['Tag']['user_id'] != $user['id']
)
)
) {
return false;
}
@ -407,11 +413,28 @@ class Tag extends AppModel
return ($this->saveAll($tags));
}
public function getTagsByName($tag_names, $containTagConnectors = true)
{
$contain = array('EventTag', 'AttributeTag');
$tag_params = array(
'recursive' => -1,
'conditions' => array('name' => $tag_names)
);
if ($containTagConnectors) {
$tag_params['contain'] = $contain;
}
$tags_temp = $this->find('all', $tag_params);
$tags = array();
foreach ($tags_temp as $temp) {
$tags[strtoupper($temp['Tag']['name'])] = $temp;
}
return $tags;
}
public function getTagsForNamespace($namespace, $containTagConnectors = true)
{
$contain = array('EventTag');
$contain[] = 'AttributeTag';
$contain = array('EventTag', 'AttributeTag');
$tag_params = array(
'recursive' => -1,
'conditions' => array('UPPER(name) LIKE' => strtoupper($namespace) . '%'),

140
app/Model/Taxonomy.php
View File

@ -101,7 +101,7 @@ class Taxonomy extends AppModel
}
$this->deleteAll(array('Taxonomy.namespace' => $current['Taxonomy']['namespace']));
}
$taxonomy['Taxonomy'] = array('namespace' => $vocab['namespace'], 'description' => $vocab['description'], 'version' => $vocab['version'], 'enabled' => $enabled);
$taxonomy['Taxonomy'] = array('namespace' => $vocab['namespace'], 'description' => $vocab['description'], 'version' => $vocab['version'], 'exclusive' => !empty($vocab['exclusive']), 'enabled' => $enabled);
$predicateLookup = array();
foreach ($vocab['predicates'] as $k => $predicate) {
$taxonomy['Taxonomy']['TaxonomyPredicate'][$k] = $predicate;
@ -283,7 +283,8 @@ class Taxonomy extends AppModel
if (empty($taxonomy)) {
return false;
}
$tags = $this->Tag->getTagsForNamespace($taxonomy['Taxonomy']['namespace'], false);
$tag_names = Hash::extract($taxonomy, 'entries.{n}.tag');
$tags = $this->Tag->getTagsByName($tag_names, false);
if (isset($taxonomy['entries'])) {
foreach ($taxonomy['entries'] as $key => $temp) {
$taxonomy['entries'][$key]['existing_tag'] = isset($tags[strtoupper($temp['tag'])]) ? $tags[strtoupper($temp['tag'])] : false;
@ -489,27 +490,29 @@ class Taxonomy extends AppModel
return $taxonomies;
}
public function getTaxonomyForTag($tagName, $metaOnly = false)
public function getTaxonomyForTag($tagName, $metaOnly = false, $fullTaxonomy = False)
{
if (preg_match('/^[^:="]+:[^:="]+="[^:="]+"$/i', $tagName)) {
$temp = explode(':', $tagName);
$pieces = array_merge(array($temp[0]), explode('=', $temp[1]));
$pieces[2] = trim($pieces[2], '"');
$contain = array(
'TaxonomyPredicate' => array(
'TaxonomyEntry' => array()
)
);
if (!$fullTaxonomy) {
$contain['TaxonomyPredicate']['conditions'] = array(
'LOWER(TaxonomyPredicate.value)' => strtolower($pieces[1])
);
$contain['TaxonomyPredicate']['TaxonomyEntry']['conditions'] = array(
'LOWER(TaxonomyEntry.value)' => strtolower($pieces[2])
);
}
$taxonomy = $this->find('first', array(
'recursive' => -1,
'conditions' => array('LOWER(Taxonomy.namespace)' => strtolower($pieces[0])),
'contain' => array(
'TaxonomyPredicate' => array(
'conditions' => array(
'LOWER(TaxonomyPredicate.value)' => strtolower($pieces[1])
),
'TaxonomyEntry' => array(
'conditions' => array(
'LOWER(TaxonomyEntry.value)' => strtolower($pieces[2])
)
)
)
)
'contain' => $contain
));
if ($metaOnly && !empty($taxonomy)) {
return array('Taxonomy' => $taxonomy['Taxonomy']);
@ -517,16 +520,16 @@ class Taxonomy extends AppModel
return $taxonomy;
} elseif (preg_match('/^[^:="]+:[^:="]+$/i', $tagName)) {
$pieces = explode(':', $tagName);
$contain = array('TaxonomyPredicate' => array());
if (!$fullTaxonomy) {
$contain['TaxonomyPredicate']['conditions'] = array(
'LOWER(TaxonomyPredicate.value)' => strtolower($pieces[1])
);
}
$taxonomy = $this->find('first', array(
'recursive' => -1,
'conditions' => array('LOWER(Taxonomy.namespace)' => strtolower($pieces[0])),
'contain' => array(
'TaxonomyPredicate' => array(
'conditions' => array(
'LOWER(TaxonomyPredicate.value)' => strtolower($pieces[1])
)
)
)
'contain' => $contain
));
if ($metaOnly && !empty($taxonomy)) {
return array('Taxonomy' => $taxonomy['Taxonomy']);
@ -536,4 +539,97 @@ class Taxonomy extends AppModel
return false;
}
}
// Remove the value for triple component tags or the predicate for double components tags
public function stripLastTagComponent($tagName)
{
$shortenedTag = '';
if (preg_match('/^[^:="]+:[^:="]+="[^:="]+"$/i', $tagName)) {
$shortenedTag = explode('=', $tagName)[0];
} elseif (preg_match('/^[^:="]+:[^:="]+$/i', $tagName)) {
$shortenedTag = explode(':', $tagName)[0];
}
return $shortenedTag;
}
public function checkIfNewTagIsAllowedByTaxonomy($newTagName, $tagNameList=array())
{
$newTagShortened = $this->stripLastTagComponent($newTagName);
$prefixIsFree = true;
foreach ($tagNameList as $tagName) {
$tagShortened = $this->stripLastTagComponent($tagName);
if ($newTagShortened == $tagShortened) {
$prefixIsFree = false;
}
}
if (!$prefixIsFree) {
// at this point, we have a duplicated namespace(-predicate)
$taxonomy = $this->getTaxonomyForTag($newTagName);
if (!empty($taxonomy['Taxonomy']['exclusive'])) {
return false; // only one tag of this taxonomy is allowed
} elseif (!empty($taxonomy['TaxonomyPredicate'][0]['exclusive'])) {
return false; // only one tag belonging to this predicate is allowed
}
}
return true;
}
public function checkIfTagInconsistencies($tagList)
{
$eventTags = array();
$localEventTags = array();
foreach($tagList as $tag) {
if ($tag['local'] == 0) {
$eventTags[] = $tag['Tag']['name'];
} else {
$localEventTags[] = $tag['Tag']['name'];
}
}
$tagConflicts = $this->getTagConflicts($eventTags);
$localTagConflicts = $this->getTagConflicts($localEventTags);
return array(
'global' => $tagConflicts,
'local' => $localTagConflicts
);
}
public function getTagConflicts($tagNameList)
{
$potentiallyConflictingTaxonomy = array();
$conflictingTaxonomy = array();
foreach ($tagNameList as $tagName) {
$tagShortened = $this->stripLastTagComponent($tagName);
if (isset($potentiallyConflictingTaxonomy[$tagShortened])) {
$potentiallyConflictingTaxonomy[$tagShortened]['taxonomy'] = $this->getTaxonomyForTag($tagName);
$potentiallyConflictingTaxonomy[$tagShortened]['count']++;
} else {
$potentiallyConflictingTaxonomy[$tagShortened] = array(
'count' => 1
);
}
$potentiallyConflictingTaxonomy[$tagShortened]['tagNames'][] = $tagName;
}
foreach ($potentiallyConflictingTaxonomy as $potTaxonomy) {
if ($potTaxonomy['count'] > 1) {
$taxonomy = $potTaxonomy['taxonomy'];
if (isset($taxonomy['Taxonomy']['exclusive']) && $taxonomy['Taxonomy']['exclusive']) {
$conflictingTaxonomy[] = array(
'tags' => $potTaxonomy['tagNames'],
'taxonomy' => $taxonomy,
'conflict' => sprintf(__('Taxonomy `%s` is an exclusive Taxonomy'), $taxonomy['Taxonomy']['namespace'])
);
} elseif (isset($taxonomy['TaxonomyPredicate'][0]['exclusive']) && $taxonomy['TaxonomyPredicate'][0]['exclusive']) {
$conflictingTaxonomy[] = array(
'tags' => $potTaxonomy['tagNames'],
'taxonomy' => $taxonomy,
'conflict' => sprintf(
__('Predicate `%s` is exclusive'),
$taxonomy['TaxonomyPredicate'][0]['value']
)
);
}
}
}
return $conflictingTaxonomy;
}
}

83
app/Model/User.php
View File

@ -3,6 +3,9 @@ App::uses('AppModel', 'Model');
App::uses('AuthComponent', 'Controller/Component');
App::uses('RandomTool', 'Tools');
/**
* @property Log $Log
*/
class User extends AppModel
{
public $displayField = 'email';
@ -609,7 +612,19 @@ class User extends AppModel
throw new NotFoundException('Invalid user ID.');
}
$conditions = array('User.id' => $id);
$user = $this->find('first', array('conditions' => $conditions, 'recursive' => -1,'contain' => array('Organisation', 'Role', 'Server')));
$user = $this->find(
'first',
array(
'conditions' => $conditions,
'recursive' => -1,
'contain' => array(
'Organisation',
'Role',
'Server',
'UserSetting'
)
)
);
if (empty($user)) {
return $user;
}
@ -617,6 +632,7 @@ class User extends AppModel
$user['User']['Role'] = $user['Role'];
$user['User']['Organisation'] = $user['Organisation'];
$user['User']['Server'] = $user['Server'];
$user['User']['UserSetting'] = $user['UserSetting'];
unset($user['Organisation'], $user['Role'], $user['Server']);
return $user['User'];
}
@ -1406,20 +1422,12 @@ class User extends AppModel
// query
$this->Log = ClassRegistry::init('Log');
$this->Log->create();
$this->Log->save(array(
'org' => $user['Organisation']['name'],
'model' => $model,
'model_id' => $modelId,
'email' => $user['email'],
'action' => $action,
'title' => $description,
'change' => isset($fieldsResult) ? $fieldsResult : ''));
$result = $this->Log->createLogEntry($user, $action, $model, $modelId, $description, $fieldsResult);
// write to syslogd as well
App::import('Lib', 'SysLog.SysLog');
$syslog = new SysLog();
$syslog->write('notice', $description . ' -- ' . $action . (empty($fieldResult) ? '' : '-- ' . $fieldResult));
$syslog->write('notice', "$description -- $action" . (empty($fieldResult) ? '' : ' -- ' . $result['Log']['change']));
}
/**
@ -1448,4 +1456,57 @@ class User extends AppModel
return new Crypt_GPG($options);
}
public function getOrgActivity($orgId, $params=array())
{
$conditions = array();
$options = array();
foreach($params as $paramName => $value) {
$options['filter'] = $paramName;
$filterParam[$paramName] = $value;
$conditions = $this->Event->set_filter_timestamp($filterParam, $conditions, $options);
}
$conditions['Event.orgc_id'] = $orgId;
$events = $this->Event->find('all', array(
'recursive' => -1,
'fields' => array('Event.orgc_id', 'Event.timestamp', 'Event.attribute_count'),
'conditions' => $conditions,
'order' => 'Event.timestamp'
));
$sparklineData = array();
foreach ($events as $event) {
$date = date("Y-m-d", $event['Event']['timestamp']);
if (!isset($sparklineData[$event['Event']['attribute_count']][$date])) {
$sparklineData[$date] = $event['Event']['attribute_count'];
} else {
$sparklineData[$date] += $event['Event']['attribute_count'];
}
}
// get first and last timestamp
if (isset($params['from'])) {
$startDate = $params['from'];
} else {
$startDate = $this->resolveTimeDelta($params['event_timestamp']);
}
if (isset($params['to'])) {
$endDate = $params['to'];
} else {
$endDate = time();
}
$dates = array();
for ($d=$startDate; $d < $endDate; $d=$d+3600*24) {
$dates[] = date('Y-m-d', $d);
}
$csv = 'Date,Close\n';
foreach ($dates as $date) {
$csv .= sprintf('%s,%s\n', $date, isset($sparklineData[$date]) ? $sparklineData[$date] : 0);
}
$data = array(
'csv' => $csv,
'data' => $sparklineData,
'orgId' => $orgId
);
return $data;
}
}

268
app/View/Attributes/add.ctp
View File

@ -1,166 +1,130 @@
<div class="attributes <?php if (!isset($ajax) || !$ajax) echo 'form';?>">
<?php
$url_params = $action == 'add' ? 'add/' . $event_id : 'edit/' . $attribute['Attribute']['id'];
echo $this->Form->create('Attribute', array('id', 'url' => '/attributes/' . $url_params));
?>
<fieldset>
<legend><?php echo $action == 'add' ? __('Add Attribute') : __('Edit Attribute'); ?></legend>
<div id="formWarning" class="message ajaxMessage"></div>
<div id="compositeWarning" class="message <?php echo !empty($ajax) ? 'ajaxMessage' : '';?>" style="display:none;">Did you consider adding an object instead of a composite attribute?</div>
<div class="add_attribute_fields">
<?php
echo $this->Form->hidden('event_id');
echo $this->Form->input('category', array(
'empty' => __('(choose one)'),
'label' => __('Category ') . $this->element('formInfo', array('type' => 'category')),
));
echo $this->Form->input('type', array(
'empty' => __('(first choose category)'),
'label' => __('Type ') . $this->element('formInfo', array('type' => 'type')),
));
$initialDistribution = 5;
if (Configure::read('MISP.default_attribute_distribution') != null) {
if (Configure::read('MISP.default_attribute_distribution') === 'event') {
$initialDistribution = 5;
} else {
$initialDistribution = Configure::read('MISP.default_attribute_distribution');
}
}
?>
<div class="input clear"></div>
<?php
$distArray = array(
'options' => array($distributionLevels),
'label' => __('Distribution ') . $this->element('formInfo', array('type' => 'distribution')),
);
if ($action == 'add') {
$distArray['selected'] = $initialDistribution;
}
echo $this->Form->input('distribution', $distArray);
?>
<div id="SGContainer" style="display:none;">
<?php
if (!empty($sharingGroups)) {
echo $this->Form->input('sharing_group_id', array(
'options' => array($sharingGroups),
'label' => __('Sharing Group'),
));
}
?>
</div>
<?php
echo $this->Form->input('value', array(
'type' => 'textarea',
'error' => array('escape' => false),
'div' => 'input clear',
'class' => 'input-xxlarge'
));
?>
<div class="input clear"></div>
<?php
echo $this->Form->input('comment', array(
$modelForForm = 'Attribute';
echo $this->element('genericElements/Form/genericForm', array(
'form' => $this->Form,
'data' => array(
'title' => $action === 'add' ? __('Add Attribute') : __('Edit Attribute'),
'model' => $modelForForm,
'fields' => array(
array(
'field' => 'event_id',
'class' => 'org-id-picker-hidden-field',
'type' => 'text',
'label' => __('Contextual Comment'),
'error' => array('escape' => false),
'hidden' => true
),
array(
'field' => 'category',
'class' => 'input',
'empty' => __('(choose one)'),
'options' => $categories,
'stayInLine' => 1
),
array(
'field' => 'type',
'class' => 'input',
'empty' => __('(choose category first)'),
'options' => $types
),
array(
'field' => 'distribution',
'class' => 'input',
'options' => $distributionLevels,
'default' => isset($attribute['Attribute']['distribution']) ? $attribute['Attribute']['distribution'] : $initialDistribution,
'stayInLine' => 1
),
array(
'field' => 'sharing_group_id',
'class' => 'input',
'options' => $sharingGroups,
'label' => __("Sharing Group")
),
array(
'field'=> 'value',
'type' => 'textarea',
'class' => 'input span6',
'div' => 'input clear'
),
array(
'field' => 'comment',
'type' => 'text',
'class' => 'input span6',
'div' => 'input clear',
'class' => 'input-xxlarge'
));
?>
<div class="input clear"></div>
<?php
echo $this->Form->input('to_ids', array(
'label' => __('for Intrusion Detection System'),
'type' => 'checkbox'
));
echo $this->Form->input('batch_import', array(
'label' => __("Contextual Comment")
),
array(
'field' => 'to_ids',
'type' => 'checkbox',
'label' => __("for Intrusion Detection System"),
//'stayInLine' => 1
),
array(
'field' => 'batch_import',
'type' => 'checkbox'
));
echo '<div class="input clear"></div>';
echo $this->Form->input('disable_correlation', array(
),
array(
'field' => 'disable_correlation',
'type' => 'checkbox'
));
?>
</div>
</fieldset>
<p id="notice_message" style="display:none;"></p>
<?php if ($ajax): ?>
<div class="overlay_spacing">
<span id="submitButton" class="btn btn-primary" style="margin-bottom:5px;float:left;" title="<?php echo __('Submit'); ?>" role="button" tabindex="0" aria-label="<?php echo __('Submit'); ?>" onClick="submitPopoverForm('<?php echo $action == 'add' ? $event_id : $attribute['Attribute']['id'];?>', '<?php echo $action; ?>')"><?php echo __('Submit'); ?></span>
<span class="btn btn-inverse" style="float:right;" title="<?php echo __('Cancel'); ?>" role="button" tabindex="0" aria-label="<?php echo __('Cancel'); ?>" id="cancel_attribute_add"><?php echo __('Cancel'); ?></span>
</div>
<?php
else:
?>
<?php
echo $this->Form->button('Submit', array('class' => 'btn btn-primary'));
endif;
echo $this->Form->end();
?>
</div>
<?php
),
array(
'field' => 'first_seen',
'type' => 'text',
'hidden' => true
),
array(
'field' => 'last_seen',
'type' => 'text',
'hidden' => true
),
'<div id="extended_event_preview" style="width:446px;"></div>'
),
'submit' => array(
'action' => $action,
'ajaxSubmit' => sprintf(
'submitPopoverForm(%s, %s, 0, 1)',
"'" . ($action == 'add' ? h($event_id) : h($attribute['Attribute']['id'])) . "'",
"'" . h($action) . "'"
)
),
'metaFields' => array(
'<div id="bothSeenSliderContainer" style="height: 170px;"></div>'
)
)
));
if (!$ajax) {
$event['Event']['id'] = $event_id;
$event['Event']['published'] = $published;
echo $this->element('/genericElements/SideMenu/side_menu', array('menuList' => 'event', 'menuItem' => 'addAttribute', 'event' => $event));
}
?>
<script type="text/javascript">
var notice_list_triggers = <?php echo $notice_list_triggers; ?>;
var fieldsArray = new Array('AttributeCategory', 'AttributeType', 'AttributeValue', 'AttributeDistribution', 'AttributeComment', 'AttributeToIds', 'AttributeBatchImport', 'AttributeSharingGroupId');
<?php
$formInfoTypes = array('distribution' => 'Distribution', 'category' => 'Category', 'type' => 'Type');
echo 'var formInfoFields = ' . json_encode($formInfoTypes) . PHP_EOL;
foreach ($formInfoTypes as $formInfoType => $humanisedName) {
echo 'var ' . $formInfoType . 'FormInfoValues = {' . PHP_EOL;
foreach ($info[$formInfoType] as $key => $formInfoData) {
echo '"' . $key . '": "<span class=\"blue bold\">' . h($formInfoData['key']) . '</span>: ' . h($formInfoData['desc']) . '<br />",' . PHP_EOL;
}
echo '}' . PHP_EOL;
echo $this->element('/genericElements/SideMenu/side_menu', array('menuList' => 'event-collection', 'menuItem' => $this->action === 'add' ? 'add' : 'editEvent'));
}
?>
//
//Generate Category / Type filtering array
//
var category_type_mapping = new Array();
<?php
<script type="text/javascript">
var notice_list_triggers = <?php echo $notice_list_triggers; ?>;
var composite_types = <?php echo json_encode($compositeTypes); ?>;
var category_type_mapping = new Array();
<?php
foreach ($categoryDefinitions as $category => $def) {
echo "category_type_mapping['" . addslashes($category) . "'] = {";
$first = true;
foreach ($def['types'] as $type) {
if ($first) $first = false;
else echo ', ';
if ($first) {
$first = false;
} else {
echo ', ';
}
echo "'" . addslashes($type) . "' : '" . addslashes($type) . "'";
}
echo "}; \n";
}
?>
var composite_types = <?php echo json_encode($compositeTypes); ?>;
$(document).ready(function() {
<?php
if ($action == 'edit'):
?>
checkNoticeList('attribute');
<?php
endif;
?>
initPopoverContent('Attribute');
$('#AttributeDistribution').change(function() {
if ($('#AttributeDistribution').val() == 4) $('#SGContainer').show();
else $('#SGContainer').hide();
checkSharingGroup('Attribute');
});
$("#AttributeCategory").on('change', function(e) {
$('#AttributeCategory').change(function() {
formCategoryChanged('Attribute');
if ($(this).val() === 'Internal reference') {
$("#AttributeDistribution").val('0');
$('#SGContainer').hide();
checkSharingGroup('Attribute');
}
});
@ -168,22 +132,16 @@ $(document).ready(function() {
checkNoticeList('attribute');
});
$("#AttributeCategory, #AttributeType, #AttributeDistribution").change(function() {
var start = $("#AttributeType").val();
initPopoverContent('Attribute');
$("#AttributeType").val(start);
if ($.inArray(start, composite_types) > -1) {
$('#compositeWarning').show();
} else {
$('#compositeWarning').hide();
}
$(document).ready(function() {
<?php
if ($action == 'edit'):
?>
checkNoticeList('attribute');
<?php
endif;
?>
checkSharingGroup('Attribute');
});
<?php if ($ajax): ?>
$('#cancel_attribute_add').click(function() {
cancelPopoverForm();
});
<?php endif; ?>
});
</script>
<?php echo $this->element('form_seen_input'); ?>
<?php echo $this->Js->writeBuffer(); // Write cached scripts

17
app/View/Attributes/ajax/attributeEditFirst_seenForm.ctp Normal file
View File

@ -0,0 +1,17 @@
<?php
echo $this->Form->create('Attribute', array(
'id' => 'Attribute' . '_' . $object['id'] . '_first_seen_form',
'url' => '/attributes/editField/' . $object['id']
));
?>
<?php
echo $this->Form->input('first_seen', array(
'label' => false,
'type' => 'text',
'value' => 0,
'id' => 'Attribute' . '_' . $object['id'] . '_first_seen_field',
'div' => false
));
echo $this->Form->end();
?>
</div>

15
app/View/Attributes/ajax/attributeEditLast_seenForm.ctp Normal file
View File

@ -0,0 +1,15 @@
<?php
echo $this->Form->create('Attribute', array(
'id' => 'Attribute' . '_' . $object['id'] . '_last_seen_form',
'url' => '/attributes/editField/' . $object['id']
));
echo $this->Form->input('last_seen', array(
'label' => false,
'type' => 'text',
'value' => 0,
'id' => 'Attribute' . '_' . $object['id'] . '_last_seen_field',
'div' => false
));
echo $this->Form->end();
?>
</div>

4
app/View/Attributes/ajax/attributeViewFieldForm.ctp
View File

@ -1,8 +1,8 @@
<?php
if ($value === 'No') {
echo '<input type="checkbox" disabled></input>';
echo '<input type="checkbox" disabled>';
} else if ($value === 'Yes') {
echo '<input type="checkbox" checked disabled></input>';
echo '<input type="checkbox" checked disabled>';
} else {
echo nl2br(h($value)) . '&nbsp;';
}

22
app/View/Attributes/search.ctp
View File

@ -32,13 +32,27 @@
'type' => 'checkbox',
'label' => __('Alternate Search Result (Events)')
));
echo $this->Form->input('first_seen', array(
'type' => 'text',
'div' => 'input hidden',
'required' => false,
));
echo $this->Form->input('last_seen', array(
'type' => 'text',
'div' => 'input hidden',
'required' => false,
));
?>
<div class="clear">
<h3><?php echo __('First seen and Last seen.'); ?></h3>
<p><?php echo __('Attributes not having first seen or last seen set might not appear in the search'); ?></p>
</div>
</fieldset>
<?php
echo $this->Form->button(__('Search'), array('class' => 'btn btn-primary'));
echo $this->Form->end();
?>
<?php echo $this->Form->end(); ?>
<div id="bothSeenSliderContainer"></div>
<button onclick="$('#AttributeSearchForm').submit();" class="btn btn-primary">Submit</button>
</div>
<?php echo $this->element('form_seen_input'); ?>
<script type="text/javascript">
//
// Generate Category / Type filtering array

9
app/View/Communities/view.ctp
View File

@ -56,12 +56,3 @@
<?php
echo $this->element('/genericElements/SideMenu/side_menu', array('menuList' => 'sync', 'menuItem' => 'view_community'));
?>
<script type="text/javascript">
<?php
$startingTab = 'description';
if (!$local) $startingTab = 'events';
?>
$(document).ready(function () {
organisationViewContent('<?php echo $startingTab; ?>', '<?php echo h($id);?>');
});
</script>

18
app/View/DecayingModel/decaying_tool.ctp
View File

@ -5,21 +5,21 @@
<div class="span9 form-inline" style="border: 1px solid #ddd; border-radius: 4px; margin-bottom: 15px;">
<div style="border-bottom: 1px solid #ddd;">
<label class="checkbox inline">
<input id="table_toggle_all_type" type="checkbox"></input>
<input id="table_toggle_all_type" type="checkbox">
<?php echo __('Show All Types'); ?>
</label>
<label class="checkbox inline">
<input id="table_toggle_objects" type="checkbox"></input>
<input id="table_toggle_objects" type="checkbox">
<?php echo __('Show MISP Objects'); ?>
</label>
<input id="table_type_search" class="input" type="text" placeholder="<?php echo __('Search Attribute Type'); ?>"></input>
<input id="table_type_search" class="input" type="text" placeholder="<?php echo __('Search Attribute Type'); ?>">
<button class="btn btn-primary btn-small" onclick="decayingTool.restoreSelection()"><span class="fa fa-history"></span></button>
</div>
<div class="AttributeTypeTableContainer">
<table id="table_attribute_type" class="table table-striped table-bordered">
<thead>
<tr>
<th><input id="checkAll" type="checkbox" title="<?php echo __('Check all'); ?>"></input></th>
<th><input id="checkAll" type="checkbox" title="<?php echo __('Check all'); ?>"></th>
<th><?php echo __('Attribute Type'); ?></th>
<th><?php echo __('Category'); ?></th>
<th><?php echo __('Model ID'); ?></th>
@ -41,7 +41,7 @@
}
?>
<tr class="<?php echo $class; ?>">
<td><input type="checkbox"></input></td>
<td><input type="checkbox"></td>
<td class="useCursorPointer isFilteringField isAttributeTypeField">
<?php if(isset($info['isObject']) && $info['isObject'] && !(isset($info['isAttribute']) && $info['isAttribute'])): ?>
<it class="fa fa-cube" title="<?php echo __('Belong to a MISP Object'); ?>"></it>
@ -85,17 +85,17 @@
<span class="add-on param-name" data-toggle="tooltip" data-placement="left" style="min-width: 100px;" title="<?php echo isset($config['info']) ? h($config['info']) : ''?>">
<?php echo h($config['name']) . (isset($config['greek']) ? ' <strong>' . h($config['greek']).'</strong>' : ''); ?>
</span>
<input id="input_<?php echo h($param); ?>" class="input-mini" type="number" min=0 step=<?php echo h($config['step']); ?> value=<?php echo h($config['value']); ?> max=<?php echo isset($config['max']) ? h($config['max']) : ''; ?> oninput="refreshGraph(this);" ></input>
<span class="add-on"><input id="input_<?php echo h($param); ?>_range" type="range" min=0 <?php echo isset($config['max']) ? 'max=' . h($config['max']) : '' ?> step=<?php echo h($config['step']); ?> value=<?php echo h($config['value']); ?> oninput="$('#input_<?php echo h($param); ?>').val(this.value).trigger('input');"></input></span>
<input id="input_<?php echo h($param); ?>" class="input-mini" type="number" min=0 step=<?php echo h($config['step']); ?> value=<?php echo h($config['value']); ?> max=<?php echo isset($config['max']) ? h($config['max']) : ''; ?> oninput="refreshGraph(this);">
<span class="add-on"><input id="input_<?php echo h($param); ?>_range" type="range" min=0 <?php echo isset($config['max']) ? 'max=' . h($config['max']) : '' ?> step=<?php echo h($config['step']); ?> value=<?php echo h($config['value']); ?> oninput="$('#input_<?php echo h($param); ?>').val(this.value).trigger('input');"></span>
<?php if (isset($config['unit'])): ?>
<span class="add-on"><?php echo h($config['unit']); ?></span>
<?php endif; ?>
</div>
<?php endforeach; ?>
<input id="input_default_base_score" value=0 class="hidden"></input>
<input id="input_default_base_score" value=0 class="hidden">
<div class="input-append" style="margin-bottom: 0px;">
<input id="input_base_score_config" class="hidden" value="[]"></input>
<input id="input_base_score_config" class="hidden" value="[]">
<button class="btn btn-primary" style="border-radius: 4px 0px 0px 4px;" onclick="decayingTool.toggleBasescoreForm()">
<span class="fa fa-tags"> <?php echo __('Adjust base score'); ?></span>
</button>

12
app/View/DecayingModel/decaying_tool_basescore.ctp
View File

@ -1,11 +1,11 @@
<div id="basescore_configurator" class="row">
<div class="span8" class="taxonomyTableContainer">
<input id="table_taxonomy_search" class="input" style="width: 250px; margin: 0px;" type="text" placeholder="<?php echo __('Search Taxonomy'); ?>"></input>
<input id="table_taxonomy_search" class="input" style="width: 250px; margin: 0px;" type="text" placeholder="<?php echo __('Search Taxonomy'); ?>">
<i class="fa fa-times useCursorPointer" title="<?php echo __('Clear search field'); ?>" onclick="$('#table_taxonomy_search').val('').trigger('input');"></i>
<span style="float: right; margin-top: 6px;" class="badge badge-info"><b><?php echo h($taxonomies_not_having_numerical_value); ?></b><?php echo __(' not having numerical value'); ?></span>
<div class="input-prepend" style="margin: 4px;">
<span class="add-on"><?php echo __('Default basescore') ?></span>
<input id="base_score_default_value" type="number" min=0 max=100 class="input-mini" value="0" placeholder="0"></input>
<input id="base_score_default_value" type="number" min=0 max=100 class="input-mini" value="0" placeholder="0">
</div>
<table id="tableTaxonomy" class="table table-striped table-bordered table-condensed">
<thead>
@ -49,8 +49,8 @@
</div>
</td>
<td>
<input id="slider_<?php echo h($name) ?>" data-taxonomyname="<?php echo sprintf('%s:%s', h($name), h($predicate['value'])); ?>" type="range" min=0 max=100 step=1 value="<?php echo isset($taxonomy['value']) ? h($taxonomy['value']) : 0 ?>" onchange="sliderChanged(this);" oninput="sliderChanged(this);"></input>
<input type="number" min=0 max=100 step=1 value="<?php echo isset($taxonomy['value']) ? h($taxonomy['value']) : 0 ?>" class="taxonomySlider" data-taxonomyname="<?php echo sprintf('%s:%s', h($name), h($predicate['value'])); ?>" onchange="inputChanged(this);" oninput="inputChanged(this);"></input>
<input id="slider_<?php echo h($name) ?>" data-taxonomyname="<?php echo sprintf('%s:%s', h($name), h($predicate['value'])); ?>" type="range" min=0 max=100 step=1 value="<?php echo isset($taxonomy['value']) ? h($taxonomy['value']) : 0 ?>" onchange="sliderChanged(this);" oninput="sliderChanged(this);">
<input type="number" min=0 max=100 step=1 value="<?php echo isset($taxonomy['value']) ? h($taxonomy['value']) : 0 ?>" class="taxonomySlider" data-taxonomyname="<?php echo sprintf('%s:%s', h($name), h($predicate['value'])); ?>" onchange="inputChanged(this);" oninput="inputChanged(this);">
</td>
</tr>
<?php else: // numerical_value on predicate ?>
@ -77,8 +77,8 @@
</div>
</td>
<td>
<input id="slider_<?php echo h($name) ?>" data-taxonomyname="<?php echo h($name); ?>" type="range" min=0 max=100 step=1 value="<?php echo isset($taxonomy['value']) ? h($taxonomy['value']) : 0 ?>" onchange="sliderChanged(this);" oninput="sliderChanged(this);"></input>
<input type="number" min=0 max=100 step=1 value="<?php echo isset($taxonomy['value']) ? h($taxonomy['value']) : 0 ?>" class="taxonomySlider" data-taxonomyname="<?php echo h($name); ?>" onchange="inputChanged(this);" oninput="inputChanged(this);"></input>
<input id="slider_<?php echo h($name) ?>" data-taxonomyname="<?php echo h($name); ?>" type="range" min=0 max=100 step=1 value="<?php echo isset($taxonomy['value']) ? h($taxonomy['value']) : 0 ?>" onchange="sliderChanged(this);" oninput="sliderChanged(this);">
<input type="number" min=0 max=100 step=1 value="<?php echo isset($taxonomy['value']) ? h($taxonomy['value']) : 0 ?>" class="taxonomySlider" data-taxonomyname="<?php echo h($name); ?>" onchange="inputChanged(this);" oninput="inputChanged(this);">
</td>
</tr>
<?php break; ?>

10
app/View/Elements/Events/View/attribute_correlations.ctp
View File

@ -1,6 +1,16 @@
<?php
$i = 0;
$linkColour = ($scope == 'Attribute') ? 'red' : 'white';
// remove duplicates
$relatedEvents = array();
foreach ($event['Related' . $scope][$object['id']] as $k => $relatedAttribute) {
if (isset($relatedEvents[$relatedAttribute['id']])) {
unset($event['Related' . $scope][$object['id']][$k]);
} else {
$relatedEvents[$relatedAttribute['id']] = true;
}
}
$event['Related' . $scope][$object['id']] = array_values($event['Related' . $scope][$object['id']]);
$count = count($event['Related' . $scope][$object['id']]);
foreach ($event['Related' . $scope][$object['id']] as $relatedAttribute) {
if ($i == 4 && $count > 5) {

2
app/View/Elements/Events/View/eventFilteringQueryBuilder.ctp
View File

@ -1,7 +1,7 @@
<div id="eventFilteringQBWrapper" style="padding: 5px; display: none; border: 1px solid #dddddd; border-bottom: 0px;">
<div id="eventFilteringQB" style="overflow-y: auto; padding-right: 5px; resize: vertical; max-height: 750px; height: 400px;"></div>
<div style="display: flex; justify-content: flex-end; margin-top: 5px;">
<input id="eventFilteringQBLinkInput" class="form-control" style="width: 66%;"></input>
<input id="eventFilteringQBLinkInput" class="form-control" style="width: 66%;">
<button id="eventFilteringQBLinkCopy" type="button" class="btn btn-inverse" style="margin-right: 5px; margin-left: 5px;" onclick="clickMessage(this);"> <i class="fa fa-clipboard"></i> <?php echo h('Copy to clipboard'); ?> </button>
<button id="eventFilteringQBSubmit" type="button" class="btn btn-success" style="margin-right: 5px;"> <i class="fa fa-filter"></i> <?php echo h('Filter'); ?> </button>
<button id="eventFilteringQBClear" type="button" class="btn btn-xs btn-danger" style="" title="<?php echo h('Clear filtering rules'); ?>"> <i class="fa fa-times"></i> <?php echo h('Clear'); ?> </button>

7
app/View/Elements/Events/View/related_event.ctp
View File

@ -1,5 +1,8 @@
<?php
$href_url = isset($href_url) ? $href_url : $baseurl . '/events';
$href_url = isset($href_url) ? $href_url : $baseurl . '/events/view/' . h($related['id']);
if (isset($from_id)) {
$href_url .= sprintf('/%s/%s', 1, $from_id);
}
$hide = isset($hide) ? $hide : false;
?>
<span class="<?php echo $hide ? 'hidden correlation-expanded-area' : '' ?>">
@ -11,7 +14,7 @@
<?php echo $this->OrgImg->getOrgImg(array('name' => $related['Orgc']['name'], 'id' => $related['Orgc']['id'], 'size' => 24)); ?>
</td>
<td style="line-height: 14px; padding-left: 2px; white-space: nowrap; text-overflow: ellipsis; overflow: hidden; max-width: 430px;">
<a title="<?php echo h($related['info']); ?>" href="<?php echo h($href_url) . '/' . $related['id']?>">
<a title="<?php echo h($related['info']); ?>" href="<?php echo h($href_url)?>">
<span><?php echo h($related['info']) ?>
</a>
</td>

81
app/View/Elements/Events/View/row_attribute.ctp
View File

@ -54,6 +54,9 @@
<td class="short context hidden">
<?php echo h($object['uuid']); ?>
</td>
<td class="short context hidden">
<?php echo $this->element('/Events/View/seen_field', array('object' => $object)); ?>
</td>
<td class="short">
<?php echo date('Y-m-d', $object['timestamp']); ?>
</td>
@ -156,7 +159,7 @@
</td>
<td class="short">
<div class="attributeTagContainer" id="#Attribute_<?php echo h($object['id']);?>_tr .attributeTagContainer">
<?php echo $this->element('ajaxTags', array('attributeId' => $object['id'], 'tags' => $object['AttributeTag'], 'tagAccess' => ($isSiteAdmin || $mayModify || $me['org_id'] == $event['Event']['org_id']), 'context' => $context, 'scope' => 'attribute')); ?>
<?php echo $this->element('ajaxTags', array('attributeId' => $object['id'], 'tags' => $object['AttributeTag'], 'tagAccess' => ($isSiteAdmin || $mayModify || $me['org_id'] == $event['Event']['org_id']), 'context' => $context, 'scope' => 'attribute', 'tagConflicts' => isset($object['tagConflicts']) ? $object['tagConflicts'] : array())); ?>
</div>
</td>
<?php
@ -318,44 +321,54 @@
</ul>
</td>
<td class="short">
<div id = "Attribute_<?php echo $object['id']; ?>_to_ids_placeholder" class = "inline-field-placeholder"></div>
<div id = "Attribute_<?php echo $object['id']; ?>_to_ids_solid" class="inline-field-solid">
<input type="checkbox" class="toids-toggle" id="toids_toggle_<?php echo h($object['id']); ?>" data-attribute-id="<?php echo h($object['id']); ?>" aria-label="<?php echo __('Toggle IDS flag');?>" title="<?php echo __('Toggle IDS flag');?>" <?php echo $object['to_ids'] ? 'checked' : ''; ?> ></input>
</div>
<div id = "Attribute_<?php echo $object['id']; ?>_to_ids_placeholder" class = "inline-field-placeholder"></div>
<div id = "Attribute_<?php echo $object['id']; ?>_to_ids_solid" class="inline-field-solid">
<input type="checkbox" class="toids-toggle" id="toids_toggle_<?php echo h($object['id']); ?>" data-attribute-id="<?php echo h($object['id']); ?>" aria-label="<?php echo __('Toggle IDS flag');?>" title="<?php echo __('Toggle IDS flag');?>" <?php echo $object['to_ids'] ? 'checked' : ''; ?> >
</div>
</td>
<td class="short" onmouseenter="quickEditHover(this, '<?php echo $editScope; ?>', '<?php echo $object['id']; ?>', 'distribution', <?php echo $event['Event']['id'];?>);">
<?php
$turnRed = '';
if ($object['distribution'] == 0) $turnRed = 'style="color:red"';
?>
<div id = "Attribute_<?php echo $object['id']; ?>_distribution_placeholder" class = "inline-field-placeholder"></div>
<div id = "Attribute_<?php echo $object['id']; ?>_distribution_solid" <?php echo $turnRed; ?> class="inline-field-solid">
<?php
if ($object['distribution'] == 4):
$turnRed = '';
if ($object['distribution'] == 0) {
$turnRed = 'style="color:red"';
}
?>
<a href="/sharing_groups/view/<?php echo h($object['sharing_group_id']); ?>"><?php echo h($object['SharingGroup']['name']);?></a>
<?php
else:
echo h($shortDist[$object['distribution']]);
endif;
?>
</div>
<div id = "Attribute_<?php echo $object['id']; ?>_distribution_placeholder" class = "inline-field-placeholder"></div>
<div id = "Attribute_<?php echo $object['id']; ?>_distribution_solid" <?php echo $turnRed; ?> class="inline-field-solid">
<?php
if ($object['distribution'] == 4):
?>
<a href="/sharing_groups/view/<?php echo h($object['sharing_group_id']); ?>"><?php echo h($object['SharingGroup']['name']);?></a>
<?php
else:
echo h($shortDist[$object['distribution']]);
endif;
?>
</div>
</td>
<?php
echo $this->element('/Events/View/sighting_field', array(
'object' => $object,
'tr_class' => $tr_class,
'page' => $page
));
?>
<?php if (!empty($includeDecayScore)): ?>
<td class="decayingScoreField">
<div id = "Attribute_<?php echo h($object['id']); ?>_score_solid" class="inline-field-solid">
<?php echo $this->element('DecayingModels/View/attribute_decay_score', array('scope' => 'object', 'object' => $object, 'uselink' => true)); ?>
</div>
</td>
<?php endif; ?>
<td class="short action-links">
<?php
echo $this->element('/Events/View/sighting_field', array(
'object' => $object,
'tr_class' => $tr_class,
'page' => $page
));
if (!empty($includeSightingdb)) {
echo $this->element('/Events/View/sightingdb_field', array(
'object' => $object,
'tr_class' => $tr_class,
'page' => $page
));
}
if (!empty($includeDecayScore)): ?>
<td class="decayingScoreField">
<div id = "Attribute_<?php echo h($object['id']); ?>_score_solid" class="inline-field-solid">
<?php echo $this->element('DecayingModels/View/attribute_decay_score', array('scope' => 'object', 'object' => $object, 'uselink' => true)); ?>
</div>
</td>
<?php
endif;
?>
<td class="short action-links">
<?php
if ($object['deleted']):
if ($isSiteAdmin || $mayModify):

32
app/View/Elements/Events/View/row_object.ctp
View File

@ -1,7 +1,7 @@
<?php
$tr_class = '';
$linkClass = 'white';
$currentType = 'denyForm';
$currentType = 'Object';
$tr_class = 'tableHighlightBorderTop borderBlue';
if ($event['Event']['id'] != $object['event_id']) {
if (!$isSiteAdmin && $event['extensionEvents'][$object['event_id']]['Orgc']['id'] != $me['org_id']) {
@ -36,6 +36,9 @@
<td class="short context hidden">
<?php echo h($object['uuid']); ?>
</td>
<td class="short context hidden">
<?php echo $this->element('/Events/View/seen_field', array('object' => $object)); ?>
</td>
<td class="short">
<?php echo date('Y-m-d', $object['timestamp']); ?>
</td>
@ -85,18 +88,21 @@
}
?>
</td>
<td class="shortish">
<?php echo h($object['comment']); ?>
<td class="showspaces bitwider" onmouseenter="quickEditHover(this, 'Object', '<?php echo $object['id']; ?>', 'comment', <?php echo $event['Event']['id'];?>);">
<div id = "Object_<?php echo $object['id']; ?>_comment_placeholder" class = "inline-field-placeholder"></div>
<div id = "Object_<?php echo $object['id']; ?>_comment_solid" class="inline-field-solid">
<?php echo nl2br(h($object['comment'])); ?>&nbsp;
</div>
</td>
<td colspan="4">&nbsp;
</td>
<td class="shortish">
<td class="shortish" onmouseenter="quickEditHover(this, 'Object', '<?php echo $object['id']; ?>', 'distribution', <?php echo $event['Event']['id'];?>);">
<?php
$turnRed = '';
if ($object['objectType'] == 0 && $object['distribution'] == 0) $turnRed = 'style="color:red"';
?>
<div id = "<?php echo $currentType . '_' . $object['id'] . '_distribution_placeholder'; ?>" class = "inline-field-placeholder"></div>
<div id = "<?php echo $currentType . '_' . $object['id'] . '_distribution_solid'; ?>" <?php echo $turnRed; ?> class="inline-field-solid" ondblclick="activateField('<?php echo $currentType; ?>', '<?php echo $object['id']; ?>', 'distribution', <?php echo $event['Event']['id'];?>);">
<div id = "<?php echo $currentType . '_' . $object['id'] . '_distribution_solid'; ?>" <?php echo $turnRed; ?> class="inline-field-solid">
<?php
if ($object['objectType'] == 0) {
if ($object['distribution'] == 4):
@ -112,19 +118,24 @@
</td>
<td>&nbsp;</td>
<td>&nbsp;</td>
<?php if (!empty($includeDecayScore)): ?>
<td class="decayingScoreField">&nbsp;</td>
<?php endif; ?>
<?php
$paddedFields = array('includeSightingdb', 'includeDecayScore');
foreach ($paddedFields as $paddedField) {
if (!empty(${$paddedField})) {
echo '<td>&nbsp;</td>';
}
}
?>
<td class="short action-links">
<?php
if ($mayModify && empty($object['deleted'])) {
echo sprintf(
'<a href="%s/objects/edit/%s" title="Edit" aria-label="Edit" class="fa fa-edit icon-white useCursorPointer"></a>',
'<a href="%s/objects/edit/%s" title="Edit" aria-label="Edit" class="fa fa-edit white useCursorPointer"></a>',
$baseurl,
h($object['id'])
);
echo sprintf(
'<span class="fa fa-trash icon-white useCursorPointer" title="%1$s" role="button" tabindex="0" aria-label="%1$s" onClick="%2$s"></span>',
'<span class="fa fa-trash white useCursorPointer" title="%1$s" role="button" tabindex="0" aria-label="%1$s" onClick="%2$s"></span>',
(empty($event['Event']['publish_timestamp']) ? __('Permanently delete object') : __('Soft delete object')),
sprintf(
'deleteObject(\'objects\', \'delete\', \'%s\', \'%s\');',
@ -150,5 +161,6 @@
'child' => $attrKey == $lastElement ? 'last' : true
));
}
echo '<tr class="objectAddFieldTr"><td><span class="fa fa-plus-circle objectAddField" title="' . __('Add an Object Attribute') .'" onclick="popoverPopup(this, ' . h($object['id']) . ', \'objects\', \'quickFetchTemplateWithValidObjectAttributes\')"></span></td></tr>';
}
?>

16
app/View/Elements/Events/View/row_proposal.ctp
View File

@ -49,6 +49,9 @@
<td class="short context hidden">
<?php echo $object['objectType'] == 0 ? h($object['uuid']) : '&nbsp;'; ?>
</td>
<td class="short context hidden">
<?php echo $this->element('/Events/View/seen_field', array('object' => $object)); ?>
</td>
<td class="short">
<div id = "<?php echo $currentType . '_' . $object['id'] . '_timestamp_solid'; ?>">
<?php
@ -174,9 +177,14 @@
<td class="shortish">&nbsp;</td>
<td class="shortish">&nbsp;</td>
<td class="short">&nbsp;</td>
<?php if (!empty($includeDecayScore)): ?>
<td class="decayingScoreField">&nbsp;</td>
<?php endif; ?>
<?php
$paddedFields = array('includeSightingdb', 'includeDecayScore');
foreach ($paddedFields as $paddedField) {
if (!empty(${$paddedField})) {
echo '<td>&nbsp;</td>';
}
}
?>
<td class="short action-links">
<?php
if (($event['Orgc']['id'] == $me['org_id'] && $mayModify) || $isSiteAdmin) {
@ -188,7 +196,7 @@
}
if (($event['Orgc']['id'] == $me['org_id'] && $mayModify) || $isSiteAdmin || ($object['org_id'] == $me['org_id'])) {
?>
<span class="fa fa-trash icon-white useCursorPointer" title="<?php echo __('Discard proposal');?>" role="button" tabindex="0" aria-label="<?php echo __('Discard proposal');?>" onClick="deleteObject('shadow_attributes', 'discard' ,'<?php echo $object['id']; ?>', '<?php echo $event['Event']['id']; ?>');"></span>
<span class="fa fa-trash white useCursorPointer" title="<?php echo __('Discard proposal');?>" role="button" tabindex="0" aria-label="<?php echo __('Discard proposal');?>" onClick="deleteObject('shadow_attributes', 'discard' ,'<?php echo $object['id']; ?>', '<?php echo $event['Event']['id']; ?>');"></span>
<?php
}
?>

5
app/View/Elements/Events/View/row_proposal_delete.ctp
View File

@ -50,6 +50,9 @@
<td class="short context hidden">
<?php echo h($object['uuid']); ?>
</td>
<td class="short context hidden">
<?php echo $this->element('/Events/View/seen_field', array('object' => $object)); ?>
</td>
<td style="font-weight:bold;text-align:left;">DELETE</td>
<?php
if ($extended):
@ -79,7 +82,7 @@
}
if (($event['Orgc']['id'] == $me['org_id'] && $mayModify) || $isSiteAdmin || ($object['org_id'] == $me['org_id'])) {
?>
<span class="fa fa-trash icon-white useCursorPointer" title="<?php echo __('Discard proposal');?>" role="button" tabindex="0" aria-label="<?php echo __('Discard proposal');?>" onClick="deleteObject('shadow_attributes', 'discard' ,'<?php echo $object['id']; ?>', '<?php echo $event['Event']['id']; ?>');"></span>
<span class="fa fa-trash white useCursorPointer" title="<?php echo __('Discard proposal');?>" role="button" tabindex="0" aria-label="<?php echo __('Discard proposal');?>" onClick="deleteObject('shadow_attributes', 'discard' ,'<?php echo $object['id']; ?>', '<?php echo $event['Event']['id']; ?>');"></span>
<?php
}
?>

9
app/View/Elements/Events/View/seen_field.ctp Normal file
View File

@ -0,0 +1,9 @@
<?php if ($object['first_seen'] != null || $object['last_seen'] != null): ?>
<div>
<div><?php echo $object['first_seen'] != null ? h($object['first_seen']) : '<span style="display: block; text-align:center;">_</span>'; ?></div>
<i style="display: block; text-align: center;" class="fas fa-arrow-down"></i>
<div><?php echo $object['last_seen'] != null ? h($object['last_seen']) : '<span style="display: block; text-align:center;">_</span>'; ?></div>
</div>
<?php else: ?>
<div></div>
<?php endif; ?>

33
app/View/Elements/Events/View/sightingdb_field.ctp Normal file
View File

@ -0,0 +1,33 @@
<?php
$hits = array();
if (!empty($object['Sightingdb'])) {
foreach ($object['Sightingdb'] as $sightingdb_hit) {
$popover = array();
$popoverData = array(
'SightingDB' => $sightingdbs[$sightingdb_hit['sightingdb_id']]['Sightingdb']['name'],
'Count' => $sightingdb_hit['count'],
'First seen' => date('Y-m-d H:i:s', $sightingdb_hit['first_seen']),
'Last seen' => date('Y-m-d H:i:s', $sightingdb_hit['last_seen']),
'SightingDB owner' => $sightingdbs[$sightingdb_hit['sightingdb_id']]['Sightingdb']['owner'],
'SightingDB description' => $sightingdbs[$sightingdb_hit['sightingdb_id']]['Sightingdb']['description']
);
foreach ($popoverData as $k => $v) {
$popover[] = sprintf(
'<span class="bold black">%s</span>: <span class="blue">%s</span>',
h($k),
h($v)
);
}
$hits[] = sprintf(
'<span data-toggle="popover" data-content="%s" data-trigger="hover" data-placement="left"><span class="blue bold">%s</span>: %s</span>',
implode('<br />', h($popover)),
h($sightingdbs[$sightingdb_hit['sightingdb_id']]['Sightingdb']['name']),
h($sightingdb_hit['count'])
);
}
}
echo sprintf(
'<td class="short">%s</td>',
implode('<br />', $hits)
);
?>

5
app/View/Elements/Events/View/value_field.ctp
View File

@ -5,7 +5,7 @@ switch ($object['type']) {
case 'attachment':
case 'malware-sample':
if ($object['type'] === 'attachment' && isset($object['image'])) {
if (extension_loaded('gd')) {
if ($object['image'] === true) {
$img = '<it class="fa fa-spin fa-spinner" style="font-size: large; left: 50%; top: 50%;"></it>';
$img .= '<img class="screenshot screenshot-collapsed useCursorPointer img-rounded hidden" src="' . $baseurl . '/attributes/viewPicture/' . h($object['id']) . '/1' . '" title="' . h($object['value']) . '" onload="$(this).show(200); $(this).parent().find(\'.fa-spinner\').remove();"/>';
echo $img;
@ -24,7 +24,8 @@ switch ($object['type']) {
$filename = $filenameHash[0];
}
$url = array('controller' => 'attributes', 'action' => 'download', $object['id']);
$controller = isset($object['objectType']) && $object['objectType'] === 'proposal' ? 'shadow_attributes' : 'attributes';
$url = array('controller' => $controller, 'action' => 'download', $object['id']);
echo $this->Html->link($filename, $url, array('class' => $linkClass));
if (isset($filenameHash[1])) {
echo '<br />' . $filenameHash[1];

4
app/View/Elements/Feeds/View/row_attribute.ctp
View File

@ -1,5 +1,4 @@
<?php
if ($object['value'] == 'MERGE') debug($object);
$tr_class = '';
$linkClass = 'blue';
$otherColour = 'blue';
@ -23,6 +22,9 @@ if ($object['value'] == 'MERGE') debug($object);
<td class="short">
<?php echo date('Y-m-d', $object['timestamp']); ?>
</td>
<td class="short">
<?php echo $this->element('/Servers/View/seen_field', array('object' => $object)); ?>
</td>
<td class="short">
<div id="Attribute_<?php echo $object['uuid']; ?>_category_solid" class="inline-field-solid">
<?php echo h($object['category']); ?>

3
app/View/Elements/Feeds/View/row_object.ctp
View File

@ -12,6 +12,9 @@
<td class="short">
<?php echo date('Y-m-d', $object['timestamp']); ?>
</td>
<td class="short">
<?php echo $this->element('/Servers/View/seen_field', array('object' => $object)); ?>
</td>
<td colspan="<?php echo $fieldCount -2;?>">
<span class="bold"><?php echo __('Name: ');?></span><?php echo h($object['name']);?>
<span class="fa fa-expand useCursorPointer" title="<?php echo __('Expand or Collapse');?>" role="button" tabindex="0" aria-label="<?php echo __('Expand or Collapse');?>" data-toggle="collapse" data-target="#Object_<?php echo h($object['uuid']); ?>_collapsible"></span>

Some files were not shown because too many files have changed in this diff Show More