Sami Mokaddem
67d76d9b7a
Merge branch 'develop' into fix-duplicated-uuids
2022-09-05 12:40:29 +02:00
Sami Mokaddem
0799ff8c6c
chg: [appModel] Removed unused functions
2022-09-05 12:37:53 +02:00
Sami Mokaddem
fd6c824172
Merge branch 'develop' of github.com:MISP/MISP into fix-duplicated-uuids
2022-09-05 12:33:38 +02:00
Sami Mokaddem
bc6d4a8305
chg: [appModel] Usage of `update` worker instead of `prio`
2022-09-01 15:55:37 +02:00
Sami Mokaddem
646d921d67
Merge remote-tracking branch 'origin/develop' into tomking2_feature/propagate_tag_deletion
2022-09-01 11:07:35 +02:00
Sami Mokaddem
7b504fd133
Merge branch 'develop' of github.com:MISP/MISP into feature-periodic-notification
2022-09-01 09:58:41 +02:00
Sami Mokaddem
c9327174b9
Merge branch 'fix-truncate-overcorrelating-value' into 2.4
2022-08-29 15:10:48 +02:00
Sami Mokaddem
05ad105750
fix: [update-91] Remove duplicates before creating the constraint
2022-08-29 12:12:39 +02:00
Sami Mokaddem
962754dd3b
chg: [overCorrelatingValue] Truncated the `value` column
...
- We keep the unique constraint on the table
- Correlating values over the max. allowed size are truncated to fit the size requirement. That means large correlating values might be marked as over-correlating despite the fact they are not (as only the starting portion of the value is evaluated).
2022-08-29 10:50:59 +02:00
Sami Mokaddem
8db8cbd398
new: [user:periodicNotification] Started development of system allowing users to receive period notifications by email
2022-08-29 09:03:42 +02:00
Jakub Onderka
ec209a98b9
chg: [internal] Drop unnecessary indexes from default_correlations table
2022-08-11 12:54:49 +02:00
iglocska
c764bb0e8f
Merge branch 'log_last_api' into develop
2022-08-11 09:36:30 +02:00
iglocska
097ad78d54
chg: [correlations] performance tuning
...
- added a new constraint to check for correlation uniqueness (a_id, 1_a_id, value_id)
- stopped dropping correlations on a full recorrelation
- only correlate "upwards" towards higher IDs
2022-08-09 14:41:59 +02:00
iglocska
9dc233bc9e
fix: [proposal] correlations are deprecated
2022-08-05 12:55:04 +02:00
Tom King
64d508b4e3
fix: Rollback change to DB upgrade 86
2022-08-05 09:41:05 +01:00
Tom King
d2a7b4b3c8
fix: Use correct field for altering table to include last_api_access
2022-08-05 09:35:15 +01:00
iglocska
ea0932b7b1
chg: [correlations] value field changed to varchar
2022-08-04 13:49:30 +02:00
Sami Mokaddem
79d182dc79
fix: [appModel:update] Fixed forgotten schema update for workflows
2022-08-04 11:08:19 +02:00
Sami Mokaddem
5074528d91
fix: [appModel:update] Set update success to true for case 89 and 90
2022-08-04 10:58:44 +02:00
Sami Mokaddem
49575533ad
Merge remote-tracking branch 'origin/develop' into feature-workflows-2
2022-08-04 10:08:56 +02:00
iglocska
bce4508716
chg: [grammar] minor fix
...
- fat finger induced typo
2022-08-04 08:20:20 +02:00
iglocska
a04f3964e7
new: [correlation rework] round 2
...
- long list of fixes
- update scripts
- correlation engine management interface
- recorrelation/truncation tools
- various performance tunings and bug fixes
2022-08-03 15:44:27 +02:00
Sami Mokaddem
44b2a1c3cd
new: [workflow:debugging] Added debugging capability by POSTing workflow exec to an URL
2022-08-03 11:45:17 +02:00
Tom King
de351faaac
new: [internal] Add option to log last API request
2022-08-01 15:02:49 +01:00
Sami Mokaddem
d83039f520
fix: [appmodel] Fixed merged conflict and typo in sql query
2022-08-01 10:19:38 +02:00
Sami Mokaddem
151d23a8c2
Merge branch 'develop' of github.com:MISP/MISP into feature-workflows-2
2022-08-01 09:59:44 +02:00
iglocska
3ce754d595
Merge branch 'develop' into new_correlations
2022-07-31 23:58:40 +02:00
iglocska
3a4289d631
new: [correlation] engine rewrite
...
- allow for multiple concurrent engines
- default: similar behaviour as before, ACL enforced
- No ACL: for endpoint misps, disable the enforcement of ACL for correlations altogether
- rework:
- correlation entries are fully indexed reference tables
- values are now stored separately
- built in protection against overcorrelating values (defaults to 20 max)
- 1 way correlations to cut the size in half
- unsigned IDs to double the ID space
- loads of performance improvements
- fix to the broken event index with correlation counts enabled
- UI improvements
- search for values from the correlation column directly (in case there are non-correlating versions of the same value)
- added correlations to the attribute search/index
- TODO:
- upgrade scripts
2022-07-31 23:48:38 +02:00
Sami Mokaddem
a6f676a66c
chg: [workflow] Simplified call to executeTrigger
2022-07-27 10:43:09 +02:00
Sami Mokaddem
5e29f975d1
chg: [workflow] Enable some module by default when updating
2022-07-26 11:16:08 +02:00
Sami Mokaddem
2be9d07502
chg: [workflowBlueprint] Added new column `default` to recognize default BP
2022-07-19 11:48:12 +02:00
Sami Mokaddem
87b92109d2
chg: [workflow] Various fixes, improved enrichment support and new logic module
2022-07-15 11:47:08 +02:00
Sami Mokaddem
2a31408555
fix: [workflow] Create table sql statement updated
2022-07-14 14:07:35 +02:00
Sami Mokaddem
99a76812bc
Merge remote-tracking branch 'origin/develop' into feature-workflows-2
2022-07-07 09:09:39 +02:00
Sami Mokaddem
dbcedbc505
chg: [workflow] Various improvement and added support of `enrichment_before_query` trigger
2022-07-05 11:19:18 +02:00
Jakub Onderka
640a732c29
Merge pull request #8358 from JakubOnderka/memory-leak-fix
...
fix: [internal] PHP memory leak
2022-06-29 13:32:26 +02:00
Sami Mokaddem
8b8055a625
chg: [WorkflowBlueprint] Renamed WorkflowPart into WorkflowBlueprint
2022-06-24 17:09:49 +02:00
Sami Mokaddem
351a3cfd4e
chg: [workflow] non-blocking workflows are run by background workers
2022-06-24 12:20:03 +02:00
Sami Mokaddem
a13c1a39e9
chg: [workflow] Improved logging capabilities and stop aborting execution if non-blocking module return false
2022-06-24 11:22:26 +02:00
Sami Mokaddem
df8f0f9ed0
new: [workflow_module:webhook] Added new webhook module
2022-06-22 13:38:19 +02:00
Sami Mokaddem
4e47782a04
new: [workflowPart] Started integration of workflow parts
2022-06-22 09:46:57 +02:00
Sami Mokaddem
07b99a1a5a
chg: [workflow] Added more logging while executing WF
2022-06-13 14:36:44 +02:00
Jakub Onderka
2a2b52f64e
chg: [internal] Convert to const
2022-06-08 10:44:33 +02:00
Sami Mokaddem
d8f8225b9e
Merge branch 'develop' of github.com:MISP/MISP into feature-workflows
2022-05-30 14:37:36 +02:00
Sami Mokaddem
9e29830dfa
chg: [workflow] Better module loading and execution errors get propagated to the caller for blocking path
2022-05-30 14:34:45 +02:00
Jakub Onderka
6a4facf8ed
chg: [internal] Faster Model::isUnique and Model::exists method
2022-05-29 16:21:25 +02:00
Jakub Onderka
715ca6d1e2
fix: [internal] PHP memory leak
2022-05-22 18:19:45 +02:00
Sami Mokaddem
ce9b823cbb
chg: [workflows:trigger] Added support of execution order for blocking triggers
2022-05-19 12:02:35 +02:00
Luciano Righetti
217be89e00
new: add MysqlExtended DboSource to support index query hints
2022-05-19 11:08:31 +02:00
Sami Mokaddem
62e53f81f6
chg: [workflow] Started working on priority_level and trigger list - WiP
2022-05-16 14:44:54 +02:00
Sami Mokaddem
c5af331bc5
Merge branch 'develop' of github.com:MISP/MISP into feature-workflows
2022-05-16 10:50:47 +02:00
Sami Mokaddem
62107f14b9
chg: [workflows] Added enabled state
2022-05-16 10:47:55 +02:00
Jakub Onderka
2f7c671adb
new: [internal] Simplify checking if connection is MySQL/MariaDB
2022-05-14 10:17:06 +02:00
Sami Mokaddem
c477935752
Merge branch 'develop' of github.com:MISP/MISP into feature-workflows
2022-05-11 13:03:03 +02:00
Luciano Righetti
53b0e4918b
Merge pull request #8331 from righel/add-attr-timestamp-index
...
chg: add index for attributes.timestamp, show index diagnostics in di…
2022-05-10 10:33:06 +02:00
Jakub Onderka
2294232442
fix: [internal] Strict types
2022-05-07 10:40:41 +02:00
Luciano Righetti
a6e71bd649
fix: add missing break
2022-05-04 09:22:01 +02:00
Luciano Righetti
5fbb80d04d
chg: add index for attributes.timestamp, show index diagnostics in diagnostics page
2022-05-04 09:04:30 +02:00
Sami Mokaddem
6670333f10
chg: [workflow] Added database migration
2022-05-04 00:00:19 +02:00
Tom King
5de04ecffa
chg: Merge from develop
2022-04-20 08:07:36 +01:00
iglocska
be9fb9e802
Merge branch 'develop' of github.com:MISP/MISP into develop
2022-04-19 15:18:40 +02:00
iglocska
2688961cc3
Merge branch 'develop' of github.com:MISP/MISP into develop
2022-04-18 17:59:34 +02:00
iglocska
ccbd119417
chg: [internal] setupSyncRequest made public
...
- to be able to access it via shell scripts
2022-04-18 17:57:38 +02:00
iglocska
93821c0de6
fix: [security] Sanitise paths for several file interactions
...
- remove :// anywhere we don't expect a protocol to be supplied
- remove phar:// in certauth plugin's fetcher
- as reported by Dawid Czarnecki of Zigrin Security
2022-04-17 18:25:51 +02:00
iglocska
0108f1bde2
fix: [security] unregister phar from stream wrappers globally for all Model code
...
- blanket protection against phar deserialization vulnerabilities
- as reported by Dawid Czarnecki of Zigrin Security
2022-04-17 16:30:14 +02:00
Jakub Onderka
9e8608b3d3
chg: [jsonTool] Properly handle invalid JSON for PHP 7.2 and older
2022-04-10 09:38:25 +02:00
Jakub Onderka
9d8fc81678
chg: [internal] Throw exception if Redis class not found
2022-03-19 12:31:55 +01:00
Sami Mokaddem
d65ef9c966
chg: [cryptographicKeys] Indexed more column and bumped db_schema
2022-03-17 14:43:01 +01:00
iglocska
b80a7af2c3
new: [cryptographic keys] model and controllers added
...
- sets MISP up for information signing
- sign data during synchronisation
2022-03-13 12:35:12 +01:00
iglocska
870b9d761b
fix: [db schema] fixed
2022-03-04 16:10:12 +01:00
iglocska
7834ec3760
fix: [organisations] made meta fields default to '' and not allow null values
...
- fixes a filtering issue with sharing group blueprints leading to sharing groups that are more restrictive than expected
2022-03-02 17:32:35 +01:00
iglocska
639a4929e3
new: [sharing group blueprints]
...
- create a rule based blueprint that is used to create and update a sharing group
- nest sharing groups
- filter organisations by metadata fields
- nested via boolean operators
- CLI exposed
- API exposed
- Lightweight ownership model (only blueprint owner can see and edit the blueprint)
2022-03-02 02:09:20 +01:00
Tom King
5f618ed0ca
chg: Update from upstream develop
2022-02-17 10:41:58 +00:00
Jakub Onderka
bd9cecbc2c
chg: [internal] Use ProcessTool in Sighting
2021-11-22 09:57:39 +01:00
Jakub Onderka
0cc7804219
chg: [internal] Current branch and commit checking
2021-11-14 19:09:38 +01:00
Jakub Onderka
a424c01844
chg: [internal] Small optim
2021-11-14 19:09:38 +01:00
Jakub Onderka
db3183ae54
chg: [internal] Faster way how to get current commit
2021-11-14 19:09:38 +01:00
Jakub Onderka
553036de94
Merge pull request #7955 from JakubOnderka/code-style-background-jobs
...
chg: [internal] Code style
2021-11-12 11:48:28 +01:00
Jakub Onderka
fb99021ab9
chg: [internal] Code style
2021-11-12 09:44:21 +01:00
Jakub Onderka
6806cdf574
new: [user] Add sub field for user
2021-11-11 16:34:27 +01:00
Luciano Righetti
ba0399a6fb
chg: remove MISP.use_simple_background_jobs setting in favor of SimpleBackgroundJobs.enabled
2021-11-10 16:44:02 +01:00
Luciano Righetti
9e23a2744f
chg: merge develop, fix conflicts
2021-11-09 14:23:51 +01:00
Luciano Righetti
4218787934
chg: move initTool() logic to constructor
2021-11-09 08:43:39 +01:00
Jakub Onderka
5c5f2b54ca
Merge pull request #7938 from JakubOnderka/authkeys-encrypted-vol2
...
Authkeys encrypted vol2
2021-11-09 08:36:44 +01:00
Luciano Righetti
96a46b6899
chg: merge develop, fix conflicts.
2021-11-08 14:37:27 +01:00
iglocska
dc925a321f
Merge branch 'develop' of github.com:MISP/MISP into develop
2021-11-08 14:17:01 +01:00
iglocska
952a98dba8
Merge branch 'local_only' into develop
2021-11-08 14:15:18 +01:00
Luciano Righetti
f2c9d12eae
chg: merge develop, fix conflicts.
2021-11-08 11:35:20 +01:00
Jakub Onderka
5569d7d2bf
new: [security] Store authkeys for servers encrypted
2021-11-07 17:51:04 +01:00
Jakub Onderka
3459a09943
chg: [internal] App model cleanup
2021-11-07 10:40:01 +01:00
Jakub Onderka
c9b8564c0a
Merge pull request #7927 from JakubOnderka/system-setting
...
System setting in database
2021-11-06 16:15:23 +01:00
Luciano Righetti
f80fcec2db
chg: rename settings
2021-11-05 15:46:42 +01:00
Jakub Onderka
c9597ba3e9
new: Store system settings in database
2021-11-05 09:19:56 +01:00
Luciano Righetti
309f7ed50e
fix: wrong update query
2021-11-04 16:51:56 +01:00
Luciano Righetti
5242d3204d
chg: rename conf name
2021-11-04 11:39:24 +01:00
iglocska
a0e8acc45f
Merge branch 'develop' of github.com:MISP/MISP into develop
2021-11-04 11:05:32 +01:00
iglocska
eddf95355a
fix: [database] upgrade script using mb4 defaulted to 255 key length
...
- default should be 191
2021-11-04 11:04:56 +01:00
Luciano Righetti
946bdb68e6
chg: add db update
2021-11-03 17:09:42 +01:00
Luciano Righetti
b1eaed2b1b
chg: merge develop, fix conflicts
2021-11-03 13:54:20 +01:00
Luciano Righetti
453418d6a5
fix: use class registry to get job class
2021-11-02 16:53:42 +01:00
Luciano Righetti
cce5e581d4
fix: add missing jobId param to enqueue() calls
2021-11-02 16:25:43 +01:00
Luciano Righetti
ae3300d5d0
chg: use new bg jobs tool in AppModel
2021-11-02 11:55:12 +01:00
Jakub Onderka
f1283996ad
chg: [internal] Optimise AuditLog
2021-10-31 10:52:47 +01:00
Jakub Onderka
9ae7d88f23
chg: [internal] Optimise Tag::findTagIdsByTagNames
2021-10-30 19:43:17 +02:00
Jakub Onderka
94808488ce
Merge pull request #7852 from JakubOnderka/optimise-sighting
...
chg: [internal] Optimise sightings
2021-10-29 21:25:42 +02:00
Jakub Onderka
0d8292289d
chg: [internal] Optimise afterFind and simplify ISODatetimeToUTC
2021-10-28 13:56:27 +02:00
Jakub Onderka
6ade27a27c
chg: [internal] Move attribute validation to different tool
2021-10-28 09:42:04 +02:00
Loïc Fortemps
dc71f6e451
Merge branch 'develop' into local_tags
2021-10-27 17:06:02 +02:00
Loic Fortemps
a1ba1c053e
Adding a local_only option for Tags and Galaxies
2021-10-27 16:51:54 +02:00
Luciano Righetti
5e19ca6761
chg: refactor all background job calls from event model and controller to use new tool
2021-10-27 11:33:32 +02:00
Luciano Righetti
35cf34fab2
Merge branch 'develop' into add_simple_background_jobs
2021-10-26 10:55:13 +02:00
Jakub Onderka
006f603d76
chg: [schema] Add index for object_references.event_id column
2021-10-24 16:01:22 +02:00
Jakub Onderka
fd0faf53d7
Merge pull request #7884 from JakubOnderka/faster-notifications
...
chg: [internal] Simplified notifications loading
2021-10-23 20:12:28 +02:00
Jakub Onderka
4997af0aa2
chg: [internal] Disable order for hasAny method
2021-10-23 19:54:38 +02:00
Jakub Onderka
0d78d0f994
chg: [internal] Simplified notifications loading
2021-10-23 19:50:38 +02:00
Jakub Onderka
958bfe7383
fix: [schema] Modify User.change_pw column to boolean
2021-10-23 19:22:45 +02:00
Luciano Righetti
f3d05e6ff8
add: add initial new simple background jobs.
2021-10-21 13:39:26 +02:00
Tom King
4e74a165ed
Merge remote-tracking branch 'upstream/develop' into feature/propagate_tag_deletion
2021-10-19 09:44:41 +01:00
Jakub Onderka
2ed41a0964
new: [oidc] User setting for oidc metadata
2021-10-18 10:08:18 +02:00
Jakub Onderka
6fe3a895c8
chg: [internal] Simplify validation
2021-10-18 09:04:06 +02:00
Jakub Onderka
1ffdfed0b1
chg: [internal] Optimise validators
2021-10-18 09:04:06 +02:00
Jakub Onderka
a1c22e9fe5
chg: [internal] Remove unused validation rule
2021-10-18 09:04:06 +02:00
Jakub Onderka
f3f50c14fd
chg: [internal] Remove unused method
2021-10-18 09:04:06 +02:00
Jakub Onderka
e1895cba81
chg: [internal] Optimise sightings
2021-10-17 16:17:17 +02:00
Jakub Onderka
28afe209ca
chg: [internal] Fix setting cleanDb admin setting
2021-10-16 19:32:32 +02:00
Jakub Onderka
d45ac63bf9
chg: [internal] Do less work when checking if db is updated
2021-10-16 09:58:05 +02:00
Jakub Onderka
a3d1ff8762
chg: [internal] Another bunch of event filter optim
2021-10-15 15:54:31 +02:00
Jakub Onderka
9d84b6923d
Merge pull request #7835 from JakubOnderka/stix-export
...
chg: [internal] Simplified loading python bin
2021-10-13 17:27:02 +02:00
Jakub Onderka
2b31ada0a5
chg: [stix-export] Simplified loading python bin
2021-10-13 09:57:25 +02:00
Jakub Onderka
9610fb1e93
chg: [internal] Use FileAccessTool for Feed::unzipFirstFile
2021-10-12 14:32:50 +02:00
Jakub Onderka
d9e89955bc
chg: [internal] Simplify fetching Kafka topic
2021-10-11 09:41:46 +02:00
Jakub Onderka
b3591f2d72
chg: [internal] Faster event tag attaching
2021-10-05 20:24:02 +02:00
Jakub Onderka
544c9ee8b9
new: [internal] Store MISP live status also in Redis
2021-10-04 10:07:32 +02:00
Jakub Onderka
fdf6f5ca9e
chg: [internal] Cleanup AdminShell::{updateJSON,runUpdates}
2021-09-26 21:52:18 +02:00
Jakub Onderka
da1818c3cc
fix: [internal] Fetching filter rules
2021-08-30 17:51:17 +02:00
Jakub Onderka
98b87d8987
chg: [security] Use const hasher also for login
2021-08-24 21:45:37 +02:00
Jakub Onderka
439ec5cceb
fix: [internal] Code cleanup
2021-08-23 17:03:44 +02:00
Jakub Onderka
3c3cee7735
Merge pull request #7659 from JakubOnderka/unique-indexes
...
chg: [schema] Mark more indexes as unique
2021-08-20 09:22:23 +02:00
Jakub Onderka
1cd09dd919
chg: [schema] Mark more indexes as unique
2021-08-13 13:59:47 +02:00
Jakub Onderka
c7f3be0929
chg: [sync] Remove `commit` and MISP-version from HTTP header
2021-08-11 17:02:55 +02:00
Jakub Onderka
da195589df
chg: [internal] Convert PubSubTool to static
2021-08-11 12:07:32 +02:00
Tom King
370176a037
Merge remote-tracking branch 'upstream/develop' into feature/propagate_tag_deletion
2021-08-11 10:56:30 +01:00
iglocska
4739e1f234
Merge branch '2.4' into develop
2021-08-02 13:31:19 +02:00
iglocska
269883c4ef
fix: [afterHook] for setting changes wasn't returning true, fixes 7477
...
- this caused the CLI setting change to error out
2021-08-02 13:30:29 +02:00
Jakub Onderka
db242d6901
chg: [internal] Convert array to const
2021-07-27 18:49:40 +02:00
Jakub Onderka
15e3cc799c
chg: [internal] Removed unused variables
2021-07-27 18:43:16 +02:00
Jakub Onderka
525339e333
chg: [internal] Remove unused method
2021-07-21 17:57:25 +02:00
Jakub Onderka
5f48fee84a
chg: [internal] Move variable from AppModel to Server model
2021-07-21 17:53:47 +02:00
Jakub Onderka
b62c95389c
Merge pull request #7157 from JakubOnderka/sighting-push-filtering
...
new: [sync] Method for filtering out existing sightings
2021-07-09 12:19:32 +02:00
Tom King
297eb9baad
Merge remote-tracking branch 'upstream/2.4' into feature/propagate_tag_deletion
2021-07-07 08:49:13 +01:00