Jakub Onderka
02a4c51406
fix: [correlation] Fix correlation skipping when doing full correlation
2022-09-08 09:45:02 +02:00
Jakub Onderka
1daab04ce2
chg: [correlation] Faster saving correlations
2022-09-08 09:30:45 +02:00
Sami Mokaddem
962754dd3b
chg: [overCorrelatingValue] Truncated the `value` column
...
- We keep the unique constraint on the table
- Correlating values over the max. allowed size are truncated to fit the size requirement. That means large correlating values might be marked as over-correlating despite the fact they are not (as only the starting portion of the value is evaluated).
2022-08-29 10:50:59 +02:00
Jakub Onderka
48c8a7eab3
chg: [internal] Code cleanup
2022-08-11 12:55:23 +02:00
Jakub Onderka
14501e8a78
chg: [internal] Use less SQL queries for event fetching
2022-08-11 12:55:23 +02:00
Jakub Onderka
986e109f76
fix: [internal] Advanced correlations
2022-08-11 12:55:21 +02:00
Jakub Onderka
b53d8c828d
chg: [internal] Cleanup code for new correlation engine
2022-08-11 12:53:14 +02:00
iglocska
d3d042cf2a
new: [overcorrelations] quality of life improvements
...
- Added new tool to generate occurrence counts (real numbers this time)
- Added hook to truncate over correlating value table on recorrelation
- No longer store the partial counts as occurrences when generating correlations
2022-08-10 14:17:20 +02:00
iglocska
097ad78d54
chg: [correlations] performance tuning
...
- added a new constraint to check for correlation uniqueness (a_id, 1_a_id, value_id)
- stopped dropping correlations on a full recorrelation
- only correlate "upwards" towards higher IDs
2022-08-09 14:41:59 +02:00
iglocska
b34a6d670a
fix: [compatibility] removed function return types from correlations
2022-08-08 16:02:50 +02:00
iglocska
9a6dc85a96
chg: [compatibility] for PHP < 7.2 for an organisation that shall go unnamed
2022-08-08 15:59:59 +02:00
Sami Mokaddem
5bed18614b
fix: [correlation:getRelatedAttribute] Always return an array
2022-08-04 14:51:37 +02:00
Sami Mokaddem
c215c47e18
chg: [correlation:getRelatedAttributes] Updated documentation to reflect returned type
2022-08-04 14:49:48 +02:00
iglocska
a04f3964e7
new: [correlation rework] round 2
...
- long list of fixes
- update scripts
- correlation engine management interface
- recorrelation/truncation tools
- various performance tunings and bug fixes
2022-08-03 15:44:27 +02:00
iglocska
be67304988
fix: [correlation] fixed missing passed sharing group ID array
2022-08-01 00:43:36 +02:00
iglocska
3ce754d595
Merge branch 'develop' into new_correlations
2022-07-31 23:58:40 +02:00
iglocska
3a4289d631
new: [correlation] engine rewrite
...
- allow for multiple concurrent engines
- default: similar behaviour as before, ACL enforced
- No ACL: for endpoint misps, disable the enforcement of ACL for correlations altogether
- rework:
- correlation entries are fully indexed reference tables
- values are now stored separately
- built in protection against overcorrelating values (defaults to 20 max)
- 1 way correlations to cut the size in half
- unsigned IDs to double the ID space
- loads of performance improvements
- fix to the broken event index with correlation counts enabled
- UI improvements
- search for values from the correlation column directly (in case there are non-correlating versions of the same value)
- added correlations to the attribute search/index
- TODO:
- upgrade scripts
2022-07-31 23:48:38 +02:00
Luciano Righetti
dc4236ef9b
new: add LightPaginator when total page count is not needed/to expensive
2022-07-20 15:56:30 +02:00
Jakub Onderka
6b6169458a
fix: [internal] Faster changing IDS flag for CIDR
2022-07-07 09:37:37 +02:00
Jakub Onderka
3aa9df3fb8
chg: [internal] Reduce memory usage when generating correlations vol. 8
2022-05-15 09:22:08 +02:00
Jakub Onderka
ee2931dbc9
chg: [internal] Reduce memory usage when generating correlations vol. 7
2022-05-15 09:21:00 +02:00
Jakub Onderka
0db2041a6f
chg: [internal] Reduce memory usage when generating correlations vol. 6
2022-05-15 09:21:00 +02:00
Jakub Onderka
6e18645c84
chg: [internal] Reduce memory usage when generating correlations vol. 5
2022-05-15 09:21:00 +02:00
Jakub Onderka
5b3aa10fc8
chg: [internal] Reduce memory usage when generating correlations vol. 4
2022-05-15 09:21:00 +02:00
Jakub Onderka
9735b2a5fc
fix: [internal] Reduce number of fetched attributes
2022-05-15 09:21:00 +02:00
Jakub Onderka
ab728e102a
chg: [internal] Reduce memory usage when generating correlations vol. 3
2022-05-15 09:21:00 +02:00
Jakub Onderka
a0309a36c8
chg: [internal] Cache CIDR ranges in PHP array to speedup correlations
2022-05-15 09:21:00 +02:00
Jakub Onderka
65a1e4db25
chg: [internal] Reduce memory usage when generating correlations
2022-05-15 09:21:00 +02:00
Jakub Onderka
e2e769e36a
fix: [correlation] Remove unused fields from event query
2022-05-15 09:21:00 +02:00
Jakub Onderka
3ccf211bae
fix: [correlation] MISP.enable_advanced_correlations config option works again
2022-05-15 09:21:00 +02:00
Jakub Onderka
a82de04e8c
chg: [correlation] Reduce memory usage when generating all correlations
2022-05-15 09:20:43 +02:00
Jakub Onderka
6e4e4ab850
chg: [correlation] Code cleanup
2022-05-15 09:20:43 +02:00
Jakub Onderka
2aa2b7c79b
fix: [correlation] Avoid duplicate correlation
2022-05-15 09:20:43 +02:00
Jakub Onderka
14ff781b5b
chg: [correlation exclusion] Check both part of attribute
2022-05-15 09:20:43 +02:00
Jakub Onderka
fc17263401
fix: [internal] Better check which value correlated
2022-05-15 09:20:43 +02:00
Jakub Onderka
2f7c671adb
new: [internal] Simplify checking if connection is MySQL/MariaDB
2022-05-14 10:17:06 +02:00
Jakub Onderka
316df3a96c
fix: [internal] Generate top correlations for all values
2022-05-09 14:59:07 +02:00
Jakub Onderka
f2580dda7a
chg: [internal] Top correlations cleanup
2022-05-09 14:34:38 +02:00
Luciano Righetti
cce5e581d4
fix: add missing jobId param to enqueue() calls
2021-11-02 16:25:43 +01:00
Luciano Righetti
4d43789311
chg: use new bg jobs tool in correlation model
2021-11-02 15:35:23 +01:00
Jakub Onderka
497910b941
fix: [internal] Remove ssdeep data when deleting attribute
2021-09-03 16:38:49 +02:00
Jakub Onderka
e55f8171d9
chg: [internal] Faster calls
2021-09-03 16:20:14 +02:00
Jakub Onderka
cd0a2bdf4b
chg: [internal] Faster deleting correlation when deleting attribute
2021-09-03 12:22:16 +02:00
Jakub Onderka
2c2d879996
chg: [internal] Use object variable and not Configure again and again
2021-09-03 11:58:02 +02:00
Jakub Onderka
645fb52b64
chg: [internal] Do not fetch 'Event.disable_correlation' field
2021-09-03 11:54:28 +02:00
Jakub Onderka
6642d8bf7b
chg: [correlation] Allow to drop Correlation.{date,info} columns
2021-08-30 09:43:47 +02:00
Jakub Onderka
14b78904ba
chg: [internal] Do not load exclusion list from Redis again and again
2021-08-12 15:05:39 +02:00
Jakub Onderka
d4c2d10fe3
chg: [internal] Convert array to const
2021-07-27 16:43:27 +02:00
Jakub Onderka
6440133989
chg: [internal] Convert variable to const
2021-07-21 09:32:51 +02:00
Jakub Onderka
99070b73ef
fix: [correlations] Correctly handle exclusion
2021-05-10 08:30:56 +02:00