Jakub Onderka
67eeb9df9d
chg: [internal] Use Attribute::fetchAttributesInChunks for correlations
2024-01-28 22:12:17 +01:00
iglocska
46dded9ea4
fix: [overorrelation] truncation should only happen on full recorrelations
2023-11-22 09:44:15 +01:00
Luciano Righetti
51e5df1ca7
fix: method call on null
2023-09-11 12:51:09 +02:00
Luciano Righetti
53e16ce752
fix: do not require jobId for AdminShell jobGenerateCorrelation, create a new job if jobId is null. fixes #9206
2023-09-11 12:35:41 +02:00
Jakub Onderka
f33b6e8231
Merge pull request #8603 from JakubOnderka/code-fixes
...
News view
2022-11-14 10:10:50 +01:00
Jakub Onderka
a6fbc5dc6c
fix: [correlation] Delete correlations when deleting event
2022-10-23 18:35:56 +02:00
Jakub Onderka
7615123738
chg: [correlation] Do not try to unblock value if doing full correlation
2022-10-21 19:16:47 +02:00
Jakub Onderka
cf8a642ba7
chg: [correlation] Check attribute ID in SQL request
2022-10-21 16:29:32 +02:00
Jakub Onderka
2379f94342
chg: [correlation] Move fetching object later in code
2022-10-21 16:29:32 +02:00
Jakub Onderka
9770ee17a9
new: [correlation] Do not correlate over correlating value again for full correlation
...
Should help with #8685
2022-10-20 09:55:39 +02:00
Jakub Onderka
de4ba342e0
chg: [correlation] Optimise saving object timestamp
2022-10-13 09:51:12 +02:00
Jakub Onderka
196c87963d
fix: [correlations] Prevent Trying to access array offset on value of type null error
2022-10-10 17:18:11 +02:00
Jakub Onderka
5465bd8bd0
chg: [internal] Cleanup Redis code
2022-10-08 18:18:25 +02:00
Jakub Onderka
38c4a6ee06
fix: [correlations] NoAclCorrelation works again even for object attributes
2022-10-05 15:32:40 +02:00
Jakub Onderka
341ac2d0a8
chg: [internal] New method RedisTool::unlink
2022-10-01 09:48:13 +02:00
Jakub Onderka
f10bdb8c67
chg: [correlations] Attach correlation exclusion just for correlating attributes
2022-09-21 14:23:38 +02:00
Jakub Onderka
a3b02cf037
fix: [internal] Code style
2022-09-13 16:13:51 +02:00
Jakub Onderka
a52b8bba7f
chg: [internal] Code cleanup
2022-09-08 15:37:36 +02:00
Jakub Onderka
c0017a0531
chg: [correlation] Do not delete over correlation if no correlation found
2022-09-08 12:00:02 +02:00
Jakub Onderka
02a4c51406
fix: [correlation] Fix correlation skipping when doing full correlation
2022-09-08 09:45:02 +02:00
Jakub Onderka
1daab04ce2
chg: [correlation] Faster saving correlations
2022-09-08 09:30:45 +02:00
Sami Mokaddem
962754dd3b
chg: [overCorrelatingValue] Truncated the `value` column
...
- We keep the unique constraint on the table
- Correlating values over the max. allowed size are truncated to fit the size requirement. That means large correlating values might be marked as over-correlating despite the fact they are not (as only the starting portion of the value is evaluated).
2022-08-29 10:50:59 +02:00
Jakub Onderka
48c8a7eab3
chg: [internal] Code cleanup
2022-08-11 12:55:23 +02:00
Jakub Onderka
14501e8a78
chg: [internal] Use less SQL queries for event fetching
2022-08-11 12:55:23 +02:00
Jakub Onderka
986e109f76
fix: [internal] Advanced correlations
2022-08-11 12:55:21 +02:00
Jakub Onderka
b53d8c828d
chg: [internal] Cleanup code for new correlation engine
2022-08-11 12:53:14 +02:00
iglocska
d3d042cf2a
new: [overcorrelations] quality of life improvements
...
- Added new tool to generate occurrence counts (real numbers this time)
- Added hook to truncate over correlating value table on recorrelation
- No longer store the partial counts as occurrences when generating correlations
2022-08-10 14:17:20 +02:00
iglocska
097ad78d54
chg: [correlations] performance tuning
...
- added a new constraint to check for correlation uniqueness (a_id, 1_a_id, value_id)
- stopped dropping correlations on a full recorrelation
- only correlate "upwards" towards higher IDs
2022-08-09 14:41:59 +02:00
iglocska
b34a6d670a
fix: [compatibility] removed function return types from correlations
2022-08-08 16:02:50 +02:00
iglocska
9a6dc85a96
chg: [compatibility] for PHP < 7.2 for an organisation that shall go unnamed
2022-08-08 15:59:59 +02:00
Sami Mokaddem
5bed18614b
fix: [correlation:getRelatedAttribute] Always return an array
2022-08-04 14:51:37 +02:00
Sami Mokaddem
c215c47e18
chg: [correlation:getRelatedAttributes] Updated documentation to reflect returned type
2022-08-04 14:49:48 +02:00
iglocska
a04f3964e7
new: [correlation rework] round 2
...
- long list of fixes
- update scripts
- correlation engine management interface
- recorrelation/truncation tools
- various performance tunings and bug fixes
2022-08-03 15:44:27 +02:00
iglocska
be67304988
fix: [correlation] fixed missing passed sharing group ID array
2022-08-01 00:43:36 +02:00
iglocska
3ce754d595
Merge branch 'develop' into new_correlations
2022-07-31 23:58:40 +02:00
iglocska
3a4289d631
new: [correlation] engine rewrite
...
- allow for multiple concurrent engines
- default: similar behaviour as before, ACL enforced
- No ACL: for endpoint misps, disable the enforcement of ACL for correlations altogether
- rework:
- correlation entries are fully indexed reference tables
- values are now stored separately
- built in protection against overcorrelating values (defaults to 20 max)
- 1 way correlations to cut the size in half
- unsigned IDs to double the ID space
- loads of performance improvements
- fix to the broken event index with correlation counts enabled
- UI improvements
- search for values from the correlation column directly (in case there are non-correlating versions of the same value)
- added correlations to the attribute search/index
- TODO:
- upgrade scripts
2022-07-31 23:48:38 +02:00
Luciano Righetti
dc4236ef9b
new: add LightPaginator when total page count is not needed/to expensive
2022-07-20 15:56:30 +02:00
Jakub Onderka
6b6169458a
fix: [internal] Faster changing IDS flag for CIDR
2022-07-07 09:37:37 +02:00
Jakub Onderka
3aa9df3fb8
chg: [internal] Reduce memory usage when generating correlations vol. 8
2022-05-15 09:22:08 +02:00
Jakub Onderka
ee2931dbc9
chg: [internal] Reduce memory usage when generating correlations vol. 7
2022-05-15 09:21:00 +02:00
Jakub Onderka
0db2041a6f
chg: [internal] Reduce memory usage when generating correlations vol. 6
2022-05-15 09:21:00 +02:00
Jakub Onderka
6e18645c84
chg: [internal] Reduce memory usage when generating correlations vol. 5
2022-05-15 09:21:00 +02:00
Jakub Onderka
5b3aa10fc8
chg: [internal] Reduce memory usage when generating correlations vol. 4
2022-05-15 09:21:00 +02:00
Jakub Onderka
9735b2a5fc
fix: [internal] Reduce number of fetched attributes
2022-05-15 09:21:00 +02:00
Jakub Onderka
ab728e102a
chg: [internal] Reduce memory usage when generating correlations vol. 3
2022-05-15 09:21:00 +02:00
Jakub Onderka
a0309a36c8
chg: [internal] Cache CIDR ranges in PHP array to speedup correlations
2022-05-15 09:21:00 +02:00
Jakub Onderka
65a1e4db25
chg: [internal] Reduce memory usage when generating correlations
2022-05-15 09:21:00 +02:00
Jakub Onderka
e2e769e36a
fix: [correlation] Remove unused fields from event query
2022-05-15 09:21:00 +02:00
Jakub Onderka
3ccf211bae
fix: [correlation] MISP.enable_advanced_correlations config option works again
2022-05-15 09:21:00 +02:00
Jakub Onderka
a82de04e8c
chg: [correlation] Reduce memory usage when generating all correlations
2022-05-15 09:20:43 +02:00