Commit Graph

11 Commits (71f8e125e5eb5c853a9a121beda70b8979ca3e1f)

Author SHA1 Message Date
iglocska 2252d16c91
fix: [over-correlations] weren't truly case insensitive, causing potential issues matching and entering values
- wrapped adding a new value in a try catch, no need to make synchronisations fail over this
- added case insensitive change to values on entry (table should be all lower-case)
- added update script to lowercase existing values
2023-05-22 11:26:05 +02:00
Jakub Onderka 332264fe9d fix: [correlation] Fix over correlating value 2022-10-21 14:37:37 +02:00
Jakub Onderka 9770ee17a9 new: [correlation] Do not correlate over correlating value again for full correlation
Should help with #8685
2022-10-20 09:55:39 +02:00
Jakub Onderka 2a594aa66e fix: [correlation] Smarter count OverCorrelating values 2022-09-13 16:13:51 +02:00
Jakub Onderka a52b8bba7f chg: [internal] Code cleanup 2022-09-08 15:37:36 +02:00
Jakub Onderka c0017a0531 chg: [correlation] Do not delete over correlation if no correlation found 2022-09-08 12:00:02 +02:00
Jakub Onderka e8ea9877a3 chg: [correlations] Optimise fetching limit 2022-09-08 09:44:25 +02:00
Sami Mokaddem 962754dd3b
chg: [overCorrelatingValue] Truncated the `value` column
- We keep the unique constraint on the table
- Correlating values over the max. allowed size are truncated to fit the size requirement. That means large correlating values might be marked as over-correlating despite the fact they are not (as only the starting portion of the value is evaluated).
2022-08-29 10:50:59 +02:00
Jakub Onderka 986e109f76 fix: [internal] Advanced correlations 2022-08-11 12:55:21 +02:00
iglocska d3d042cf2a
new: [overcorrelations] quality of life improvements
- Added new tool to generate occurrence counts (real numbers this time)
- Added hook to truncate over correlating value table on recorrelation
- No longer store the partial counts as occurrences when generating correlations
2022-08-10 14:17:20 +02:00
iglocska 3a4289d631
new: [correlation] engine rewrite
- allow for multiple concurrent engines
  - default: similar behaviour as before, ACL enforced
  - No ACL: for endpoint misps, disable the enforcement of ACL for correlations altogether

- rework:
  - correlation entries are fully indexed reference tables
  - values are now stored separately
  - built in protection against overcorrelating values (defaults to 20 max)
  - 1 way correlations to cut the size in half
  - unsigned IDs to double the ID space
  - loads of performance improvements
  - fix to the broken event index with correlation counts enabled

- UI improvements
  - search for values from the correlation column directly (in case there are non-correlating versions of the same value)
  - added correlations to the attribute search/index

- TODO:
  - upgrade scripts
2022-07-31 23:48:38 +02:00