MISP
/
MISP
mirror of https://github.com/MISP/MISP
2
2
Fork 0
Code Issues Releases Wiki Activity
25,787 Commits
34 Branches
365 Tags
189 MiB
Commit Graph

80 Commits (2.4)

Author SHA1 Message Date
iglocska 4c49a57b8b
chg: [recorrelation] added new functionality to set the recorrelation chunk size 
- recorrelate in configurable chunk sizes (rather than the old hard coded value of 500)
- immediately execute the saving of correlations after each chunk (should drastically reduce memory usage for massive events)
 2024-06-07 16:02:16 +02:00
Jakub Onderka 67eeb9df9d chg: [internal] Use Attribute::fetchAttributesInChunks for correlations 2024-01-28 22:12:17 +01:00
iglocska 46dded9ea4
fix: [overorrelation] truncation should only happen on full recorrelations 2023-11-22 09:44:15 +01:00
Luciano Righetti 51e5df1ca7 fix: method call on null 2023-09-11 12:51:09 +02:00
Luciano Righetti 53e16ce752 fix: do not require jobId for AdminShell jobGenerateCorrelation, create a new job if jobId is null. fixes #9206 2023-09-11 12:35:41 +02:00
Jakub Onderka f33b6e8231
Merge pull request #8603 from JakubOnderka/code-fixes 
News view
 2022-11-14 10:10:50 +01:00
Jakub Onderka a6fbc5dc6c fix: [correlation] Delete correlations when deleting event 2022-10-23 18:35:56 +02:00
Jakub Onderka 7615123738 chg: [correlation] Do not try to unblock value if doing full correlation 2022-10-21 19:16:47 +02:00
Jakub Onderka cf8a642ba7 chg: [correlation] Check attribute ID in SQL request 2022-10-21 16:29:32 +02:00
Jakub Onderka 2379f94342 chg: [correlation] Move fetching object later in code 2022-10-21 16:29:32 +02:00
Jakub Onderka 9770ee17a9 new: [correlation] Do not correlate over correlating value again for full correlation 
Should help with #8685
 2022-10-20 09:55:39 +02:00
Jakub Onderka de4ba342e0 chg: [correlation] Optimise saving object timestamp 2022-10-13 09:51:12 +02:00
Jakub Onderka 196c87963d fix: [correlations] Prevent Trying to access array offset on value of type null error 2022-10-10 17:18:11 +02:00
Jakub Onderka 5465bd8bd0 chg: [internal] Cleanup Redis code 2022-10-08 18:18:25 +02:00
Jakub Onderka 38c4a6ee06 fix: [correlations] NoAclCorrelation works again even for object attributes 2022-10-05 15:32:40 +02:00
Jakub Onderka 341ac2d0a8 chg: [internal] New method RedisTool::unlink 2022-10-01 09:48:13 +02:00
Jakub Onderka f10bdb8c67 chg: [correlations] Attach correlation exclusion just for correlating attributes 2022-09-21 14:23:38 +02:00
Jakub Onderka a3b02cf037 fix: [internal] Code style 2022-09-13 16:13:51 +02:00
Jakub Onderka a52b8bba7f chg: [internal] Code cleanup 2022-09-08 15:37:36 +02:00
Jakub Onderka c0017a0531 chg: [correlation] Do not delete over correlation if no correlation found 2022-09-08 12:00:02 +02:00
Jakub Onderka 02a4c51406 fix: [correlation] Fix correlation skipping when doing full correlation 2022-09-08 09:45:02 +02:00
Jakub Onderka 1daab04ce2 chg: [correlation] Faster saving correlations 2022-09-08 09:30:45 +02:00
Sami Mokaddem 962754dd3b
chg: [overCorrelatingValue] Truncated the `value` column 
- We keep the unique constraint on the table
- Correlating values over the max. allowed size are truncated to fit the size requirement. That means large correlating values might be marked as over-correlating despite the fact they are not (as only the starting portion of the value is evaluated).
 2022-08-29 10:50:59 +02:00
Jakub Onderka 48c8a7eab3 chg: [internal] Code cleanup 2022-08-11 12:55:23 +02:00
Jakub Onderka 14501e8a78 chg: [internal] Use less SQL queries for event fetching 2022-08-11 12:55:23 +02:00
Jakub Onderka 986e109f76 fix: [internal] Advanced correlations 2022-08-11 12:55:21 +02:00
Jakub Onderka b53d8c828d chg: [internal] Cleanup code for new correlation engine 2022-08-11 12:53:14 +02:00
iglocska d3d042cf2a
new: [overcorrelations] quality of life improvements 
- Added new tool to generate occurrence counts (real numbers this time)
- Added hook to truncate over correlating value table on recorrelation
- No longer store the partial counts as occurrences when generating correlations
 2022-08-10 14:17:20 +02:00
iglocska 097ad78d54
chg: [correlations] performance tuning 
- added a new constraint to check for correlation uniqueness (a_id, 1_a_id, value_id)
- stopped dropping correlations on a full recorrelation
- only correlate "upwards" towards higher IDs
 2022-08-09 14:41:59 +02:00
iglocska b34a6d670a
fix: [compatibility] removed function return types from correlations 2022-08-08 16:02:50 +02:00
iglocska 9a6dc85a96
chg: [compatibility] for PHP < 7.2 for an organisation that shall go unnamed 2022-08-08 15:59:59 +02:00
Sami Mokaddem 5bed18614b
fix: [correlation:getRelatedAttribute] Always return an array 2022-08-04 14:51:37 +02:00
Sami Mokaddem c215c47e18
chg: [correlation:getRelatedAttributes] Updated documentation to reflect returned type 2022-08-04 14:49:48 +02:00
iglocska a04f3964e7
new: [correlation rework] round 2 
- long list of fixes
- update scripts
- correlation engine management interface
- recorrelation/truncation tools
- various performance tunings and bug fixes
 2022-08-03 15:44:27 +02:00
iglocska be67304988
fix: [correlation] fixed missing passed sharing group ID array 2022-08-01 00:43:36 +02:00
iglocska 3ce754d595
Merge branch 'develop' into new_correlations 2022-07-31 23:58:40 +02:00
iglocska 3a4289d631
new: [correlation] engine rewrite 
- allow for multiple concurrent engines
  - default: similar behaviour as before, ACL enforced
  - No ACL: for endpoint misps, disable the enforcement of ACL for correlations altogether

- rework:
  - correlation entries are fully indexed reference tables
  - values are now stored separately
  - built in protection against overcorrelating values (defaults to 20 max)
  - 1 way correlations to cut the size in half
  - unsigned IDs to double the ID space
  - loads of performance improvements
  - fix to the broken event index with correlation counts enabled

- UI improvements
  - search for values from the correlation column directly (in case there are non-correlating versions of the same value)
  - added correlations to the attribute search/index

- TODO:
  - upgrade scripts
 2022-07-31 23:48:38 +02:00
Luciano Righetti dc4236ef9b
new: add LightPaginator when total page count is not needed/to expensive 2022-07-20 15:56:30 +02:00
Jakub Onderka 6b6169458a fix: [internal] Faster changing IDS flag for CIDR 2022-07-07 09:37:37 +02:00
Jakub Onderka 3aa9df3fb8 chg: [internal] Reduce memory usage when generating correlations vol. 8 2022-05-15 09:22:08 +02:00
Jakub Onderka ee2931dbc9 chg: [internal] Reduce memory usage when generating correlations vol. 7 2022-05-15 09:21:00 +02:00
Jakub Onderka 0db2041a6f chg: [internal] Reduce memory usage when generating correlations vol. 6 2022-05-15 09:21:00 +02:00
Jakub Onderka 6e18645c84 chg: [internal] Reduce memory usage when generating correlations vol. 5 2022-05-15 09:21:00 +02:00
Jakub Onderka 5b3aa10fc8 chg: [internal] Reduce memory usage when generating correlations vol. 4 2022-05-15 09:21:00 +02:00
Jakub Onderka 9735b2a5fc fix: [internal] Reduce number of fetched attributes 2022-05-15 09:21:00 +02:00
Jakub Onderka ab728e102a chg: [internal] Reduce memory usage when generating correlations vol. 3 2022-05-15 09:21:00 +02:00
Jakub Onderka a0309a36c8 chg: [internal] Cache CIDR ranges in PHP array to speedup correlations 2022-05-15 09:21:00 +02:00
Jakub Onderka 65a1e4db25 chg: [internal] Reduce memory usage when generating correlations 2022-05-15 09:21:00 +02:00
Jakub Onderka e2e769e36a fix: [correlation] Remove unused fields from event query 2022-05-15 09:21:00 +02:00
Jakub Onderka 3ccf211bae fix: [correlation] MISP.enable_advanced_correlations config option works again 2022-05-15 09:21:00 +02:00