MISP/app/Model
iglocska a326cc3506
chg: [performance] fix for events with large numbers of attributes and multiple tags from the same taxonomy
- the taxonomy conflict checks were causing multiple issues:

- non taxonomy tags were counted as a taxonomy with namespace ''
- once we identified a tag pair that could cause a conflict (same taxonomy) we loaded the taxonomy into redis
  - however, in order to see if we already have the taxonomy loaded, we went to redis to do a GET
  - In the case of 1 million attributes with at least 1 tag pair, at the minimum this means 1 million GETs on reddit with an event

- Resolution
  - remove the checks for non taxonomy tags
  - store the identified taxonomies temporarily on the model itself in memory
    - only go to redis when the model doesn't have the taxonomy cached in memory
    - still using the old approach when dealing with multiple small events

- thanks to @github-germ for flagging the issue
2023-06-08 08:16:50 +02:00
..
Behavior chg: [log] Do not audit log when unique_ips field is changed 2023-05-24 09:49:14 +02:00
Datasource/Database chg: [logs] Add SQL queries count to access log 2022-11-13 17:42:40 +01:00
DecayingModelsFormulas fix: [internal] Taxonomy code cleanup 2022-12-15 10:04:45 -05:00
WorkflowModules Merge branch 'develop' of github.com:MISP/MISP into develop 2023-06-03 17:38:13 -04:00
AccessLog.php fix: [AccessLog] MySQL command 2023-03-30 00:04:07 +02:00
AdminSetting.php
Allowedlist.php
AppModel.php chg: [taxii] added collection field to taxii servers 2023-06-07 14:26:14 +02:00
AttachmentScan.php fix: add missing jobId param to enqueue() calls 2021-11-02 16:25:43 +01:00
Attribute.php Merge branch '2.4' into develop 2023-05-31 09:01:30 +02:00
AttributeTag.php fix: [attributeTag:handleTag] Typo in argument positioning 2022-10-06 10:13:55 +02:00
AuditLog.php fix: [remote_ip] respect MISP.log_client_ip_header everywhere fixes #8781 2022-11-28 14:08:11 +01:00
AuthKey.php Revert "Feature/api log and pin ip (#8965)" 2023-03-29 09:53:29 +08:00
Bruteforce.php fix: [auth][log] log correct org/userid with failed login fixes #8807 2022-12-07 00:55:20 +01:00
Cerebrate.php new: [CLI] admin reencrypt command 2021-11-07 17:51:04 +01:00
Community.php fix: [UI] Communities 2022-12-15 10:04:45 -05:00
Correlation.php Merge pull request #8603 from JakubOnderka/code-fixes 2022-11-14 10:10:50 +01:00
CorrelationExclusion.php chg: [internal] Cleanup Redis code 2022-10-08 18:18:25 +02:00
CorrelationValue.php fix: [correlation] Use int type for value_id 2022-10-21 15:41:11 +02:00
CryptographicKey.php chg: [internal] Faster fetching event index 2022-11-01 10:56:31 +01:00
Dashboard.php fix: [dashboard] sort dashboard widgets 2022-12-06 07:14:44 +01:00
DecayingModel.php chg: [internal] Add decaying model cache 2022-09-21 09:43:48 +02:00
DecayingModelMapping.php
Event.php Merge branch 'misp-stix' of github.com:MISP/MISP into develop 2023-04-26 14:01:03 +02:00
EventBlocklist.php
EventDelegation.php
EventGraph.php fix: [acl] Event graph 2022-10-26 13:39:30 +02:00
EventLock.php new: [redis] Store some data in Redis compressed to save memory 2022-10-30 16:19:58 +01:00
EventReport.php fix: [typo] fixed after crash 2022-11-09 14:56:36 +01:00
EventTag.php fix: [UI] Error message when trying to add invalid attribute to object 2022-12-15 10:04:45 -05:00
FavouriteTag.php
Feed.php new: [feed] Store freetext feed compressed in cache 2022-10-30 11:49:38 +01:00
FuzzyCorrelateSsdeep.php fix: [internal] Enabling correlation for event 2022-05-21 11:39:51 +02:00
Galaxy.php chg: [internal] Code cleanup for galaxy import 2023-05-17 13:56:17 +02:00
GalaxyCluster.php fix: [sync] Error handling when pulling clusters 2023-05-24 09:58:09 +02:00
GalaxyClusterBlocklist.php chg: [internal] Faster checking if galaxy cluster is blocked 2023-05-24 09:58:09 +02:00
GalaxyClusterRelation.php does not exists => does not exist 2022-11-08 16:13:14 +01:00
GalaxyClusterRelationTag.php
GalaxyElement.php fix: [UI] Galaxy cluster UI cleanup 2022-12-15 10:05:13 -05:00
Inbox.php fix: [remote_ip] respect MISP.log_client_ip_header everywhere fixes #8781 2022-11-28 14:08:11 +01:00
Job.php chg: [appModel] Usage of `update` worker instead of `prio` 2022-09-01 15:55:37 +02:00
Log.php chg: [workflows] Log After Save module 2023-03-31 02:36:10 +02:00
MispObject.php fix: consistent hash calculation in object dup checks 2023-03-23 15:03:58 +01:00
Module.php chg: [UI] Import module cleanup 2022-12-15 10:04:45 -05:00
News.php chg: [internal] More clear method name 2022-10-08 13:43:18 +02:00
Noticelist.php chg: [internal] Use JsonTool more often 2022-10-19 10:11:37 +02:00
NoticelistEntry.php
NotificationLog.php
ObjectReference.php chg: [internal] Faster capturing object references 2022-05-29 16:21:25 +02:00
ObjectRelationship.php chg: [internal] Cleanup code for JSON decoding 2022-10-23 13:17:39 +02:00
ObjectTemplate.php new: [UI] Preparation for creating object from freetext 2022-12-15 10:05:13 -05:00
ObjectTemplateElement.php chg: [internal] Use JsonTool more often 2022-10-19 10:11:37 +02:00
OrgBlocklist.php new: [orgBlocklist:index] Added total blocked count and last block time for each blocked orgs 2023-04-18 15:06:22 +02:00
Organisation.php fix: [UI] Statistics EventTag call 2022-11-02 10:49:24 +01:00
OverCorrelatingValue.php fix: [over-correlations] weren't truly case insensitive, causing potential issues matching and entering values 2023-05-22 11:26:05 +02:00
Post.php new: [trigger:post_after_save] Added support of post_after_save trigger 2022-07-27 12:06:07 +02:00
Regexp.php
RestClientHistory.php fix: [UI] REST client 2022-04-10 15:04:48 +02:00
Role.php fix: [todo] CakePHP automatically i18n $validate Model vars 2023-05-21 09:40:31 +02:00
Server.php chg: [security] OTP support for HOTP 2023-05-25 23:28:14 +02:00
ShadowAttribute.php fix: [cleanup] removes some TODO messages #103 2023-05-21 10:09:05 +02:00
SharingGroup.php chg: [internal] Be sure that authorizedIds methods returns int 2022-10-22 14:35:41 +02:00
SharingGroupBlueprint.php fix: [sharing group blueprint] fixed 2022-03-05 11:06:24 +01:00
SharingGroupOrg.php chg: [internal] Be sure that authorizedIds methods returns int 2022-10-22 14:35:41 +02:00
SharingGroupServer.php chg: [internal] Be sure that authorizedIds methods returns int 2022-10-22 14:35:41 +02:00
Sighting.php Revert "Feature/api log and pin ip (#8965)" 2023-03-29 09:53:29 +08:00
Sightingdb.php
SightingdbOrg.php
SystemSetting.php fix: [systemSetting] Check if database exists 2021-11-18 15:04:54 +01:00
Tag.php fix: [internal] Taxonomy code cleanup 2022-12-15 10:04:45 -05:00
TagCollection.php chg: [acl] Move tags ACL check to one place 2022-10-26 21:10:23 +02:00
TagCollectionTag.php chg: [acl] Move tags ACL check to one place 2022-10-26 21:10:23 +02:00
Task.php
TaxiiServer.php new: [taxii preview] Browse a taxii server and view the data it contains 2023-06-07 14:31:58 +02:00
Taxonomy.php chg: [performance] fix for events with large numbers of attributes and multiple tags from the same taxonomy 2023-06-08 08:16:50 +02:00
TaxonomyEntry.php
TaxonomyPredicate.php
Template.php chg: [internal] App model cleanup 2021-11-07 10:40:01 +01:00
TemplateElement.php
TemplateElementAttribute.php
TemplateElementFile.php
TemplateElementText.php
TemplateTag.php
Thread.php
ThreatLevel.php
User.php chg: [user] log last_api_access hourly if MISP.store_api_access_time is not set 2023-05-21 20:12:44 +02:00
UserSetting.php new: [user:periodicNotification] Started development of system allowing users to receive period notifications by email 2022-08-29 09:03:42 +02:00
Warninglist.php chg: [internal] Update warninglist cache just when it is necessary 2022-11-03 12:27:00 +01:00
WarninglistEntry.php chg: [internal] Remove unused methods 2021-11-03 10:27:42 +01:00
WarninglistType.php chg: [internal] Remove unused methods 2021-11-03 10:27:42 +01:00
Workflow.php chg: [workflow:getClassFromModule] Removed error suppression while importing modules 2023-05-04 14:49:46 +02:00
WorkflowBlueprint.php fix: [workflow] Importing blueprints 2022-10-10 22:26:12 +02:00