MISP
/
MISP
mirror of https://github.com/MISP/MISP
2
2
Fork 0
Code Issues Releases Wiki Activity
21,955 Commits
42 Branches
370 Tags
181 MiB
Commit Graph

214 Commits (60160cf999094ad785afda18c8cb00da1c9b60cd)

Author SHA1 Message Date
Jakub Onderka b749610c56 chg: [feed] Check also URL without protocol 2020-10-29 18:55:17 +01:00
iglocska 42aaea20f8
fix: [server] caching notice fixed 2020-10-16 14:41:45 +02:00
Jakub Onderka 0e7d3ddc5d chg: [internal] Better error messages for unzipping feed file 2020-10-07 09:07:11 +02:00
Jakub Onderka c2f6908002 chg: [optimisation] Load MISP version and commit just once 2020-10-03 20:02:55 +02:00
Jakub Onderka 95dfd75af7 chg: [correlations] Really limit number of correlations, not number of attributes 2020-09-28 19:52:19 +02:00
Jakub Onderka d1eec44d8d chg: [internal] Initialize Job class just once 2020-09-28 19:40:32 +02:00
Jakub Onderka 7f29b6b089 chg: [internal] Delete Redis key in pipeline 2020-09-28 18:33:56 +02:00
Jakub Onderka 552315555c chg: [feed] Faster feed list fetching 2020-09-28 17:19:25 +02:00
Jakub Onderka c87616e6ec fix: [correlations] Properly delete feeds caches 2020-09-28 17:00:00 +02:00
Jakub Onderka a11e27bedc chg: [correlations] Fetch just server of feed that has data in Redis 2020-09-28 15:49:54 +02:00
Jakub Onderka 0c5be866fc chg: [correlations] Refactor feed cached correlations 2020-09-28 15:02:08 +02:00
Jakub Onderka 2deb9d3c8b
Merge pull request #6156 from JakubOnderka/feed-httpsocket-optional 
fix: [feed] Make HttpSocket instance optional for local feeds
 2020-08-03 10:51:03 +02:00
iglocska b5dffc2023
fix: [internal] older PHP still not happy with the return from a generator 2020-07-30 17:32:35 +02:00
iglocska dfc743aab2
fix: [internal] make ancient PHP versions happy 2020-07-30 17:24:19 +02:00
Jakub Onderka d9f21f75f5 chg: [feed] Better exception messages for invalid JSON 2020-07-28 18:33:05 +02:00
Jakub Onderka 6c411203c0 fix: [feed] Make HttpSocket instance optional for local feeds 2020-07-28 17:59:55 +02:00
mokaddem afbf870e16
fix: [feed:importFreeText] Make sure to update the timestamp when soft-deleting after delta-merge 
Fix #6013
 2020-07-21 12:13:59 +02:00
mokaddem 3d2fd93de2
fix: [feed] Accept more text content. Fix #5969 2020-07-21 10:10:08 +02:00
Andras Iklody 8eaa16c94d
Merge pull request #6036 from JakubOnderka/cache-freetext-feed-optim 
chg: [feed] Faster freetext feed caching
 2020-07-07 12:19:41 +02:00
Andras Iklody e6995dab67
Merge pull request #6035 from JakubOnderka/rest-search-optim 
chg: [internal] Attribute REST search optimisations and error handling
 2020-07-07 12:18:29 +02:00
Jakub Onderka cb3608b009 fix: [feed] Incorrect call in Feed::__saveEvent 2020-06-26 14:56:38 +02:00
Jakub Onderka e768d5071f chg: [feed] Faster freetext feed caching 2020-06-20 10:13:01 +02:00
Jakub Onderka f46ca1634a chg: [internal] Use tmp file fro Feed::getCache 2020-06-20 09:54:32 +02:00
Jakub Onderka b2c193c984 fix: [pull] Correct progress for pull job 2020-06-07 22:25:32 +02:00
mokaddem d022b0fca2
fix: [feeds:saveFreetext] Soft-delete Attributes when performing a 
delta-merge
 2020-06-03 16:17:16 +02:00
mokaddem dcd3664bba
Merge branch '2.4' of github.com:MISP/MISP into pr-5256 2020-05-18 14:14:51 +02:00
mokaddem b531762bf9
Merge branch '2.4' into pr-5862 2020-05-18 09:42:47 +02:00
Andras Iklody 5f9197c0a8
Merge pull request #5897 from JakubOnderka/fixed_event_freetext_feed_speedup 
chg: [feed] Modify value when checking if value exists in current event
 2020-05-14 09:55:29 +02:00
Andras Iklody db154195ec
Merge pull request #5903 from JakubOnderka/correlation-speedup 
Correlation speedup
 2020-05-14 08:33:09 +02:00
Jakub Onderka d249d0cce0 fix: [correlations] Do not check all attributes when cache is empty 2020-05-13 13:03:39 +02:00
Jakub Onderka d91a42d6b9 chg: [feed] Modify value when checking if value exists in current event 2020-05-12 11:42:55 +02:00
Jakub Onderka d92f7480b5 fix: [feed] Job progressbar fix 2020-05-11 18:17:29 +02:00
Jakub Onderka 246cf22571 fix: [feed] Optimise saving freetext feeds with a lot of attributes 2020-05-11 17:06:55 +02:00
Jakub Onderka 94b36c9259 chg: [feed] Provide more info when caching feeds about failures 2020-05-11 15:00:52 +02:00
Jakub Onderka 060cf3d459 fix: [feed] Follow redirect when fetching manifest 2020-05-05 12:49:34 +02:00
iglocska 49499f1df8
Merge branch '5834' into 2.4 2020-04-30 07:07:25 +02:00
Jakub Onderka 4dec19dedf new: [feed] Support for compressed feeds 2020-04-28 16:16:34 +02:00
mokaddem 436fd184c9
Merge branch '2.4' into JakubOnderka-redis-delete-multiple 2020-04-28 09:16:13 +02:00
iglocska 2d63f68fc1
new: [feeds] index refactor and new features 
- added the ability to select an orgc ID for CSV/freetext feeds
  - all events created from this feed will carry the selected orgc_id

- Refactored the index fully
  - using the factories
  - better warnings against the dangerous new feed each pull setting
  - event index search added
  - several settings cleaned up / made more clear

- auto reload of default feed configuration disabled, fixes #2542, fixes #5789
  - added a button / endpoint to handle that instead to allow for the deleted default feeds to stay deleted
 2020-04-17 14:17:54 +02:00
Bechkalo Evgeny 4d5f1aaf00 Fixed bugs with PostgreSQL in bruteforce and feed models 2020-03-30 16:19:43 +03:00
iglocska 30ff4b6451
chg: [security] Added setting to restrict the encoding of local feeds 
- By adding local feeds, a malicious administrator could point MISP to ingest configuration files that the apache user has access to
- This includes some more sensitive files (database.php / config.php / .gnupg data)
- Whilst this is currently not leading to an exploitable vulnerability as the current implementation wouldn't trigger on the values,
  having a setting to disable this will become much more interesting once we have a system in place for custom feed parsers
- The setting can only be enabled/disabled via the CLI

- As reported by Matthias Weckbecker
 2020-03-30 14:02:14 +02:00
Jakub Onderka 7539cbff2c
chg: [feed] Use precomputed hashes to speedup attaching correlation 2019-10-20 19:08:01 +02:00
garanews 85c28ce36e Fix some typo 
Fix some typo
 2019-10-04 13:02:59 +02:00
Pierre-Jean Grenier 8415cea5db fix: [feed] fix REST problems 2019-10-02 16:53:44 +02:00
Pierre-Jean Grenier 027aa91dd4 fix: [feed] array from NULL is not NULL 2019-10-02 16:37:52 +02:00
Andras Iklody 20311ae3e6
Merge pull request #5217 from JakubOnderka/patch-36 
new: [internal] New AppModel::logException method
 2019-09-30 21:26:09 +02:00
Jakub Onderka 28e2e71668
chg: [feed] Compute md5 value just once 2019-09-26 11:55:47 +02:00
Jakub Onderka ca3e9427bb
chg: [feed] Use new AppModel::logException method 2019-09-24 20:54:11 +02:00
Andras Iklody d656c2b318
Merge pull request #5097 from JakubOnderka/patch-18 
chg: [feed] Break loop when match is found
 2019-09-16 09:15:37 +02:00
Jakub Onderka 85754ccc83 fix: [internal] Deleting multiple Redis keys 2019-09-15 13:05:55 +02:00
iglocska 3240a6cac4
new: [CLI] Added cleanup tool to purge all events related to a feed 
- Simply run /var/www/MISP/app/Console/cake Admin purgeFeedEvents [user_id] [feed_id]
- works for CSV/Freetext feeds
 2019-09-13 10:48:43 +02:00
Jakub Onderka 609998e24c
fix: [feed] When fetching feeds, accept also text/plain in HTTP 2019-09-10 12:21:42 +02:00
Jakub Onderka 7904509fde fix: [feed] Fetching event manifest 2019-09-05 19:16:26 +02:00
Jakub Onderka e20fc9ce60
chg: [feed] Break loop when match is found 2019-09-03 18:04:47 +02:00
Jakub Onderka a45575a44d chg: [feed] Error handling for saveFreetextFeedData 2019-09-03 17:24:06 +02:00
Jakub Onderka 38e2c1cfc2 fix: [feed] Use own solution how to redirect when fetching feeds 2019-09-02 19:52:41 +02:00
Jakub Onderka d5e3be18b5 chg: [feed] Throw exception if feed filter rules is not valid JSON 2019-09-02 19:52:41 +02:00
Jakub Onderka de2b143748 chg: [feed] Better error handling for cacheFeedInitiator method 2019-09-02 19:52:41 +02:00
Jakub Onderka d7809df2d0 chg: [feed] More accurate progress logging for downloadFromFeed 2019-09-02 19:52:41 +02:00
Jakub Onderka 6c878ff31c chg: [feed] Better exception logging 2019-09-02 19:52:41 +02:00
Jakub Onderka 35d67c261e chg: [feed] New method Feed::jobProgress 2019-09-02 17:51:37 +02:00
Jakub Onderka 77fe7e10fc chg: [feed] New private method Feed::isFeedLocal 2019-09-02 17:51:37 +02:00
Jakub Onderka 49df6ccd1a chg: [feed] Use Feed::feedGetUri in Feed::getFreetextFeed 2019-09-02 17:51:37 +02:00
Jakub Onderka 18eb86ff32 chg: [feed] Feed::getNewEventUuids always return array with 'add' and 'edit' fields 2019-09-02 17:51:37 +02:00
Jakub Onderka 8973cb75f7 chg: [feed] Proper error handling for fetching feed cache 2019-09-02 17:51:37 +02:00
Jakub Onderka ba1d6ed541 chg: [feed] New method Feed::feedGetUri to deduplicate code 2019-09-02 17:51:37 +02:00
Jakub Onderka 78f113c563 chg: [feed] Proper error handling for fetching feed events 2019-09-02 17:51:37 +02:00
Jakub Onderka 3c70f816c6 chg: [feed] Deduplicate preparing filter rules 2019-09-02 17:51:37 +02:00
Jakub Onderka 5b2bb6d0fa chg: [feed] Proper error handling for fetching freetext feed 2019-09-02 17:51:37 +02:00
Jakub Onderka 3adadb100e chg: [feed] Follow redirects by internal HttpSocket functionality 2019-09-02 17:51:37 +02:00
Jakub Onderka 9ee3570379 chg: [feed] Proper error handling for fetching feed manifest 2019-09-02 17:51:37 +02:00
Jakub Onderka 0ddfbc9ccf fix: [feed] shell_exec don't throw Exception 2019-09-02 17:51:37 +02:00
Jakub Onderka 0aeacde256 chg: [feed] Name variables after explore in Feed::attachFeedCorrelations 2019-08-24 17:18:10 +02:00
Jakub Onderka d346de0118 chg: [feed] Compute Redis cache key prefix just once 2019-08-21 22:16:27 +02:00
Jakub Onderka 9846635603 chg: [feed] Fetch sources just when there is correlation 2019-08-21 22:14:30 +02:00
Jakub Onderka 0cf1baf8f3 chg: [feed] Compute composite types just once 2019-08-21 22:14:30 +02:00
Jakub Onderka ada263c79d fix: [feed] Remove unused variables 2019-08-21 22:14:30 +02:00
iglocska a26b38e2f5 fix: [feeds] Stop the reset of the attribute count after a feed pull is completed, fixes #4414 2019-04-05 10:46:32 +02:00
iglocska ab858d5f17 new: [feeds] Feed/Server cache search added 
- /feeds/searchCaches added
 2019-04-01 16:09:24 +02:00
iglocska fc34510eda new: [Feeds] New overlap tool finished 
- compare a feed against a combination of feeds/servers to find if you can cover the contents with a combination of other cached feeds
 2019-03-10 18:09:46 +01:00
iglocska 947e59aec7 new: [Feeds] Implementation of the feed coverage tool (WIP) 2019-03-09 22:19:37 +01:00
iglocska 0cbd53f3e7 fix: [feed] Feed pulls don't update the timestamp of the event 
- affects freetext feeds
- this blocks the further propagation of the modifications
 2019-02-27 20:53:33 +01:00
iglocska 80a33be0e8 chg: [feed correlations] Don't attach feed correlations to attributes that have correlations disabled 2019-01-31 11:08:51 +01:00
iglocska 498a7ae77c new: [feeds] Opened up feed inspection to host org users and added servers to overlap matrix 2019-01-20 10:19:05 +01:00
iglocska eb56fe4e01 chg: [remote cache] Further progress on caching remote instances 2019-01-18 15:33:04 +01:00
iglocska 2d0259ce13 fix: [CS] coding standards script re-run 2018-11-23 14:11:33 +01:00
iglocska c3ec498f46 fix: [CLI] Fixed the CLI feed fetcher 
- use "all" to fetch all feeds
 2018-11-08 13:10:27 +01:00
iglocska e60579dfd8 fix: [Feed] If no data is returned from a freetext feed a notice was generated 
- added more graceful handling
 2018-10-02 15:37:42 +02:00
iglocska fac5ffa7f3 fix: [Feeds] Don't try to find caches for feeds that don't have caching enabled 2018-09-07 13:49:56 +02:00
iglocska b34d99aa63 fix: [feeds] Custom headers / authorization broken on csv/freetext feeds, fixes #3581 2018-08-29 09:56:14 +02:00
iglocska a81894f14c chg: [CS] Changed to PSR-2 
- to make contributions easier, adopted PSR-2
- used php-cs-fixer to rework the style
- *sniff sniff* Goodbye tab indentation
 2018-07-19 11:48:22 +02:00
iglocska af9aeab62b fix: [feed] Invalid lookup when editing events via MISP feeds throws notice error, fixes #3366 2018-07-08 23:39:15 +02:00
iglocska 46e44aa9a4 chg: [clarity] Made the file path validationfailing more obvious when adding local feeds 
- Warning to catch issues that arise due to Steve's fat fingers
 2018-06-22 05:30:03 +02:00
iglocska f6ed105249 fix: [validation] Fixed urlOrExistingFilepath validation script no longer uses hard-coded error messages 2018-06-02 23:24:36 +02:00
iglocska cfd6ab3e2d fix: [Feed caching] Readded the feed correlations for non correlating attributes 
- it was breaking the indexing for the attached correlations
 2018-05-09 16:35:08 +02:00
iglocska 1d3504b406 fix: Fixed invalid indeces in the feed lookup via the event view 2018-05-08 14:26:41 +02:00
iglocska b0e13b0faf fix: Don't correlate attribute to feeds if the correlations are disabled on the attribute 2018-04-25 15:54:36 +02:00
iglocska a28c1da164 fix: Fixed a bug where background jobs for feeds would not work correctly due to headers not being passed along with the feed object 2018-04-16 07:59:44 +02:00
iglocska e89d7ebc16 new: Added separation between enabled feeds and feeds enabled for caching 2018-03-30 23:30:54 +02:00
iglocska 571249c09b fix: Empty events are created when pulling empty feeds, fixes #3008 
- as described by Emanuele Acri (@crossbowerbt)
 2018-03-07 16:07:35 +01:00