0cbd53f3e7
fix: [feed] Feed pulls don't update the timestamp of the event
...
- affects freetext feeds
- this blocks the further propagation of the modifications
2019-02-27 20:53:33 +01:00
80a33be0e8
chg: [feed correlations] Don't attach feed correlations to attributes that have correlations disabled
2019-01-31 11:08:51 +01:00
498a7ae77c
new: [feeds] Opened up feed inspection to host org users and added servers to overlap matrix
2019-01-20 10:19:05 +01:00
eb56fe4e01
chg: [remote cache] Further progress on caching remote instances
2019-01-18 15:33:04 +01:00
2d0259ce13
fix: [CS] coding standards script re-run
2018-11-23 14:11:33 +01:00
c3ec498f46
fix: [CLI] Fixed the CLI feed fetcher
...
- use "all" to fetch all feeds
2018-11-08 13:10:27 +01:00
e60579dfd8
fix: [Feed] If no data is returned from a freetext feed a notice was generated
...
- added more graceful handling
2018-10-02 15:37:42 +02:00
fac5ffa7f3
fix: [Feeds] Don't try to find caches for feeds that don't have caching enabled
2018-09-07 13:49:56 +02:00
b34d99aa63
fix: [feeds] Custom headers / authorization broken on csv/freetext feeds, fixes #3581
2018-08-29 09:56:14 +02:00
a81894f14c
chg: [CS] Changed to PSR-2
...
- to make contributions easier, adopted PSR-2
- used php-cs-fixer to rework the style
- *sniff sniff* Goodbye tab indentation
2018-07-19 11:48:22 +02:00
af9aeab62b
fix: [feed] Invalid lookup when editing events via MISP feeds throws notice error, fixes #3366
2018-07-08 23:39:15 +02:00
46e44aa9a4
chg: [clarity] Made the file path validationfailing more obvious when adding local feeds
...
- Warning to catch issues that arise due to Steve's fat fingers
2018-06-22 05:30:03 +02:00
f6ed105249
fix: [validation] Fixed urlOrExistingFilepath validation script no longer uses hard-coded error messages
2018-06-02 23:24:36 +02:00
cfd6ab3e2d
fix: [Feed caching] Readded the feed correlations for non correlating attributes
...
- it was breaking the indexing for the attached correlations
2018-05-09 16:35:08 +02:00
1d3504b406
fix: Fixed invalid indeces in the feed lookup via the event view
2018-05-08 14:26:41 +02:00
b0e13b0faf
fix: Don't correlate attribute to feeds if the correlations are disabled on the attribute
2018-04-25 15:54:36 +02:00
a28c1da164
fix: Fixed a bug where background jobs for feeds would not work correctly due to headers not being passed along with the feed object
2018-04-16 07:59:44 +02:00
e89d7ebc16
new: Added separation between enabled feeds and feeds enabled for caching
2018-03-30 23:30:54 +02:00
571249c09b
fix: Empty events are created when pulling empty feeds, fixes #3008
...
- as described by Emanuele Acri (@crossbowerbt)
2018-03-07 16:07:35 +01:00
fd858d627b
new: Allow passing headers along with feeds
...
- add any arbitrary header to a feed
- can be used for authentication via basic auth for example
2018-01-19 00:28:44 +01:00
32d00a09e0
fix: Fixed various visual feed issues, fixes #2818 , fixes #2819
2018-01-16 12:30:30 +01:00
56dc5ea3bc
fix: Better error handling when previewing csv/freetext feeds if no valid data is returned
2018-01-13 15:36:27 +01:00
995a1c700e
fix: Better handling of something going wrong whilst fetching a MISP feed's manifest
2018-01-13 15:33:00 +01:00
a5ceed1522
fix: Recursively follow redirects for feeds, fixes #2774
2018-01-08 15:56:13 +01:00
8164e96e56
fix: Feed quick sync added
2017-12-22 14:47:00 +01:00
3e112be322
new: Various improvements to the CSV export
...
- The @FloatingCode and @ilmoka care package
- Improved CSV performance for instances with large number of events
- Added "value" filter for CSV (use-case: I want all indicators for this value with context)
- Added attribute tags to the output of the CSV export
2017-12-05 09:21:31 +01:00
4f6dba5f35
new: various improvements
...
- use the feed uuid caches to link directly to affected MISP events
- various UI improvements
- Feed preview pagination / POSTed event ID filters added
2017-12-05 00:05:11 +01:00
1fb8722487
new: Various features
...
- Added quickhashing to the feed generator
- Objects added to feed preview for MISP feeds
- Attribute tags added to MISP feeds
2017-11-07 16:50:08 +01:00
0efa2bef1a
new: Rework of the feed correlation lookups for the event view
...
- massive performance boost by using redis pipelining
- for events with 10k+ attributes, show truncated feed correlation lookups, informing the user about the number of correlating attributes and a boolean flag on attributes saying that they correlate
- The overall feed correlation counter also allows users to pivot to a view that loads all correlations, though it should be used with some caution as it can be somewhat heavy
2017-10-11 11:12:34 +02:00
5290214c9b
new: First iteration of the graphing engine rework
2017-10-06 10:05:00 +02:00
8fb0b19706
fix: Follow redirect from feed pull if the response is a 302
2017-08-25 12:16:10 +02:00
2e8d0f30fd
fix: Cleanup for feeds fixed
2017-08-25 09:01:15 +02:00
6c7c40e773
fix: Further performance tweaks to the feed fetcher
2017-07-07 12:58:51 +02:00
6b6ea52b58
fix: Made the feed pull for CSV/Freetext feeds much faster for large feeds
...
- value de-duplication is now a lot more efficient
2017-07-07 10:12:55 +02:00
3083d213a9
fix: Fixed an error causing combined feed cache issues
2017-06-13 09:11:27 +02:00
cffc8a4f8f
fix: Fixes a feed caching issue introduced by the performance tweaks
...
- moved the combined feed generation for the fast lookups to the feed caching algorigthms as opposed to an on an on-the-fly merge
2017-06-12 15:25:08 +02:00
3d74dbee28
new: First round of massive performance tuning (tm)(c)
...
- Make MISP fast again
2017-06-09 15:38:45 +02:00
bfc70829b8
fix: Fixed issues with feeds that time out causing failures
2017-06-06 11:57:34 +02:00
01a581d4a7
fix: Added empty string as default for feed data
...
- to handle cases where no data is returned.
2017-06-05 11:36:57 +02:00
1467c69ab4
chg: Also store the lookup_visible field from the field import
2017-05-30 11:46:27 +02:00
9f8954999a
new: Auto load the default feeds from file
2017-05-30 11:42:57 +02:00
e18344b3f5
fix: several fixes to the feed overlay matrix
...
- lookup was broken for csv/freetext feeds
- allow users to see the feeds if the admin allows it
2017-05-09 16:35:59 +02:00
765ce33575
Merge branch '2164' into 2.4
2017-05-09 10:44:04 +02:00
90eef89223
fix: Several feed fixes
2017-05-08 16:17:34 +02:00
96574ec335
new: First implementation of the feed analysis system
2017-05-08 14:22:27 +02:00
926895733b
Cosmetic changes
...
Change space indents to tabs
Remove ?> at end of file
Add or remove some indentation where appropriate
2017-05-08 00:45:57 +02:00
b24be9b894
fix: throw an error if the local feed file is not found.
2017-02-21 09:11:20 +01:00
a59aab9b23
fix: Re-added the accidentally removed code in a merge, fixes #1965
...
- affects f0e1a27b7d
2017-02-20 18:43:36 +01:00
88123b4496
fix: Fixed an issue with the feed url validation
2017-01-25 05:56:32 +01:00
1dc8942340
fix: Fixed an old bug returning an invalid feed pull result
...
- no new events / nothing to update returned an error before
2017-01-24 16:05:33 +01:00
892ea7c021
chg: Added validation errors for a local feed pointing to the wrong resource
...
- should be a file for non misp feeds
- should be a directory for misp feeds
2017-01-24 16:04:00 +01:00
aba2491344
new: Local feeds
...
- still needs testing
2017-01-24 14:07:55 +01:00
26df6d3fe5
new: Added a new field for an exclude regex for the CSV / Freetext feeds
...
- just set a php compatible PCRE regex pattern to exclude values
2017-01-01 16:25:08 +01:00
df85d488d2
chg: Changed the feed cache locations
2017-01-01 12:52:17 +01:00
f8dc191f53
fix: Fix to several issues with the feeds:
...
- settings (csv column number, delimiter) were ignored
- skipped fields were still counted by the paginator showing some pages with fewer than the expected 60 values
2017-01-01 12:25:35 +01:00
76e9398df9
new: Various new feed features
...
- import feed descriptor json pastes to add a list of pre-defined feeds
- improvements to the feed pull (a single non validating attribute shouldn't break the process)
- altered the saving of the attributes to happen in chunks during a feed pull to avoid very large feeds from stalling the process
- split the feeds into 3 tabs: default, custom, all
2016-12-31 09:04:46 +01:00
7f8a81e161
new: Added caching and pagination to freetext/csv feeds
2016-12-30 16:16:56 +01:00
a637542a56
fix: Fixed an issue that incorrectly reported a feed update to have failed when not using delta-merge mode
...
- the issue was that in the case of a feed update to a fixed event without delta merge, MISP tried to insert all parsed attributes, which correctly automatically blocked duplicates
- however, since these attributes were blocked by the validator, the feed fetcher reported that the fetch didn't succeed as it contained validation errors
- this fix simply runs non-delta merge mode updates through the comparisons to the existing event, removing duplicates in advance
2016-11-24 10:30:53 +01:00
c2fc803fed
chg: Use the TLD lists from the warninglists, fixes #1149
...
- simply load any enable warninglist entries from the pre-defined TLD warninglists
- Pass the resulting array to the complex type tool
- during domain type heuristics, if the TLD list is not empty use the supplied list
- alternatively generate a list based on the old TLD rules
- does not alter any functionality otherwise
2016-10-25 22:23:01 +02:00
2ab78bb1cb
chg: Validate the event_id as a numeric value
2016-10-11 15:09:59 +02:00
6e4c4da094
fix: Fixes an issue where the csv feed pull would be routed through the freetext code path
2016-10-09 13:24:59 +02:00
9891234662
new: CSV feeds and various fixes
...
- Added the CSV feed format
- users can specify which fields in the CSV should be parsed
- comment lines are automatically omitted
- new settings system added to feeds, currently only used for the value fields
- Slight rework of the correlation lookup for the feeds
- got the Speed Force treatment
- correctly checks against value1 and value2 instead of value
- Various freetext import fixes
2016-10-08 14:36:24 +02:00
09d3720187
new: Added correlations to the freetext feed preview
2016-10-08 10:25:30 +02:00
9d4403ff3e
fix: Resolved several issues with error handling in the new feed system
2016-10-07 20:08:57 +02:00
9649c608b4
fix: Various fixes to the feed system
...
- allow users to override the IDS flags and keep all attributes pulled from a freetext feed IDS = off
- UI changes
- fix to a bug that caused already deleted attributes to be counted as existing ones
2016-10-07 18:27:13 +02:00
503661a240
new: First implementation of the freetext feed pull
2016-10-07 17:33:54 +02:00
a706e5c59d
fix: Added the git commit ID to the feed request
2016-09-30 17:26:12 +02:00
25e52a6786
chg: remove some references to variables
2016-09-15 17:08:58 +02:00
822b0bf8fa
chg: Cleanup of the controllers and models
...
- removed incorrect, useless boiler plate comments
- kept useful comments intact
- added some missing line breaks to make the codebase a bit more uniform
- removed some obviously obsolete TODO comments
2016-08-25 11:38:37 +02:00
0fe692c56a
remove whitespace at end of line
2016-06-04 01:10:45 +02:00
898ea1d97c
remove whitespace (space/tab) from empty lines
2016-06-04 01:08:16 +02:00
c27210611b
Fixed a typo in the feed adder, fixes #1022
2016-03-14 15:25:55 +01:00
47ffaf65b8
Fix to the previous commit
...
- also enabling the test feed by default
2016-03-14 14:07:47 +01:00
79e41da383
First version of Feed system ready
...
- tied into background processes
2016-03-09 02:31:55 +01:00
8e8de0d258
Added default feeds
2016-03-09 01:34:02 +01:00
3845c7442a
Correctly assign tag / sharing group to event fetched from feed
2016-03-09 00:44:09 +01:00
fe3a94d3d1
New fields added
...
- set the distribution and sharing group of a feed
- will set all events received to the appropriate setting
- set a tag that should be applied by default to the events received from the feed
2016-03-08 23:27:53 +01:00
0ee380d7c0
Fixed an issue with the filtering
...
- needle, haystack or haystack, needle.
2016-03-07 09:02:35 +01:00
f767ddaec6
Added downloading of an event from the index, better error handling
2016-03-07 03:26:55 +01:00
7a20704f36
Preview Event implemented
2016-03-07 02:23:37 +01:00
03e19f6db0
Preview the index of a feed
2016-03-06 23:24:03 +01:00
bebdde03e7
Further progress
2016-03-04 14:56:56 +01:00
255c65942e
Further progress on the feeds
2016-02-29 22:32:04 +01:00
iglocska