MISP
/
misp-modules
mirror of https://github.com/MISP/misp-modules
2
0
Fork 0
Code Issues Releases Wiki Activity
1,450 Commits
6 Branches
54 Tags
18 MiB
Commit Graph

736 Commits (be2786990346f8f5b12812cc0572bf51ba291f7c)

Author SHA1 Message Date
Sebdraven 80e71f582c debug ipv4 or ipv6 2018-07-11 11:58:42 +02:00
Sebdraven 386d38c88f add debug 2018-07-11 11:55:59 +02:00
Sebdraven 45decc728d debug 2018-07-11 11:55:31 +02:00
Sebdraven 45c473aef5 change status 2018-07-11 11:52:10 +02:00
Sebdraven 64e7f9c8b6 change history dns 2018-07-11 11:47:10 +02:00
Sebdraven 560dacbf7e add logs to debug 2018-07-11 11:40:22 +02:00
Sebdraven 74c611d2fb correct call function 2018-07-11 11:37:07 +02:00
Sebdraven f47a64b364 add history mx and soa 2018-07-11 11:24:49 +02:00
Sebdraven 43a49dafc6 add history dns and handler exception 2018-07-11 09:48:14 +02:00
Sebdraven 54d996cb00 add history dns 2018-07-11 09:39:09 +02:00
Sebdraven dcdb6e5895 switch type ip 2018-07-11 09:02:47 +02:00
Sebdraven 42c362d2fd refactoring expand_whois 2018-07-11 09:00:23 +02:00
Sebdraven 41635d43c7 correct typo 2018-07-11 08:49:59 +02:00
Sebdraven 3a96e189ed add ipv6 and ipv4 2018-07-11 08:43:23 +02:00
Sebdraven f2333a4978 change type 2018-07-10 16:55:13 +02:00
Sebdraven 9e6162a434 change type 2018-07-10 16:53:06 +02:00
Sebdraven 26950ea7de change loop 2018-07-10 16:51:31 +02:00
Sebdraven e9747a3379 add time sleep in each request 2018-07-10 16:41:44 +02:00
Sebdraven 602da3d1a3 control return of records 2018-07-10 16:35:01 +02:00
Sebdraven 495c720d0f add history ipv4 2018-07-10 16:31:39 +02:00
Sebdraven 21794249d0 add logs 2018-07-10 15:17:37 +02:00
Sebdraven b677cd5fc7 change categories 2018-07-10 15:16:02 +02:00
Sebdraven 1d100833a4 concat results 2018-07-10 15:12:27 +02:00
Sebdraven 1223d93d52 change name keys 2018-07-10 15:07:54 +02:00
Sebdraven 714c15f079 change return value 2018-07-10 15:05:10 +02:00
Sebdraven e1a1648f14 add logs 2018-07-10 15:01:04 +02:00
Sebdraven f710162bed change errors 2018-07-10 14:59:39 +02:00
Sebdraven 2a8fb76e84 add logs 2018-07-10 14:56:20 +02:00
Steve Clement 562a6b1308 - Removed test modules from view 
- Moved skeleton expansion module to it's proper place
 2018-07-03 08:27:54 +02:00
chrisr3d 90e42c0305
fix: Put the stix2-pattern library import in a try statement 
--> Error more easily caught
 2018-07-02 12:14:21 +02:00
chrisr3d 08d8459e1a
add: STIX2 pattern syntax validator 2018-07-02 11:38:33 +02:00
Steve Clement 549f32547d - Reverted to <3.6 compatibility 2018-07-01 22:09:02 +08:00
Steve Clement 9f0313a97e - Fixed log output 2018-06-30 12:01:21 +08:00
Steve Clement 184065cf74 - Forgot to import sys 2018-06-30 11:58:44 +08:00
Steve Clement ffce2aa5cc - Added logger functionality for debug sessions 2018-06-30 11:52:12 +08:00
Steve Clement 2f5dd9928e - content was already a wand.obj 2018-06-30 11:38:26 +08:00
Steve Clement 90f2fe9d19 Merge remote-tracking branch 'upstream/master' 2018-06-30 01:05:01 +08:00
Steve Clement f97359de6a Merge branch 'master' of github.com:SteveClement/misp-modules 2018-06-30 01:04:30 +08:00
Steve Clement ef3837077e - Some more comments 
- Removed libmagic, wand can handle it better
 2018-06-30 00:58:25 +08:00
Sebdraven 34da5cdb76 add expand whois 2018-06-29 17:57:11 +02:00
Sebdraven f1c6095914 typo 2018-06-29 17:26:56 +02:00
Sebdraven 78d6de9b7a add categories and comments 2018-06-29 17:25:37 +02:00
Sebdraven 0965def6bf add expand subdomains 2018-06-29 17:22:19 +02:00
Sebdraven 64847a8a04 add expand subdomains 2018-06-29 17:19:21 +02:00
Sebdraven 2d1adf4aa9 change categories 2018-06-29 16:30:47 +02:00
Sebdraven 0275e3ecd8 changes keys 2018-06-29 16:20:35 +02:00
Sebdraven f3962d2d05 add status ! 2018-06-29 16:17:32 +02:00
Sebdraven 09c52788b8 add methods 2018-06-29 16:11:24 +02:00
Sebdraven cfe971a271 add expand domains 2018-06-29 15:50:26 +02:00
Sebdraven 60f772b905 add new module dnstrails 2018-06-29 11:27:36 +02:00
Christophe Vandeplas ff793bc221
threatanalyzer_import - order of category tuned 2018-06-29 11:17:03 +02:00
Alexandre Dulaunoy d8eeb73a4a
Merge branch 'master' into master 2018-06-29 06:49:40 +02:00
Steve Clement fbb3617f25 - Quick comment ToDo: Avoid using Magic in future releases 2018-06-29 12:01:17 +08:00
Steve Clement 60a3fbe282 - added wand requirement 
- fixed missing return png byte-stream
- move module import to handler to catch and  report errorz
 2018-06-28 23:20:38 +08:00
Steve Clement 7885017981 - fixed typo move image back in scope 2018-06-28 16:59:03 +08:00
chrisr3d 7dd8e988c0
Updated the list of modules (removed stiximport) 2018-06-28 10:51:40 +02:00
chrisr3d b1c90b411e
add: Sigma syntax validator expansion module 
--> Checks sigma rules syntax
- Updated the expansion modules list as well
- Updated the requirements list
 2018-06-28 10:41:32 +02:00
chrisr3d 7c691af807
Updated the list of expansion modules 2018-06-28 10:39:40 +02:00
Steve Clement 59b7688bdc - Added initial PDF support, nothing is processed yet 
- Test to replace PIL with wand
 2018-06-28 16:00:14 +08:00
milkmix 349dd99d47 added support for scheduledtasks 2018-06-24 21:13:56 +02:00
milkmix 7c037ed090 added support for service-displayname, regkey|value 2018-06-24 21:09:42 +02:00
milkmix 0c6a205136 initial implementation supporting regkey. mutexes support waiting osquery table 2018-06-23 15:51:38 +02:00
Sebdraven 785aac3e6b add return handle domains 2018-06-22 16:18:23 +02:00
Sebdraven 87b07b89b5 add search 2018-06-22 16:15:34 +02:00
Sebdraven 396b71ef3b add domain to expand 2018-06-22 16:06:34 +02:00
Sebdraven de6a81d488 correct bugs 2018-06-22 16:04:14 +02:00
Sebdraven 83999d6402 add domain expansion 2018-06-22 15:57:52 +02:00
Sebdraven 96c829470d add comment 2018-06-22 15:14:44 +02:00
Sebdraven 8d03354399 correct bugs 2018-06-22 15:12:10 +02:00
Sebdraven e9c18b3d5f correct comments 2018-06-22 13:03:09 +02:00
Sebdraven e230c88c15 add threat list expansion 2018-06-22 11:59:09 +02:00
Sebdraven 1d1fd36569 change method to concat methods 2018-06-20 18:05:28 +02:00
Sebdraven e712a31760 set status after requests 2018-06-20 18:04:12 +02:00
Sebdraven a9b7a10c41 set status after requests 2018-06-20 18:03:34 +02:00
Sebdraven 4166475f9e add logs 2018-06-20 18:02:12 +02:00
Sebdraven fe00f099f6 add logs 2018-06-20 17:59:49 +02:00
Sebdraven 153d8bd340 add logs 2018-06-20 17:56:19 +02:00
Sebdraven 9195887f98 pep 8 2018-06-20 17:51:46 +02:00
Sebdraven 2afd2b8aaf correct bug 2018-06-20 17:50:28 +02:00
Sebdraven 04e932cce0 add datascan expansion 2018-06-20 17:47:11 +02:00
Sebdraven b56f8cfa36 add reverse infos 2018-06-20 16:30:56 +02:00
Sebdraven d4be9d9fda add reverse infos 2018-06-20 16:29:04 +02:00
Sebdraven 4a8a79c560 add reverse infos 2018-06-20 16:26:09 +02:00
Sebdraven 0d120af647 add reverse infos 2018-06-20 16:24:17 +02:00
Sebdraven a24b529868 add forward infos 2018-06-20 15:33:21 +02:00
Sebdraven d0f42c1772 add comment of attributes 2018-06-20 15:07:55 +02:00
Sebdraven 915747073a add comment of attributes 2018-06-20 15:05:00 +02:00
Sebdraven 7eba7c0386 error loops 2018-06-20 14:53:08 +02:00
Sebdraven d1e72676f1 error method 2018-06-20 14:50:48 +02:00
Sebdraven 3a4294391f error type 2018-06-20 14:48:18 +02:00
Sebdraven 9427c76603 error keys 2018-06-20 14:45:06 +02:00
Sebdraven e1bc67afad add expansion synscan 2018-06-20 14:41:57 +02:00
Sebdraven 5426ec5380 change key access domains 2018-06-20 12:40:52 +02:00
Sebdraven 7a3c4b1084 change add in results 2018-06-20 12:38:41 +02:00
Sebdraven e8aefde2ee add logs 2018-06-20 12:36:32 +02:00
Sebdraven 7195f33f5d correct error keys 2018-06-20 12:34:07 +02:00
Sebdraven c14d05adef test patries expansion 2018-06-20 12:32:54 +02:00
Sebdraven 8ae7210aef add onyphe full module 2018-06-20 11:07:33 +02:00
Sebdraven 023c35f5d8 add onyphe full module and code the stub 2018-06-14 16:47:11 +02:00
Sebdraven 14695bbeb9 correct codecov 2018-06-11 13:34:45 +02:00
Sebdraven 755d907580 pep 8 compliant 2018-06-11 13:21:21 +02:00
Sebdraven f6b8655f64 correct type of comments 2018-06-11 12:29:51 +02:00
Sebdraven 43402fde26 correct typo 2018-06-11 12:28:40 +02:00
Sebdraven e0631c9651 correct typo 2018-06-11 12:02:34 +02:00
Sebdraven 59b49f9d20 add domains forward 2018-06-11 12:00:46 +02:00
Sebdraven d9ee5286e3 add domains 2018-06-11 11:59:00 +02:00
Sebdraven 2e0e63fad6 add targeting os 2018-06-11 11:25:17 +02:00
Sebdraven 7580c63433 add category for AS number 2018-06-11 10:59:06 +02:00
Sebdraven f069cd9bf4 change keys 2018-06-11 10:56:40 +02:00
Sebdraven 0a543ca0d5 change type 2018-06-11 10:55:44 +02:00
Sebdraven ef035d051b add category 2018-06-11 10:54:06 +02:00
Sebdraven 735e626058 add as number with onyphe 2018-06-11 10:41:05 +02:00
Sebdraven 04032d110c add as number with onyphe 2018-06-08 18:31:08 +02:00
Sebdraven cad35b5332 error indentation 2018-06-08 18:11:04 +02:00
Sebdraven 3ec1535897 correct key in map result 2018-06-08 18:09:59 +02:00
Sebdraven f18f8fe05a correct a bug 2018-06-08 18:01:58 +02:00
Sebdraven 6eeca0fba1 add pastebin url imports 2018-06-08 17:53:50 +02:00
Sebdraven e6bac113ba add onyphe module 2018-06-08 16:38:41 +02:00
Andras Iklody 0b0f57b30c
Update countrycode.py 2018-06-06 08:31:41 +02:00
Alexandre Dulaunoy 2d9b0cd172
Merge branch 'master' of github.com:MISP/misp-modules 2018-05-29 21:59:25 +02:00
Alexandre Dulaunoy 9664127b85
add: new expansion module to check hashes against hashdd.com including NSLR dataset. 2018-05-29 21:54:22 +02:00
chrisr3d 2b509a2fd3
Updated delimiter finder function 2018-05-18 11:38:13 +02:00
chrisr3d 1fb72f3c7a
add: Added user config to specify if there is a header in the csv to import 2018-05-18 11:33:53 +02:00
chrisr3d dba8bd8c5b
fix: Avoid trying to build attributes with not intended fields 
- Previously: if the header field is not an attribute type, then
              it was added as an attribute field.
              PyMISP then used to skip it if needed

- Now: Those fields are discarded before they are put in an attribute
 2018-05-17 16:24:11 +02:00
chrisr3d c088b13f03
fix: Using userConfig to define the header instead of moduleconfig 2018-05-17 13:47:49 +02:00
Christophe Vandeplas 0593dbb408 ta import - more filter for pollution 2018-05-16 11:50:47 +02:00
Christophe Vandeplas 67cecc89d0 threatanalyzer_import - minor generic noise removal 2018-05-15 13:02:17 +02:00
Christophe Vandeplas 27a22e5d86 threatanalyzer_import - loads sample info + pollution fix 2018-05-03 09:42:38 +02:00
Christophe Vandeplas 370011c081 threatanalyzer_import - fix regkey issue 2018-05-02 12:43:34 +02:00
Nick Driver 252d190714
fix missing comma 
fix ip-dst and vulnerability input
 2018-03-30 14:27:37 -04:00
Koen Van Impe 6d23d4f4c7 Fix VMRay API access error 
hotfix for the "Unable to access VMRay API" error
 2018-03-30 15:11:25 +02:00
Fred Morris d0f618b648 Add exception blocks for query errors. 2018-03-08 15:26:39 -08:00
x41\x43 0436118747
Improving regex (validating e-mail) 
Line 48:
The previous regex ` ^[\w\.\+\-]+\@[\w]+\.[a-z]{2,3}$ ` matched only a small subset of valid e-mail address (e.g.: didn't match domain names longer than 3 chars or user@this-domain.de or user@multiple.level.dom) and needed to be with start (^) and end ($).
This ` [a-zA-Z0-9!#$%&'*+\/=?^_`{|}~-]+(?:\.[a-zA-Z0-9!#$%&'*+\/=?^_`{|}~-]+)*@(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]*[a-zA-Z0-9])?\.)+[a-zA-Z0-9](?:[a-zA-Z0-9-]*[a-zA-Z0-9])? ` is not perfect (e.g: can't match oriental chars), but imho is much more complete. 

Regex tested with several e-mail addresses with Python 3.6.4 and Python 2.7.14 on Linux 4.14.
 2018-03-06 18:12:36 +01:00
chrisr3d d885286792
Clarified functions arguments using a class 2018-03-05 19:59:30 +01:00
chrisr3d 4d7642ac91
add: Added Object References in the objects imported 2018-03-05 14:58:31 +01:00
chrisr3d 82fe8ba78c
fix: Fixed input & output of the module 2018-03-02 11:03:21 +01:00
chrisr3d 70436b7ddb Merge branch 'csvimport' of github.com:chrisr3d/misp-modules into goaml 2018-03-02 09:40:46 +01:00
chrisr3d c9ef578262
Removed print 2018-03-02 09:09:12 +01:00
chrisr3d 8d345d8cf5 Merge branch 'master' of github.com:MISP/misp-modules into csvimport 2018-03-02 09:05:46 +01:00
chrisr3d e6c55f5dde
fix: Fixed input & output of the module 
Also updated some functions
 2018-03-02 09:03:51 +01:00
chrisr3d 03d20856d9
add: added goamlimport 2018-02-28 22:46:39 +01:00
chrisr3d 323f71cdd3
Fixed some details about the module output 2018-02-28 17:41:45 +01:00
chrisr3d 8f5c08e2c6
Converting GoAML into MISPEvent 2018-02-28 15:07:55 +01:00
chrisr3d cad62464c5
Now parsing all the transaction attributes 2018-02-27 11:08:37 +01:00
chrisr3d a02dbd6a8d
fix: Fixed typo of the aml type for country codes 2018-02-26 18:52:28 +01:00
chrisr3d 478cd53912
add: Added dictionary to map aml types into MISP types 2018-02-26 18:13:43 +01:00
chrisr3d 5df2d309a0
typo 2018-02-26 15:58:53 +01:00
chrisr3d 81a6be17d3
chg: Structurded data 2018-02-26 11:47:35 +01:00
chrisr3d 359ac9100e
fix: typo in references mapping dictionary 2018-02-23 15:58:04 +01:00
Christian Studer 983b7da7b7
fix: Added an object checking 
- Checking if there are objects in the event, and then if there is at least 1 transaction object
- This prevents the module from crashing, but does not guaranty having a valid GoAML file (depending on objects and their relations)
 2018-02-22 16:55:52 +01:00
chrisr3d b2b0fccd47
fix: Added an object checking 
- Checking if there are objects in the event, and then
  if there is at least 1 transaction object
- This prevents the module from crashing, but does not
  guaranty having a valid GoAML file (depending on
  objects and their relations)
 2018-02-22 16:37:27 +01:00
chrisr3d 53b4a43448 Merge branch 'master' of github.com:chrisr3d/misp-modules into aml_import 2018-02-22 14:29:36 +01:00
chrisr3d c942013812
chg: Modified the mapping dictionary to support misp-objects updates 2018-02-22 01:23:08 +01:00
chrisr3d 5995458aab
fix: Added the moduleinfo field need to have MISP event in standard format 2018-02-21 17:14:26 +01:00
Alexandre Dulaunoy c3ac53a069
fix: goamlexport added 2018-02-20 17:18:36 +01:00
chrisr3d f361fb4ee3
Reading the entire document, to create a big dictionary containing the data, as a beginning 2018-02-20 17:00:13 +01:00
chrisr3d 02b8938b2a
typo 2018-02-20 16:57:27 +01:00
chrisr3d 11dddb974b Merge branch 'master' of github.com:MISP/misp-modules 2018-02-20 15:18:45 +01:00
chrisr3d eb9e06f1cc
explicit name 
Avoiding confusion with the coming import module for goaml
 2018-02-20 15:18:12 +01:00
Andras Iklody 978903f911
Quick fix to the invalid hash types offered on all returned hashes, hopefully fixes #162 2018-02-20 14:08:14 +01:00
chrisr3d 92ab1d5c23
Added "t_to" and "t_from" required fields: funds code & country 2018-02-14 21:30:48 +01:00
chrisr3d be1b541966
Added a required field & the latest attributes in transaction 2018-02-14 12:18:12 +01:00
chrisr3d 43e9010858
Added report expected information fields 2018-02-13 16:39:19 +01:00
chrisr3d d4538382d0
Simplified ObjectReference dictionary reading 2018-02-13 13:41:22 +01:00
chrisr3d b7098d1cff Merge branch 'master' of github.com:MISP/misp-modules 2018-02-13 11:58:56 +01:00
chrisr3d a97eeb44fe
Added some report information 
Also changed the ObjectReference parser to replace
all the if conditions by a dictionary reading
 2018-02-13 11:51:34 +01:00
Dennis Rand 43db92dbe6 Added Yara syntax validation expansion module 2018-02-12 19:11:54 +00:00
chrisr3d 8569c3d702
Suporting the recent objects added to misp-objects 
- Matching the aml documents structure
- Some parts of the document still need to be added
 2018-02-12 13:40:49 +01:00
chrisr3d 8983ebc4b2
wip: added location & signatory information 2018-02-05 15:51:03 +01:00
chrisr3d 54ebb8a96f Merge branch 'master' of github.com:MISP/misp-modules into test 2018-02-04 17:16:25 +01:00
Thomas Gardner 69d733bb35 added csvimport to __init__.py 2018-02-01 10:22:28 -07:00
chrisr3d 8dce7935ae
Outputting xml format 
Also mapping MISP and GoAML types
 2018-02-01 14:55:48 +01:00
chrisr3d 48869335ee
first tests for the GoAML export module 2018-01-31 18:09:45 +01:00
chrisr3d 71c00954d0
fix: Solved reading problems for some files 2018-01-30 11:20:28 +01:00
chrisr3d b2ec186ccb
Updated delimiter finder method 2018-01-29 17:04:32 +01:00
chrisr3d 529d22cca8
fix: skipping empty lines 2018-01-29 09:19:58 +01:00
chrisr3d 56cbd72b65
Fixed data treatment & other updates 2018-01-28 18:12:40 +01:00
chrisr3d 4d846f968f
Updated delimiter parsing & data reading functions 2018-01-26 17:11:01 +01:00
chrisr3d b9d72bb043
First version of csv import module 
- If more than 1 misp type is recognized, for each one an
  attribute is created

- Needs to have header set by user as parameters of the module atm

- Review needed to see the feasibility with fields that can create
  confusion and be interpreted both as misp type or attribute field
  (for instance comment is a misp type and an attribute field)
 2018-01-25 15:44:08 +01:00
Christophe Vandeplas 8a1a860cda added CrowdStrike Falcon Intel Indicators expansion module 2018-01-19 14:42:25 +01:00
chrisr3d d045cf7d5f
chg: Modified output format 2018-01-16 19:46:52 +01:00
chrisr3d dcab9aa150 Merge github.com:MISP/misp-modules 2018-01-16 17:15:36 +01:00
Alexandre Dulaunoy c3823b74cf
Merge pull request #149 from cvandeplas/master 
Added ThreatAnalyzer sandbox import
 2018-01-16 17:11:38 +01:00
chrisr3d 18523c4ada
Check an IPv4 address against known RBLs 2018-01-16 17:08:44 +01:00
Christophe Vandeplas 0be1886444
fix farsight_passivedns - rdata 404 not found 2018-01-16 15:13:17 +01:00
Christophe Vandeplas 46975f4f16 Added ThreatAnalyzer sandbox import 
Experimental module - some parts should be migrated to
 2018-01-16 11:05:26 +01:00
Alexandre Dulaunoy 5c4df3075e
Fix the __init__ import 2018-01-08 20:31:26 +01:00
Robert Nixon 85f1a9bd91
Update threatStream_misp_export.py 2018-01-08 12:09:23 -05:00
Robert Nixon 1d2f3d9c3c
Updated __init__.py 
Added reference to new ThreatStream export module
 2018-01-08 11:03:42 -05:00
Robert Nixon 49d5520fa3
Added threatStream_misp_export.py 2018-01-08 11:01:16 -05:00
Christophe Vandeplas 4cdb143733 fixes missing init file in dnsdb library folder 2017-12-06 09:23:44 +01:00
Christophe Vandeplas 0ec8339d7a New Farsight DNSDB Passive DNS expansion module 2017-12-05 16:41:41 +01:00
Raphaël Vinot 02253e5a87 Merge branch 'master' of github.com:MISP/misp-modules 2017-11-20 14:57:18 +01:00
Jericho 32958324ca
minor touch-ups on error messages for user friendliness 2017-11-16 23:04:41 -07:00
Koen Van Impe 74e660d61b VulnDB Queries 
Search on CVE at https://vulndb.cyberriskanalytics.com/
    https://www.riskbasedsecurity.com/
Get extended CVE info, links + CPE
 2017-11-06 14:23:03 +01:00
Raphaël Vinot 37d9b3831c Add quick and dirty pdf export 2017-10-26 16:54:20 -04:00
Raphaël Vinot c09135d251 Merge pull request #139 from Rafiot/master 
fix: OpenIOC importer
 2017-10-25 11:41:46 -04:00
Raphaël Vinot 951a0f974b fix: OpenIOC importer 2017-10-25 11:27:59 -04:00
Alexandre Dulaunoy 03baa0b84d
fix: #137 when a CVE is not found, a return message is given 2017-10-21 19:52:19 +02:00
Thomas Gardner 72c52da7ed added threat_connect_export to export_mod.__init__ 2017-08-06 08:15:17 -06:00
Thomas Gardner 529719d9d8 added threat_connect_export.py 2017-08-03 16:21:26 -06:00
Raphaël Vinot 4c2cda9903 Merge pull request #129 from seamustuohy/utf_hate 
Added support for malformed internationalized email headers
 2017-07-18 10:06:08 +02:00
Chris Doman c4fe78b39d Add AlienVault OTX and ThreatCrowd Expansions 2017-07-11 18:16:45 +01:00
seamus tuohy 40c71af637 Added support for malformed internationalized email headers 
When an emails contains headers that use Unicode without properly crafing
them to comform to RFC-6323 the email import module would crash.
(See issue #119 & issue #93)

To address this I have added additional layers of encoding/decoding to
any possibly internationalized email headers. This decodes properly
formed and malformed UTF-8, UTF-16, and UTF-32 headers appropriately.
When an unknown encoding is encountered it is returned as an 'encoded-word'
per RFC2047.

This commit also adds unit-tests that tests properly formed and malformed
UTF-8, UTF-16, UTF-32, and CJK encoded strings in all header fields; UTF-8,
UTF-16, and UTF-32 encoded message bodies; and emoji testing for headers
and attachment file names.
 2017-07-02 18:03:14 -04:00
Raphaël Vinot c42c8a800e Update travis, fix open ioc import 2017-05-24 07:39:18 +02:00
Tristan METAYER 75c02058e6 replace tab by space 2017-05-11 09:56:43 +02:00
Tristan METAYER ba1d715ad1 Add a field for user to add tag for this import 2017-05-11 09:54:25 +02:00
Tristan METAYER 96f9cb4699 typo correction 2017-05-02 15:07:33 +02:00
Tristan METAYER 4ef7261168 Add user config to not add file as attachement in a box 2017-05-02 15:04:40 +02:00
Tristan METAYER 79f48eccfe If filename add iocfilename as attachment 2017-05-02 14:41:22 +02:00
Alexandre Dulaunoy 3cb12d6962 Merge pull request #118 from truckydev/master 
Add indent field for export
 2017-04-23 12:21:16 +02:00
Tristan METAYER 24c51a6e21 Add indent field for export 2017-04-21 15:53:48 +02:00
Hannah Ward 648c6414c3
fix: Use the proper formatting method and not the horrible % one 2017-03-08 16:35:03 +00:00
kx499 aa3a11cd5f bug fixes 2017-03-08 04:08:23 +01:00
kx499 31a8fb0fe4 threatminer initial commit 2017-03-06 21:36:00 -05:00
Raphaël Vinot 44867b2adc Cosmetic changes 2017-03-05 18:59:36 +01:00
Raphaël Vinot ad49fd3819 Merge pull request #111 from kx499/master 
Handful of changes to VirusTotal module
 2017-03-05 18:31:50 +01:00
kx499 3ecd095d1e bug fixes, tweaks, and python3 learning curve :) 2017-03-04 03:10:45 +01:00
kx499 01fdf3e52b Initial commit of IPRep module 2017-03-03 15:55:52 -05:00
kx499 bc1eab3520 fixed spacing, addressed error handling for public api, added subdomains, and added context comment 2017-02-28 22:04:24 -05:00
Raphaël Vinot c508e60f65 Add OpenIOC import module 2017-02-27 13:32:31 +01:00
Tristan METAYER 20cb534203 Exclude internal reference 2017-02-21 17:12:17 +01:00
Tristan METAYER dd2646a0f4 Add lite Export module 2017-02-21 16:48:09 +01:00
rmarsollier b5b7e09ef4 Some improvements of virustotal plugin 2017-02-10 14:16:39 +01:00
Joerg Stephan de3495ea6c passed local run check 2017-02-01 14:05:29 +01:00
Joerg Stephan 68250094ff v1 2017-01-31 16:57:16 +01:00
Joerg Stephan dad73feaa4 python3 changes 2017-01-31 16:34:41 +01:00
Joerg Stephan 3590504821 XForce Exchange v1 (alpha) 2017-01-21 23:31:19 +01:00
Richard van den Berg 3a4c540a81 Updated description to reflect merging use case 2017-01-11 10:08:35 +01:00
Richard van den Berg 50bae1f549 Simple import module to import MISP JSON format 2017-01-11 10:08:35 +01:00
seamus tuohy 83a9d695ea Email import no longer unzips major compressed text document formats. 
Let this commit serve as a warning about the perils of duck typing.
Word documents (docx,odt,etc) were being uncompressed when they were
attached to emails. The email importer now checks a list of well known
extensions and will not attempt to unzip them.

It is stuck using a list of extensions instead of using file magic because
many of these formats produce an application/zip mimetype when scanned.
 2017-01-10 09:55:33 -05:00
Raphaël Vinot 1051e2210b Keep zip content as binary 2017-01-07 19:30:00 -05:00
Raphaël Vinot 9f84db3659 Fix tests, cleanup 2017-01-07 18:36:08 -05:00
Raphaël Vinot 2db845c45c Improve support of email attachments 
Related to #90
 2017-01-07 14:39:52 -05:00
Hannah Ward 727f302dd1 Standardised key checking 2017-01-07 10:38:28 -05:00
Hannah Ward 20fd05a231 Fixed checking for submission_names in VT JSON 2017-01-07 10:37:57 -05:00
CheYenBzh d7b33532eb Update virustotal.py 2017-01-07 10:37:47 -05:00
Raphaël Vinot b51806ac9f Improve support of email importer if headers are missing 
Fix #88
 2017-01-07 10:25:38 -05:00
Raphaël Vinot 02f5e95a98 Fix python 3.6 support 2017-01-06 20:36:09 -05:00
Raphaël Vinot 329586768b Make PEP8 happy 2017-01-06 20:10:44 -05:00
Raphaël Vinot 7a9774bff7 Add email_import in the modules loaded by default 2017-01-06 19:23:23 -05:00
Raphaël Vinot 93a49c3c1d Make PEP8 happy 2017-01-06 19:01:19 -05:00
Raphaël Vinot 3f83357a2d Fix failing test (bug in the mail parser?) 2017-01-06 18:56:29 -05:00
seamus tuohy 1a7973bc06 Add additional email parsing and tests 
Added additional attribute parsing and corresponding unit-tests.
E-mail attachment and url extraction added in this commit. This includes
unpacking zipfiles and simple password cracking of encrypted zipfiles.
 2017-01-04 10:21:36 -08:00
seamus tuohy 0ff270a3be Fixed basic errors 2016-12-26 14:33:10 -08:00
seamus tuohy 08261366b7 Merged with current master 2016-12-26 14:17:20 -08:00
seamus tuohy 86ae72c444 Added attachment and url support 2016-12-26 13:55:54 -08:00
Raphaël Vinot 9bf1c936cf Do not crash if the dat file is not available 2016-12-16 15:22:16 +01:00
Raphaël Vinot 064c3e3649 Fix path to config file 2016-12-16 15:14:48 +01:00
Raphaël Vinot 29bedc7faa Merge branch 'master' of https://github.com/amuehlem/misp-modules into amuehlem-master 2016-12-16 15:05:45 +01:00
Raphaël Vinot 60d3e0a1ac Better error reporting 2016-12-16 12:02:28 +01:00
Raphaël Vinot ffc0a97126 Catch exception 2016-12-16 11:52:51 +01:00
Raphaël Vinot 467e50327d Add reverse lookup 2016-12-16 11:22:22 +01:00
Raphaël Vinot 4a8ccb54fb Refactoring of domaintools expansion module 2016-12-15 16:49:56 +01:00
Ubuntu b76f59edcb Added cuckooimport.py 2016-12-07 16:36:31 +00:00
Andreas Muehlemann cc58b05d6e added empty line to end of config file 2016-12-07 17:28:16 +01:00
Andreas Muehlemann 98a27ac3ff removed DEFAULT section from configfile 2016-12-07 16:36:02 +01:00
Andreas Muehlemann 6853d67a43 fixed more typos 2016-12-07 16:13:46 +01:00
Andreas Muehlemann 6dcc77ba5d fixed typo 2016-12-07 15:48:08 +01:00
Andreas Muehlemann a95af26424 changed configparser from python2 to python3 2016-12-07 15:30:49 +01:00
Andreas Muehlemann 1e1796b414 updated missing parenthesis 2016-12-07 15:19:54 +01:00
Andreas Muehlemann bb62394c1e Merge branch 'geoip_country' 2016-12-07 14:54:33 +01:00
Andreas Muehlemann d09c2f3d44 removed unneeded config option for misp 2016-12-07 14:29:11 +01:00
Andreas Muehlemann 6ea7acc5e4 removed debug message 2016-12-07 14:28:27 +01:00
Andreas Muehlemann f8c7271467 added config option to geoip_country.py 2016-12-07 14:18:21 +01:00
Raphaël Vinot ac33940628 Merge pull request #75 from Rafiot/domtools 
Add Domain Tools module
 2016-12-01 17:52:04 +01:00
Raphaël Vinot 2e3119b5f4 Add domaintools to the import list 2016-12-01 17:36:40 +01:00
Raphaël Vinot 0f8fa4aaec Fix Typo 2016-12-01 16:44:29 +01:00
Raphaël Vinot 17205a1913 Add domain profile and reputation 2016-12-01 16:41:50 +01:00
Raphaël Vinot 7db1216efb Add more comments 2016-12-01 13:45:14 +01:00
Raphaël Vinot 9dbd241e63 fix typo 2016-12-01 12:14:16 +01:00
Raphaël Vinot 6db5436c62 remove json.dumps 2016-12-01 11:54:04 +01:00
Raphaël Vinot afd8b71349 Avoid passing None in comments 2016-12-01 10:26:40 +01:00
Raphaël Vinot 7c6153478e Add comments to fields when possible 2016-11-30 18:09:11 +01:00
Koen Van Impe 077470b8ed Merge remote-tracking branch 'MISP/master' 2016-11-30 13:06:43 +01:00
Raphaël Vinot 48d38c2821 Add initial Domain Tools module 2016-11-28 18:12:31 +01:00
Koen Van Impe cb29506640 Extra VTI detections 2016-11-27 22:42:43 +01:00
Raphaël Vinot 79a0b9e667 Merge pull request #73 from FloatingGhost/master 
Use SpooledTemp, not NamedTemp file
 2016-11-21 16:37:11 +01:00
Hannah Ward 1f49f36205
Removed unneeded modules 2016-11-21 13:05:07 +00:00
Hannah Ward 0dfea44001
Use SpooledTemp, not NamedTemp file 2016-11-21 11:57:04 +00:00
Raphaël Vinot e78e008aa3 Merge pull request #72 from FloatingGhost/master 
Migrated stiximport to use misp-stix-converter
 2016-11-21 12:06:16 +01:00
Hannah Ward c567d1e6f2
Moved to misp_stix_converter 2016-11-21 10:59:30 +00:00
Koen Van Impe 3253d92b42 Submit malware samples 
_submit now includes malware samples (zipped content from misp)
_import checks when no vti_results are returned + bugfix
 2016-11-18 18:23:52 +01:00
Raphaël Vinot 5624104b77 Fix STIX import module 2016-11-15 16:47:17 +01:00
Raphaël Vinot c676587461 Multiple clanges in the vmray modules. 
* Generic fix to load modules requiring a local library
* Fix python3 support
* PEP8 related cleanups
 2016-11-15 16:43:11 +01:00
Koen Van Impe adda9562c0 VMRay Import & Submit module 
* First commit
* No support for archives (yet) submit
 2016-11-13 21:43:59 +01:00
seamus tuohy 5033b1a9ca Added email meta-data import module. 
This email meta-data import module collects basic meta-data from an e-mail
and populates an event with it. It populates the email subject, source
addresses, destination addresses, subject, and any attachment file names.
This commit also contains unit-tests for this module as well as updates to
the readme. Readme updates are additions aimed to make it easier for
outsiders to build modules.
 2016-10-22 17:13:20 -04:00
Roman Graf 03b6fd7b74 label replaced by text, which is existing attribute 2016-10-11 14:48:59 +02:00
Alexandre Dulaunoy d7137221db Chg: wikidata module added 2016-10-07 16:21:54 +02:00
Roman Graf d4370fc0e3 Added expansion for Wikidata. Analyst can query Wikidata by label to get additional information for particular term. 2016-10-07 12:57:01 +02:00
Andreas Muehlemann a568d1a1b3 updated geoip_country to __init__.py 2016-09-28 14:06:18 +02:00
Andreas Muehlemann 4bc76acd37 added geoip_country.py 2016-09-28 14:05:43 +02:00
Andreas Muehlemann 985f9de800 added new module reversedns.py, added reversedns to __init__.py 2016-09-22 11:42:52 +02:00
Raphaël Vinot a0cce11964 Dump host info as text 2016-09-15 15:59:08 +02:00
Raphaël Vinot ea2f106b00 Fix typo 2016-09-15 15:32:13 +02:00
Raphaël Vinot 43834b6d51 Add simple Shodan module 2016-09-15 15:11:04 +02:00
Alexandre Dulaunoy fb7411aa32 Merge pull request #49 from FloatingGhost/master 
Removed useless pickle storage of stiximport
 2016-09-06 15:22:00 +02:00
Hannah Ward 0521833c65
Removed useless pickle storage of stiximport 2016-09-06 14:12:09 +01:00
Alexandre Dulaunoy a9b95095c0 cef_export module added 2016-09-01 20:22:33 +02:00
Alexandre Dulaunoy 2df8bf970e Merge pull request #47 from FloatingGhost/CEF_Export 
CEF export, fixes in CountryCode, virustotal
 2016-09-01 19:39:16 +02:00
Hannah Ward 4f923d6606
Removed silly subdomain module 2016-09-01 16:14:25 +01:00
Hannah Ward 4e3300d66c
Added CEF export module 2016-08-22 14:18:19 +01:00
Hannah Ward a492d975c4
Now searches within observable_compositions 2016-08-19 17:21:12 +01:00
Hannah Ward 9db9247e55
Removed calls to print 2016-08-17 13:04:30 +01:00
Hannah Ward 232014f221
Added virustotal tests 2016-08-17 13:01:11 +01:00
Alexandre Dulaunoy 062f2dfd30 New modules added to __init__ 2016-08-17 11:27:07 +02:00
Hannah Ward 4ba86d4fa3
CountryCode JSON now is only grabbed once per server run 2016-08-17 09:51:16 +01:00
Hannah Ward 042bf2bb2f
Added virustotal module 2016-08-17 09:30:15 +01:00
Hannah Ward 0f9221229a
Improved virustotal module 2016-08-15 11:09:40 +01:00
Hannah Ward 917c95cad5
Added countrycode, working on virustotal 2016-08-12 17:40:00 +01:00
Hannah Ward 4f5059fca4
Added lookup by country code 2016-08-12 14:45:28 +01:00
Hannah Ward 6db269f965
stiximport now uses temporary files to store stix data. 
Set max size in config, in bytes
 2016-08-12 13:53:23 +01:00
Raphaël Vinot c6fccf1b7e Make PEP8 happy \o/ 2016-08-12 14:09:59 +02:00
Raphaël Vinot 91675a635c Move stiximport.py to misp_modules/modules/import_mod/ 2016-08-12 14:08:47 +02:00
Hannah Ward 6f770ad0c7
Merge branch 'master' of https://github.com/MISP/misp-modules 2016-08-12 12:35:47 +01:00
Hannah Ward 2f6054e97f Merge branch 'stix_import' 2016-08-12 12:17:40 +01:00
Hannah Ward c02a452c05
added tests, also disregards related_observables. Because they're useless 2016-08-12 12:16:49 +01:00
Hannah Ward a34014e245
Fixed observables within an indicator not being added 2016-08-12 11:56:48 +01:00
Raphaël Vinot 59b16950f7 Remove bin script, use cleaner way. Fix last commit. 2016-08-12 12:35:33 +02:00
Hannah Ward faddf8378e
Stiximport will now consume campaigns 2016-08-12 11:34:43 +01:00
Hannah Ward 598a030962
stiximport will now identify file hashes 2016-08-12 11:22:42 +01:00
Hannah Ward 3f7cdad0c3
Threat actors now get imported by stix 2016-08-12 10:06:53 +01:00
Alexandre Dulaunoy e7c6c36089 Fix: module_config should be set as introspection relies on it 2016-08-12 10:55:14 +02:00
Hannah Ward c106aa662b
Added docs to stiximport 2016-08-11 16:37:29 +01:00
Hannah Ward b654a9743b
Added stix import -- works for IPs/Domains 2016-08-11 16:33:02 +01:00
iglocska 6116c017c1 Update to the DNS module to support domain|ip 2016-08-10 17:11:46 +02:00
iglocska c3a3d68e43 Small change to the skeleton export 2016-08-10 16:47:55 +02:00
Iglocska eea62db199 Added test export module 2016-08-05 21:58:24 +02:00
Alexandre Dulaunoy bf035e148c Merge branch 'import-test' of github.com:MISP/misp-modules into import-test 2016-08-04 18:55:17 +02:00
Alexandre Dulaunoy 27ddbd9b92 Fix: types array 2016-08-04 18:54:21 +02:00
Raphaël Vinot b3a322a178 Pass the server port as integer to the uwhois client 2016-08-04 17:44:40 +02:00
Raphaël Vinot f72534c785 Add whois module 2016-08-04 17:23:23 +02:00
Alexandre Dulaunoy f97c5d62d6 First version of an Optical Character Recognition (OCR) module for MISP 2016-08-04 14:32:50 +02:00
Iglocska 2b84e47f34 first version of the import skeleton 2016-08-04 09:12:10 +02:00
Iglocska 3fb62fac70 Added simple import skeleton 2016-08-04 08:00:09 +02:00
Raphaël Vinot 22eaba6ab6 Make sure misp-modules can be launched from anywhere 2016-06-23 19:51:13 +09:00