MISP
/
misp-modules
mirror of https://github.com/MISP/misp-modules
2
0
Fork 0
Code Issues Releases Wiki Activity
583 Commits
7 Branches
55 Tags
20 MiB
Commit Graph

294 Commits (f3962d2d0599ef689d2d52b458c36c9cf123fea5)

Author SHA1 Message Date
seamus tuohy 40c71af637 Added support for malformed internationalized email headers 
When an emails contains headers that use Unicode without properly crafing
them to comform to RFC-6323 the email import module would crash.
(See issue #119 & issue #93)

To address this I have added additional layers of encoding/decoding to
any possibly internationalized email headers. This decodes properly
formed and malformed UTF-8, UTF-16, and UTF-32 headers appropriately.
When an unknown encoding is encountered it is returned as an 'encoded-word'
per RFC2047.

This commit also adds unit-tests that tests properly formed and malformed
UTF-8, UTF-16, UTF-32, and CJK encoded strings in all header fields; UTF-8,
UTF-16, and UTF-32 encoded message bodies; and emoji testing for headers
and attachment file names.
 2017-07-02 18:03:14 -04:00
Raphaël Vinot c42c8a800e Update travis, fix open ioc import 2017-05-24 07:39:18 +02:00
Tristan METAYER 75c02058e6 replace tab by space 2017-05-11 09:56:43 +02:00
Tristan METAYER ba1d715ad1 Add a field for user to add tag for this import 2017-05-11 09:54:25 +02:00
Tristan METAYER 96f9cb4699 typo correction 2017-05-02 15:07:33 +02:00
Tristan METAYER 4ef7261168 Add user config to not add file as attachement in a box 2017-05-02 15:04:40 +02:00
Tristan METAYER 79f48eccfe If filename add iocfilename as attachment 2017-05-02 14:41:22 +02:00
Alexandre Dulaunoy 3cb12d6962 Merge pull request #118 from truckydev/master 
Add indent field for export
 2017-04-23 12:21:16 +02:00
Tristan METAYER 24c51a6e21 Add indent field for export 2017-04-21 15:53:48 +02:00
Hannah Ward 648c6414c3
fix: Use the proper formatting method and not the horrible % one 2017-03-08 16:35:03 +00:00
kx499 aa3a11cd5f bug fixes 2017-03-08 04:08:23 +01:00
kx499 31a8fb0fe4 threatminer initial commit 2017-03-06 21:36:00 -05:00
Raphaël Vinot 44867b2adc Cosmetic changes 2017-03-05 18:59:36 +01:00
Raphaël Vinot ad49fd3819 Merge pull request #111 from kx499/master 
Handful of changes to VirusTotal module
 2017-03-05 18:31:50 +01:00
kx499 3ecd095d1e bug fixes, tweaks, and python3 learning curve :) 2017-03-04 03:10:45 +01:00
kx499 01fdf3e52b Initial commit of IPRep module 2017-03-03 15:55:52 -05:00
kx499 bc1eab3520 fixed spacing, addressed error handling for public api, added subdomains, and added context comment 2017-02-28 22:04:24 -05:00
Raphaël Vinot c508e60f65 Add OpenIOC import module 2017-02-27 13:32:31 +01:00
Tristan METAYER 20cb534203 Exclude internal reference 2017-02-21 17:12:17 +01:00
Tristan METAYER dd2646a0f4 Add lite Export module 2017-02-21 16:48:09 +01:00
rmarsollier b5b7e09ef4 Some improvements of virustotal plugin 2017-02-10 14:16:39 +01:00
Joerg Stephan de3495ea6c passed local run check 2017-02-01 14:05:29 +01:00
Joerg Stephan 68250094ff v1 2017-01-31 16:57:16 +01:00
Joerg Stephan dad73feaa4 python3 changes 2017-01-31 16:34:41 +01:00
Joerg Stephan 3590504821 XForce Exchange v1 (alpha) 2017-01-21 23:31:19 +01:00
Richard van den Berg 3a4c540a81 Updated description to reflect merging use case 2017-01-11 10:08:35 +01:00
Richard van den Berg 50bae1f549 Simple import module to import MISP JSON format 2017-01-11 10:08:35 +01:00
seamus tuohy 83a9d695ea Email import no longer unzips major compressed text document formats. 
Let this commit serve as a warning about the perils of duck typing.
Word documents (docx,odt,etc) were being uncompressed when they were
attached to emails. The email importer now checks a list of well known
extensions and will not attempt to unzip them.

It is stuck using a list of extensions instead of using file magic because
many of these formats produce an application/zip mimetype when scanned.
 2017-01-10 09:55:33 -05:00
Raphaël Vinot 1051e2210b Keep zip content as binary 2017-01-07 19:30:00 -05:00
Raphaël Vinot 9f84db3659 Fix tests, cleanup 2017-01-07 18:36:08 -05:00
Raphaël Vinot 2db845c45c Improve support of email attachments 
Related to #90
 2017-01-07 14:39:52 -05:00
Hannah Ward 727f302dd1 Standardised key checking 2017-01-07 10:38:28 -05:00
Hannah Ward 20fd05a231 Fixed checking for submission_names in VT JSON 2017-01-07 10:37:57 -05:00
CheYenBzh d7b33532eb Update virustotal.py 2017-01-07 10:37:47 -05:00
Raphaël Vinot b51806ac9f Improve support of email importer if headers are missing 
Fix #88
 2017-01-07 10:25:38 -05:00
Raphaël Vinot 02f5e95a98 Fix python 3.6 support 2017-01-06 20:36:09 -05:00
Raphaël Vinot 329586768b Make PEP8 happy 2017-01-06 20:10:44 -05:00
Raphaël Vinot 7a9774bff7 Add email_import in the modules loaded by default 2017-01-06 19:23:23 -05:00
Raphaël Vinot 93a49c3c1d Make PEP8 happy 2017-01-06 19:01:19 -05:00
Raphaël Vinot 3f83357a2d Fix failing test (bug in the mail parser?) 2017-01-06 18:56:29 -05:00
seamus tuohy 1a7973bc06 Add additional email parsing and tests 
Added additional attribute parsing and corresponding unit-tests.
E-mail attachment and url extraction added in this commit. This includes
unpacking zipfiles and simple password cracking of encrypted zipfiles.
 2017-01-04 10:21:36 -08:00
seamus tuohy 0ff270a3be Fixed basic errors 2016-12-26 14:33:10 -08:00
seamus tuohy 08261366b7 Merged with current master 2016-12-26 14:17:20 -08:00
seamus tuohy 86ae72c444 Added attachment and url support 2016-12-26 13:55:54 -08:00
Raphaël Vinot 9bf1c936cf Do not crash if the dat file is not available 2016-12-16 15:22:16 +01:00
Raphaël Vinot 064c3e3649 Fix path to config file 2016-12-16 15:14:48 +01:00
Raphaël Vinot 29bedc7faa Merge branch 'master' of https://github.com/amuehlem/misp-modules into amuehlem-master 2016-12-16 15:05:45 +01:00
Raphaël Vinot 60d3e0a1ac Better error reporting 2016-12-16 12:02:28 +01:00
Raphaël Vinot ffc0a97126 Catch exception 2016-12-16 11:52:51 +01:00
Raphaël Vinot 467e50327d Add reverse lookup 2016-12-16 11:22:22 +01:00
Raphaël Vinot 4a8ccb54fb Refactoring of domaintools expansion module 2016-12-15 16:49:56 +01:00
Ubuntu b76f59edcb Added cuckooimport.py 2016-12-07 16:36:31 +00:00
Andreas Muehlemann cc58b05d6e added empty line to end of config file 2016-12-07 17:28:16 +01:00
Andreas Muehlemann 98a27ac3ff removed DEFAULT section from configfile 2016-12-07 16:36:02 +01:00
Andreas Muehlemann 6853d67a43 fixed more typos 2016-12-07 16:13:46 +01:00
Andreas Muehlemann 6dcc77ba5d fixed typo 2016-12-07 15:48:08 +01:00
Andreas Muehlemann a95af26424 changed configparser from python2 to python3 2016-12-07 15:30:49 +01:00
Andreas Muehlemann 1e1796b414 updated missing parenthesis 2016-12-07 15:19:54 +01:00
Andreas Muehlemann bb62394c1e Merge branch 'geoip_country' 2016-12-07 14:54:33 +01:00
Andreas Muehlemann d09c2f3d44 removed unneeded config option for misp 2016-12-07 14:29:11 +01:00
Andreas Muehlemann 6ea7acc5e4 removed debug message 2016-12-07 14:28:27 +01:00
Andreas Muehlemann f8c7271467 added config option to geoip_country.py 2016-12-07 14:18:21 +01:00
Raphaël Vinot ac33940628 Merge pull request #75 from Rafiot/domtools 
Add Domain Tools module
 2016-12-01 17:52:04 +01:00
Raphaël Vinot 2e3119b5f4 Add domaintools to the import list 2016-12-01 17:36:40 +01:00
Raphaël Vinot 0f8fa4aaec Fix Typo 2016-12-01 16:44:29 +01:00
Raphaël Vinot 17205a1913 Add domain profile and reputation 2016-12-01 16:41:50 +01:00
Raphaël Vinot 7db1216efb Add more comments 2016-12-01 13:45:14 +01:00
Raphaël Vinot 9dbd241e63 fix typo 2016-12-01 12:14:16 +01:00
Raphaël Vinot 6db5436c62 remove json.dumps 2016-12-01 11:54:04 +01:00
Raphaël Vinot afd8b71349 Avoid passing None in comments 2016-12-01 10:26:40 +01:00
Raphaël Vinot 7c6153478e Add comments to fields when possible 2016-11-30 18:09:11 +01:00
Koen Van Impe 077470b8ed Merge remote-tracking branch 'MISP/master' 2016-11-30 13:06:43 +01:00
Raphaël Vinot 48d38c2821 Add initial Domain Tools module 2016-11-28 18:12:31 +01:00
Koen Van Impe cb29506640 Extra VTI detections 2016-11-27 22:42:43 +01:00
Raphaël Vinot 79a0b9e667 Merge pull request #73 from FloatingGhost/master 
Use SpooledTemp, not NamedTemp file
 2016-11-21 16:37:11 +01:00
Hannah Ward 1f49f36205
Removed unneeded modules 2016-11-21 13:05:07 +00:00
Hannah Ward 0dfea44001
Use SpooledTemp, not NamedTemp file 2016-11-21 11:57:04 +00:00
Raphaël Vinot e78e008aa3 Merge pull request #72 from FloatingGhost/master 
Migrated stiximport to use misp-stix-converter
 2016-11-21 12:06:16 +01:00
Hannah Ward c567d1e6f2
Moved to misp_stix_converter 2016-11-21 10:59:30 +00:00
Koen Van Impe 3253d92b42 Submit malware samples 
_submit now includes malware samples (zipped content from misp)
_import checks when no vti_results are returned + bugfix
 2016-11-18 18:23:52 +01:00
Raphaël Vinot 5624104b77 Fix STIX import module 2016-11-15 16:47:17 +01:00
Raphaël Vinot c676587461 Multiple clanges in the vmray modules. 
* Generic fix to load modules requiring a local library
* Fix python3 support
* PEP8 related cleanups
 2016-11-15 16:43:11 +01:00
Koen Van Impe adda9562c0 VMRay Import & Submit module 
* First commit
* No support for archives (yet) submit
 2016-11-13 21:43:59 +01:00
seamus tuohy 5033b1a9ca Added email meta-data import module. 
This email meta-data import module collects basic meta-data from an e-mail
and populates an event with it. It populates the email subject, source
addresses, destination addresses, subject, and any attachment file names.
This commit also contains unit-tests for this module as well as updates to
the readme. Readme updates are additions aimed to make it easier for
outsiders to build modules.
 2016-10-22 17:13:20 -04:00
Roman Graf 03b6fd7b74 label replaced by text, which is existing attribute 2016-10-11 14:48:59 +02:00
Alexandre Dulaunoy d7137221db Chg: wikidata module added 2016-10-07 16:21:54 +02:00
Roman Graf d4370fc0e3 Added expansion for Wikidata. Analyst can query Wikidata by label to get additional information for particular term. 2016-10-07 12:57:01 +02:00
Andreas Muehlemann a568d1a1b3 updated geoip_country to __init__.py 2016-09-28 14:06:18 +02:00
Andreas Muehlemann 4bc76acd37 added geoip_country.py 2016-09-28 14:05:43 +02:00
Andreas Muehlemann 985f9de800 added new module reversedns.py, added reversedns to __init__.py 2016-09-22 11:42:52 +02:00
Raphaël Vinot a0cce11964 Dump host info as text 2016-09-15 15:59:08 +02:00
Raphaël Vinot ea2f106b00 Fix typo 2016-09-15 15:32:13 +02:00
Raphaël Vinot 43834b6d51 Add simple Shodan module 2016-09-15 15:11:04 +02:00
Alexandre Dulaunoy fb7411aa32 Merge pull request #49 from FloatingGhost/master 
Removed useless pickle storage of stiximport
 2016-09-06 15:22:00 +02:00
Hannah Ward 0521833c65
Removed useless pickle storage of stiximport 2016-09-06 14:12:09 +01:00
Alexandre Dulaunoy a9b95095c0 cef_export module added 2016-09-01 20:22:33 +02:00
Alexandre Dulaunoy 2df8bf970e Merge pull request #47 from FloatingGhost/CEF_Export 
CEF export, fixes in CountryCode, virustotal
 2016-09-01 19:39:16 +02:00
Hannah Ward 4f923d6606
Removed silly subdomain module 2016-09-01 16:14:25 +01:00
Raphaël Vinot c69fae087c Add timeout for the modules, cleanup. 2016-08-25 17:36:28 +02:00
Raphaël Vinot 1034f73479 Fix python 3.3 and 3.4 2016-08-24 10:24:42 +02:00
Raphaël Vinot c822c2df9c Make misp-modules really asynchronous 2016-08-24 00:22:03 +02:00
Raphaël Vinot d6388e1c52 Improve tornado parallel 2016-08-23 18:02:29 +02:00
Hannah Ward 4e3300d66c
Added CEF export module 2016-08-22 14:18:19 +01:00
Alexandre Dulaunoy 6ba2731eb5 coroutine decorator added to post handler 2016-08-21 10:21:00 +02:00
Hannah Ward a492d975c4
Now searches within observable_compositions 2016-08-19 17:21:12 +01:00
Hannah Ward 9db9247e55
Removed calls to print 2016-08-17 13:04:30 +01:00
Hannah Ward 232014f221
Added virustotal tests 2016-08-17 13:01:11 +01:00
Alexandre Dulaunoy bf29e30e4b -d option added - enabling debug on queried modules 2016-08-17 13:42:58 +02:00
Alexandre Dulaunoy 062f2dfd30 New modules added to __init__ 2016-08-17 11:27:07 +02:00
Hannah Ward 4ba86d4fa3
CountryCode JSON now is only grabbed once per server run 2016-08-17 09:51:16 +01:00
Hannah Ward 042bf2bb2f
Added virustotal module 2016-08-17 09:30:15 +01:00
Hannah Ward 393b637514 Merge branch 'master' of https://github.com/MISP/misp-modules 2016-08-15 11:11:28 +01:00
Hannah Ward 0f9221229a
Improved virustotal module 2016-08-15 11:09:40 +01:00
Hannah Ward 917c95cad5
Added countrycode, working on virustotal 2016-08-12 17:40:00 +01:00
Hannah Ward 4f5059fca4
Added lookup by country code 2016-08-12 14:45:28 +01:00
Alexandre Dulaunoy d499ac0ce6 Merge pull request #44 from Rafiot/travis 
Add coverage, update logging
 2016-08-12 15:20:26 +02:00
Raphaël Vinot b24b16b30a Add coverage, update logging 2016-08-12 15:15:38 +02:00
Hannah Ward 6db269f965
stiximport now uses temporary files to store stix data. 
Set max size in config, in bytes
 2016-08-12 13:53:23 +01:00
Raphaël Vinot c6fccf1b7e Make PEP8 happy \o/ 2016-08-12 14:09:59 +02:00
Raphaël Vinot 91675a635c Move stiximport.py to misp_modules/modules/import_mod/ 2016-08-12 14:08:47 +02:00
Hannah Ward 6f770ad0c7
Merge branch 'master' of https://github.com/MISP/misp-modules 2016-08-12 12:35:47 +01:00
Hannah Ward 2f6054e97f Merge branch 'stix_import' 2016-08-12 12:17:40 +01:00
Hannah Ward c02a452c05
added tests, also disregards related_observables. Because they're useless 2016-08-12 12:16:49 +01:00
Hannah Ward a34014e245
Fixed observables within an indicator not being added 2016-08-12 11:56:48 +01:00
Raphaël Vinot 59b16950f7 Remove bin script, use cleaner way. Fix last commit. 2016-08-12 12:35:33 +02:00
Hannah Ward faddf8378e
Stiximport will now consume campaigns 2016-08-12 11:34:43 +01:00
Hannah Ward 598a030962
stiximport will now identify file hashes 2016-08-12 11:22:42 +01:00
Alexandre Dulaunoy 99749d4de2 Merge pull request #39 from Rafiot/master 
Use entry_points instead of scripts in the install.
 2016-08-12 11:33:47 +02:00
Raphaël Vinot 23aedfb6ee Use entry_points instead of scripts. 2016-08-12 11:31:23 +02:00
Hannah Ward 3f7cdad0c3
Threat actors now get imported by stix 2016-08-12 10:06:53 +01:00
Alexandre Dulaunoy e7c6c36089 Fix: module_config should be set as introspection relies on it 2016-08-12 10:55:14 +02:00
Hannah Ward c106aa662b
Added docs to stiximport 2016-08-11 16:37:29 +01:00
Hannah Ward b654a9743b
Added stix import -- works for IPs/Domains 2016-08-11 16:33:02 +01:00
iglocska 6116c017c1 Update to the DNS module to support domain|ip 2016-08-10 17:11:46 +02:00
iglocska c3a3d68e43 Small change to the skeleton export 2016-08-10 16:47:55 +02:00
Iglocska eea62db199 Added test export module 2016-08-05 21:58:24 +02:00
Alexandre Dulaunoy bf035e148c Merge branch 'import-test' of github.com:MISP/misp-modules into import-test 2016-08-04 18:55:17 +02:00
Alexandre Dulaunoy 27ddbd9b92 Fix: types array 2016-08-04 18:54:21 +02:00
Raphaël Vinot b3a322a178 Pass the server port as integer to the uwhois client 2016-08-04 17:44:40 +02:00
Raphaël Vinot f72534c785 Add whois module 2016-08-04 17:23:23 +02:00
Alexandre Dulaunoy f97c5d62d6 First version of an Optical Character Recognition (OCR) module for MISP 2016-08-04 14:32:50 +02:00
Iglocska 2b84e47f34 first version of the import skeleton 2016-08-04 09:12:10 +02:00
Iglocska 3fb62fac70 Added simple import skeleton 2016-08-04 08:00:09 +02:00
Raphaël Vinot 22eaba6ab6 Make sure misp-modules can be launched from anywhere 2016-06-23 19:51:13 +09:00