b2ed7f766b
fix: python 3.9 compat, take 2
2020-11-24 18:49:43 +01:00
46fd402d43
fix: python 3.9 compat
2020-11-24 18:47:20 +01:00
7154bfc01f
add: A few more phone numbers regexes
2020-10-31 22:06:20 +01:00
6d49d3386f
add: Added regexes for the american fictitious numbers in the list
2020-10-30 23:29:18 +01:00
8a629209f2
chg: Changed name to be displayed as warning and description
2020-10-27 11:19:05 +01:00
a50c06c9eb
chg: Turned the regexes for audiovisual works into a single one
2020-10-27 10:38:45 +01:00
543406dff4
add: New Warninglist for phone numbers that should never be attributed
...
- First examples filling the list of regexes: the
phone numbers used for audiovisual works, or
the communications companies internal numbers.
Those phone numbers are reserved and should
never be given to any user
- We'll add as well the numbers reserved for the
american audiovisual works soon
2020-10-27 04:03:44 +01:00
14fc1e9cf0
Adding replacement for wildcard and dash inputs
2020-09-30 15:43:57 -04:00
7e0e72bb5c
Changed matching algorithm to string
2020-09-23 15:58:03 -04:00
e121ff3925
Changed matching algorithm to string
2020-09-23 15:47:49 -04:00
365fb4c00e
Changed matching algorithm to string
2020-09-23 15:46:41 -04:00
7c1de70ce0
Changed matching algorithm to string
...
Example for a dangerous entry: dropbox.com with the hostname algorithm and url as a valid attribute type means that https://dropbox.com/malicious/files.exe would get excluded from the automation systems when using the warninglist.
I've changed the algorithm to full string matches.
2020-09-16 00:27:03 +02:00
a9c163ef20
Refactor last scripts, logging, central directory for downloads
...
* Refactored generate_moz-top50.py
* Download all file to new /tmp file to centralize all downloads
* Add central logging to generators.log file
* Create Bash script that generates all warninglists
* Add /tmp folder and extra files to .gitignore
* Start adding exception handling in download_to_file and write_to_file
2020-07-27 10:44:30 +02:00
610292e90f
Refactor more generators
2020-07-21 13:42:50 +02:00
623ccd6c44
remove extra .txt extension from downloaded filed
2020-07-21 09:13:15 +02:00
9dc6ea9ca8
add check if downloaded file has changed on server before downloading
2020-07-21 09:11:51 +02:00
d3e87dc7ae
Refactor code to make it simpler/more uniform
2020-07-21 01:21:28 +02:00
d32eb23a58
Chg generator-publicdns: work with new CSV format
...
1. The CSV format has changed with the update on 2020-07-14.
2. The script also generates IPv4, IPv6, and the hostname lists at once.
3. Downloaded file added to .gitignore
2020-07-21 00:38:50 +02:00
cb52a472e6
solved LGTM alerts
2020-07-17 12:42:34 +02:00
9ba47c1463
Added multiple lists from Cisco Umbrella list. Solves issue #24 and #13
2020-07-17 12:31:29 +02:00
049475ab22
add .gitignore for downloaded files, refactor code for generators: use central module, remove useless code, fix minor issues
2020-07-17 10:06:06 +02:00
c00b21de5f
Fix Microsoft Azure generator: format changed from XML to JSON + download link changed.
2020-07-17 09:25:01 +02:00
e0b3968635
Merge tranco scripts,:generate_tranco.py generates both full and 10k list
2020-07-17 09:23:39 +02:00
1d59d7f6f5
Automatically copy output to list.json file in correct folder
2020-07-16 10:02:59 +02:00
1f15bba220
Refactored mozilla certificate generator: solve relative path issue, remove unused code, refactor structure of code
2020-07-15 16:28:34 +02:00
c924d72db5
renamed cisco top1m to top1k to reflect reality
2020-07-15 02:06:02 +02:00
2bbbb52e1c
chg: chmod +x for new scripts in tools folder
2020-07-14 16:28:07 +02:00
Alexandre Dulaunoy
7a28900663
Merge branch 'main' of github.com:MISP/misp-warninglists into main
2020-07-08 14:54:14 +02:00
f493aa451d
Update TLDs list
2020-07-06 16:20:48 +02:00
e0cd7a65a0
Added tranco10k
2020-06-11 10:07:04 -05:00
4766c457c0
Validate values in CI
2020-05-22 19:28:04 +02:00
300d823638
chg: Add script to make lists unique, and sort the keys.
...
Update covid lists.
2020-04-03 13:37:17 +02:00
c14634a031
new: added covid generators / lists
2020-04-01 12:00:55 +02:00
109470bf0d
Create generate_majestic-million.py
...
Pulls top 10K of the most referred to hosts from Majestic Million.
2020-02-03 01:01:22 -06:00
17952df91e
add script to generate warning list from Tranco ( https://tranco-list.eu/ )
2020-01-10 15:27:07 +01:00
622bd3510e
add: [wikimedia] adds a warning list for wikimedia infrastructure.
2019-09-12 11:09:16 +02:00
d22c6902b3
Added list and tool to generate list for cloudflare IP ranges.
2019-07-24 17:35:39 +02:00
Alexandre Dulaunoy
4d8ce3986f
Merge pull request #106 from SteveClement/tools
...
fix: [alexa] The generator wants to decode things ;)
2019-04-24 15:38:25 +02:00
Steve Clement
14a675a70a
fix: [moz500] Fix the confusion about Moz.com and Mozilla.com
2019-04-24 14:42:41 +09:00
Steve Clement
88ad201afd
fix: [alex] The generator wants to decode things ;)
2019-04-24 10:51:11 +09:00
Steve Clement
1be60b805a
fix: [tools] Made python scripts executable.
2019-04-24 10:38:42 +09:00
Steve Clement
190312cf0f
chg: [moz500] Added Pages too. Updated list
2019-04-24 10:36:22 +09:00
Steve Clement
f41f976ce6
chg: [moz500] Added info how to regenerate, added provisional urls/files
...
to topPages.
2019-04-24 10:23:40 +09:00
Steve Clement
9e0b2ebc75
new: [list] Added Mozilla Top 500 domains
2019-04-24 09:45:56 +09:00
Steve Clement
8f1fe94b49
new: [tool] Generate The Moz top 500 Domain list from https://moz.com/top500
2019-04-23 18:00:07 +09:00
af7e5a188c
CAs: Fix final new line in json
2019-03-29 16:57:54 +01:00
d4e749c3e6
CAs: Fix json indentation (2 spaces, not 4)
2019-03-29 16:55:34 +01:00
da0dabc01b
Mozilla CA/intermediate: also match x509-fingerprint-*
2019-03-29 16:39:22 +01:00
a1c0e83bd4
Add warning lists based on Mozilla's trusted CA and Intermediates
2019-03-29 16:23:22 +01:00
04e8f468d9
fix: Wrong file name in the scripts
2019-03-25 11:43:27 +01:00
Alexandre Dulaunoy
48a5e32d50
new: [disposal-email] added
2019-03-24 12:02:30 +01:00
Alexandre Dulaunoy
5ed5403157
new: [VPN] lists of common VPN IPv4 and IPv6 addresses added
...
Source of the IPv4/IPv6 is https://github.com/ejrv/VPNs
2019-03-24 11:17:59 +01:00
Alexandre Dulaunoy
a568c14332
chg: [o365 tools] fix title of the IP address warning list
2019-03-10 09:28:31 +01:00
Alexandre Dulaunoy
6e97b8bf8a
chg: [o365] separate Microsoft Office 365 lists (hostname and IP addresses)
2019-03-10 09:27:12 +01:00
Alexandre Dulaunoy
0cc3ad9d82
Merge branch 'master' of https://github.com/kx499/misp-warninglists into kx499-master
2019-03-10 09:13:03 +01:00
91d77374ff
updated MS O365 script to handle json and updated list.json
2019-03-09 19:43:13 -05:00
Alexandre Dulaunoy
342a6d2ad5
chg: [tools] alexa script fixed
2019-03-09 17:25:27 +01:00
Alexandre Dulaunoy
58db3ef4b0
chg: [tools] fix cisco script
2019-03-09 17:23:42 +01:00
Alexandre Dulaunoy
d24febd23c
Merge pull request #93 from ater49/master
...
Adding university domains warninglist
2019-03-01 22:43:14 +01:00
120d6955c4
Correcting updater
2019-03-01 22:29:19 +01:00
a8c19f102f
Correcting updater
2019-03-01 22:28:58 +01:00
536dae5db2
Adding updater for crl warninglist
2019-03-01 19:02:59 +01:00
c7cbb9a5d7
Adding update tool for university domains list
2019-03-01 18:48:22 +01:00
5fbec29345
Fixed TLD truncation issue
...
Fixed TLD truncation issue
2018-08-20 09:13:38 -04:00
9021404d6f
Added generate_alexa.py back and added type param
2018-08-10 09:17:49 -04:00
d9457fc40a
Not sure why Travis is failing
2018-08-10 09:11:17 -04:00
c468d00355
Added type for Travis
2018-08-10 08:57:27 -04:00
7246355806
Removing gen Alexa
2018-08-10 08:48:03 -04:00
ed9700b150
New script to generate Cisco Umbrella Top 1000
2018-08-10 08:47:40 -04:00
Alexandre Dulaunoy
6b815d76f9
fix: various fixes + add number of elements in each lists
2018-03-20 22:34:19 +01:00
Alexandre Dulaunoy
9760088941
Merge branch 'master' of https://github.com/gizolka/misp-warninglists into gizolka-master
2018-03-20 21:29:34 +01:00
820aef2a74
Created a converter of MISP warning lists to asciidoctor format
2018-03-20 11:35:49 +01:00
d1d5b6f109
Bugfix (l.append)
2018-02-22 10:57:31 +01:00
3fb335eebd
Bugfix type
2018-02-22 07:53:31 +01:00
f42b95706e
Fixed typo
2018-02-21 23:30:02 +01:00
ab4bca460e
"type": "cidr",
2018-02-21 23:29:30 +01:00
4716207fec
Update generate-amazon-aws.py
2018-02-21 22:05:02 +00:00
f0c996b8de
Generate json file of Amazon AWS IP's
2018-02-21 22:50:13 +01:00
Alexandre Dulaunoy
ac2beeea02
add: Microsoft Azure Datacenter IP Ranges added including tool to
...
generate the JSON.
Fix #43
2017-12-29 15:39:52 +01:00
Alexandre Dulaunoy
23d4b68de3
add: list of Microsoft office365/azure in China + extraction tool added
...
fix #42
2017-12-29 15:30:57 +01:00
02b4bf8a52
adding wikipedia scrapper for google domains
2017-06-19 09:09:33 +02:00
cdef6f192e
Update lists, add schema.
2017-02-12 21:01:36 +01:00
f3b9178b55
typo in the name
2017-01-27 15:42:54 +01:00
7ad930e5fb
Checks for open resolvers in the list of IPs.
...
Is able to fetch the MISP warning list a say if there are some resolvers.
2016-12-07 18:30:52 +01:00
Alexandre Dulaunoy
338241e499
Add version and name to the office365 warning list
2016-12-06 12:39:45 +01:00
Alexandre Dulaunoy
42816083a3
Office 365 URLs and IP address ranges added
2016-12-06 12:16:34 +01:00
bf8b467a3e
Add a tool to generate public dns resolver list
...
The tool generate two lists, one for IPv4 (list4.json) and one for IPv6
(list6.json) to allow the user to only enable one of the two.
The list is downloaded from http://public-dns.info/ and a sample of the
list was tested with:
for dns in $( awk -F "," '{ print $1 }' < nameservers.csv ) ; do dig +noedns @$dns google.com | grep NOERROR 1>/dev/null || echo $dns ; done
~95% of the tested servers responded. So the list is not all crap.
2016-08-03 19:59:55 +02:00
16a4265688
Actualy put alexa 1000 top domains in the output list
2016-08-03 15:54:13 +02:00
4075a58165
Write the alexa top1M zip file after download
2016-08-03 15:52:28 +02:00
b856e9897c
Fix alexa top1M url
2016-08-03 15:43:35 +02:00
Alexandre Dulaunoy
34ce028295
Alexa top 1000 MISP warning list added including generation tool
2016-05-28 21:31:40 +02:00
Raphaël Vinot
Alexandre Dulaunoy
Steve Clement