# analyzer-d4-passivedns analyzer-d4-passivedns is an analyzer for a D4 network sensor including a complete Passive DNS server. The analyser can process data produced by D4 sensors (in [passivedns](https://github.com/gamelinux/passivedns) CSV format (more to come)) or independently from D4 using [COF websocket](https://datatracker.ietf.org/doc/html/draft-dulaunoy-dnsop-passive-dns-cof) streams. The package include a Passive DNS server which can be queried later to search for the Passive DNS records. # Features - [Input stream] - A D4 analyzer which can be plugged to one or more [D4 servers](https://github.com/D4-project/d4-core) to get a stream of DNS records - [Input Stream] - A websocket stream (or a file stream) in NDJSON [COF format](https://datatracker.ietf.org/doc/html/draft-dulaunoy-dnsop-passive-dns-cof) - [Output API] A compliant [Passive DNS ReST server compliant to Common Output Format](https://tools.ietf.org/html/draft-dulaunoy-dnsop-passive-dns-cof) - A flexible and simple analyser which can be configured to collect the required records from DNS records # Overview ## Requirements - Python 3.8 - Redis >5.0 or [kvrocks](https://github.com/apache/incubator-kvrocks) - Tornado - iptools ## Install ### Redis ~~~~ ./install_server.sh ~~~~ All the Python 3 code will be installed in a virtualenv (PDNSENV). ### Kvrocks ~~~ ./install_server_kvrocks.sh ~~~ All the Python 3 code will be installed in a virtualenv (PDNSENV). ## Running ### Start the redis server or kvrocks server Don't forget to set the DB directory in the redis.conf configuration. By default, the redis for Passive DNS is running on TCP port 6400 ~~~~ ./redis/src/redis-server ./etc/redis.conf ~~~~ or ~~~~ ./kvrocks/src/kvrocks -c ./etc/kvrocks.conf ~~~~ ### Start the Passive DNS COF server ~~~~ . ./PDNSENV/bin/activate cd ./bin/ python3 ./pdns-cof-server.py ~~~~ ## Feeding the Passive DNS server You have two ways to feed the Passive DNS server. You can combine multiple streams. A sample public COF stream is available from CIRCL with the newly seen IPv6 addresses and DNS records. ### (via COF websocket stream) start the importer ~~~~ python3 pdns-import-cof.py --websocket ws://crh.circl.lu:8888 ~~~~ ### (via D4) Configure and start the D4 analyzer ~~~~ cd ./etc cp analyzer.conf.sample analyzer.conf ~~~~ Edit the analyzer.conf to match the UUID of the analyzer queue from your D4 server. ~~~~ [global] my-uuid = 6072e072-bfaa-4395-9bb1-cdb3b470d715 d4-server = 127.0.0.1:6380 # INFO|DEBUG logging-level = INFO ~~~~ then you can start the analyzer which will fetch the data from the analyzer, parse it and populate the Passive DNS database. ~~~~ . ./PDNSENV/bin/activate/ cd ./bin/ python3 pdns-ingestion.py ~~~~ ## Usage ### Querying the server ~~~~shell adulau@kolmogorov ~/git/analyzer-d4-passivedns (master)$ curl -s http://127.0.0.1:8400/query/xn--ihuvudetpevap-xfb.se | jq . { "time_first": 1657878272, "time_last": 1657878272, "count": 1, "rrtype": "AAAA", "rrname": "xn--ihuvudetpevap-xfb.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured" } ~~~~ ~~~~shell curl -s http://127.0.0.1:8400/query/2a02:250:0:8::53 {"time_first": 1657878141, "time_last": 1657878141, "count": 1, "rrtype": "AAAA", "rrname": "media.vastporten.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657878929, "time_last": 1657878929, "count": 1, "rrtype": "AAAA", "rrname": "www.folkinitiativetarjeplog.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657878272, "time_last": 1657878272, "count": 1, "rrtype": "AAAA", "rrname": "xn--ihuvudetpevap-xfb.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657878189, "time_last": 1657878189, "count": 1, "rrtype": "AAAA", "rrname": "media.primesteps.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657878986, "time_last": 1657878986, "count": 1, "rrtype": "AAAA", "rrname": "media.skellefteaadventurepark.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657874940, "time_last": 1657874940, "count": 1, "rrtype": "AAAA", "rrname": "galleri.torsaspaintball.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657874205, "time_last": 1657874205, "count": 1, "rrtype": "AAAA", "rrname": "www.media1.harlaut.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657878165, "time_last": 1657878165, "count": 1, "rrtype": "AAAA", "rrname": "www.sd-nekretnine.rs", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657878678, "time_last": 1657878678, "count": 1, "rrtype": "AAAA", "rrname": "www.www2.resultat-balans.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657874288, "time_last": 1657874288, "count": 1, "rrtype": "AAAA", "rrname": "www.assistanshemtjanst.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657878943, "time_last": 1657878943, "count": 1, "rrtype": "AAAA", "rrname": "kafekultur.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657878141, "time_last": 1657878141, "count": 1, "rrtype": "AAAA", "rrname": "media1.rlab.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657878997, "time_last": 1657878997, "count": 1, "rrtype": "AAAA", "rrname": "serbiagreenbuildingexpo.com", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657879064, "time_last": 1657879064, "count": 1, "rrtype": "AAAA", "rrname": "www.framtro.nu", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657874285, "time_last": 1657874285, "count": 1, "rrtype": "AAAA", "rrname": "www.twotheartist.com", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} {"time_first": 1657878774, "time_last": 1657878774, "count": 1, "rrtype": "AAAA", "rrname": "media.narkesten.se", "rdata": "2a02:250:0:8::53", "origin": "origin not configured"} ~~~~ # License The software is free software/open source released under the GNU Affero General Public License version 3.