You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Jean-Louis Huynen 4bc6a0b635
chg: [doc] typos
1 month ago
MISP_export chg: [misp] MISP export 1 month ago
assets chg [doc] initial README material 1 month ago
conf.sample add: [grok] moving to grokking support - logic refacto 5 months ago
inputreader chg: [sshd] retry on redisreader EOF 2 months ago
logcompiler chg [sshd] omitempty json fields 1 month ago
.gitignore chg: [sshd] svg graph generation 6 months ago
LICENSE Initial commit 6 months ago
README.md chg: [doc] typos 1 month ago
go.mod chg: [mod] bump d4-golang-util - fix #13 1 month ago
go.sum chg: [mod] bump d4-golang-util - fix #13 1 month ago
install_server.sh chg: [install] backport 5 months ago
launch_server.sh chg: [install] install and launch scripts + bumping golang-utils 6 months ago
main.go chg: [misp] correct timings 1 month ago
redis.conf add: [conf] redis config 6 months ago

README.md

analyzer-d4-log

This analyzer processes loglines ingested by d4 (as type 3).

Architecture

analyzer-d4-log relies on redis to consume grokked loglines.

To grok the loglines, analyzer-d4-log relies on an external tool: both logstash https://www.elastic.co/logstash and nifi https://nifi.apache.org/ have been tested for this purpose (using this nifi template https://github.com/D4-project/d4-nifi-templates). These tools poll directly d4 server's redis for loglines and push the results into a specific redis queue that the analyzer consumes.

Grokking D4 loglines in nifi

analyzer-d4-log polls this queue periodically to produce counts and statistics of the data. At the moment, only sshd logs are supported but more will come in the future.

SSHD log analysis

Output generation

Every once in a while, analyzer-d4-log compiles the result into a svg images and csv files. It will also produce a minimalist webpage to navigate the data with a datarangepicker.;

MISP export

I addition to this graphical view, the repository contains a MISP_export folder that allows for the publication of a MISP feed of daily events. It compiles the TOP 100 usernames and sources seen in ssh login failure by D4 sensors.

Since MISP 2.4.128, MISP can conveniently display this data through specialized widgets.