D4
/
analyzer-d4-log
mirror of https://github.com/D4-project/analyzer-d4-log
2
0
Fork 0
Code Issues Releases Wiki Activity
57 Commits
3 Branches
6 Tags
461 KiB
Go 54.8%
Python 38.9%
JavaScript 4.7%
Shell 1.6%
 
 
 
 
Go to file
Jean-Louis Huynen c7372bc29a
chg: [sshd] make it compatible with analyzer-json by default 		2020-10-22 10:08:36 +02:00
MISP_export chg: [misp] MISP export 2020-06-19 11:59:48 +02:00
assets chg [doc] initial README material 2020-06-26 15:50:04 +02:00
conf.sample add: [grok] moving to grokking support - logic refacto 2020-03-06 17:02:46 +01:00
inputreader chg: [sshd] retry on redisreader EOF 2020-05-27 17:16:11 +02:00
logcompiler chg: [sshd] make it compatible with analyzer-json by default 2020-10-22 10:08:36 +02:00
.gitignore chg: [sshd] svg graph generation 2020-01-30 17:31:47 +01:00
LICENSE Initial commit 2020-01-23 17:53:04 +01:00
README.md chg: [doc] typos 2020-06-26 15:53:15 +02:00
go.mod chg: [mod] bump d4-golang-util - fix #13 2020-06-19 11:57:51 +02:00
go.sum chg: [mod] bump d4-golang-util - fix #13 2020-06-19 11:57:51 +02:00
install_server.sh chg: [install] backport 2020-02-25 16:03:32 +01:00
launch_server.sh chg: [install] install and launch scripts + bumping golang-utils 2020-02-13 14:23:46 +01:00
main.go chg: [misp] correct timings 2020-06-22 15:45:45 +02:00
redis.conf add: [conf] redis config 2020-02-13 15:28:02 +01:00

README.md

analyzer-d4-log

This analyzer processes loglines ingested by d4 (as type 3).

Architecture

analyzer-d4-log relies on redis to consume grokked loglines.

To grok the loglines, analyzer-d4-log relies on an external tool: both logstash https://www.elastic.co/logstash and nifi https://nifi.apache.org/ have been tested for this purpose (using this nifi template https://github.com/D4-project/d4-nifi-templates). These tools poll directly d4 server's redis for loglines and push the results into a specific redis queue that the analyzer consumes.

Grokking D4 loglines in nifi

analyzer-d4-log polls this queue periodically to produce counts and statistics of the data. At the moment, only sshd logs are supported but more will come in the future.

SSHD log analysis

Output generation

Every once in a while, analyzer-d4-log compiles the result into a svg images and csv files. It will also produce a minimalist webpage to navigate the data with a datarangepicker.;

MISP export

I addition to this graphical view, the repository contains a MISP_export folder that allows for the publication of a MISP feed of daily events. It compiles the TOP 100 usernames and sources seen in ssh login failure by D4 sensors.

Since MISP 2.4.128, MISP can conveniently display this data through specialized widgets.