2019-02-04 20:49:09 +01:00
% Full instructions available at:
% https://github.com/elauksap/focus-beamertheme
\documentclass { beamer}
\usetheme [numbering=progressbar] { focus}
2019-02-04 22:51:05 +01:00
\usepackage { tikz}
\usetikzlibrary { positioning}
\usetikzlibrary { shapes,arrows}
\usepackage { transparent}
\usepackage { fancyvrb}
\usepackage { listings}
2019-02-04 20:49:09 +01:00
\definecolor { main} { RGB} { 47, 161, 219}
%\definecolor{textcolor}{RGB}{128, 128, 128}
\definecolor { background} { RGB} { 240, 247, 255}
\definecolor { textcolor} { RGB} { 85, 87, 83}
\title { D4 Project}
\subtitle { Open and collaborative network monitoring}
\author { Alexandre Dulaunoy - Sami Mokaddem}
\titlegraphic { \includegraphics [scale=0.20] { d4-logo.pdf} }
\institute { Team CIRCL \\ \url { https://www.d4-project.org/} }
2019-03-28 17:43:34 +01:00
\date { 2019/03/29}
2019-02-04 20:49:09 +01:00
\begin { document}
\begin { frame}
\maketitle
\end { frame}
\begin { frame}
\frametitle { Problem statement}
\begin { itemize}
2019-02-04 22:51:05 +01:00
\item CSIRTs (or private organisations) build their { \bf own honeypot, honeynet or blackhole monitoring network}
\item Designing, managing and operating such infrastructure is a tedious and resource intensive task
\item { \bf Automatic sharing} between monitoring networks from different organisations is missing
\item Sensors and processing are often seen as blackbox or difficult to audit
2019-02-04 20:49:09 +01:00
\end { itemize}
\end { frame}
\begin { frame}
\frametitle { Objective}
\begin { itemize}
\item Based on our experience with MISP\footnote { \url { https://github.com/MISP/MISP} } where sharing played an important role, we transpose
2019-02-04 22:51:05 +01:00
the model in D4 project
\item Keeping the protocol and code base { \bf simple and minimal}
\item Allowing every organisation to { \bf control and audit their own sensor network}
\item Extending D4 or { \bf encapsulating legacy monitoring protocols} must be as simple as possible
\item Ensuring that the sensor server has { \bf no control on the sensor} (unidirectional streaming)
\item Don't force users to use dedicated sensors and allow { \bf flexibility of sensor support} (software, hardware, virtual)
\end { itemize}
\end { frame}
\begin { frame}
\frametitle { (short) History}
\begin { itemize}
\item D4 Project (co-funded under INEA CEF EU program) started - 1st November 2018
\item D4 encapsulation protocol version 1 published - 1st December 2018
\item v0.1 release of the D4 core\footnote { \url { https://www.github.com/D4-project/d4-core} } including a server and simple D4 C client - 21st January 2018
\item First version of a golang D4 client\footnote { \url { https://www.github.com/D4-project/d4-goclient/} } running on ARM, MIPS, PPC and x86 - January 2018
2019-02-04 20:49:09 +01:00
\end { itemize}
\end { frame}
2019-02-04 22:51:05 +01:00
\begin { frame}
\frametitle { D4 Overview}
2019-03-29 06:45:58 +01:00
\includegraphics [scale=0.38] { ../../diagram/d4-overview.png}
2019-02-04 22:51:05 +01:00
\end { frame}
\begin { frame}
\frametitle { Roadmap (next 2 months)}
\begin { itemize}
\item Passive DNS analyzer (alpha version released)
\item Passive SSL collector and analyzer
\item Backscatter DDoS traffic analyzer
\item { \bf Default server} (blackhole monitoring or Passive DNS collector) at CIRCL for organisations willing to contribute without running their own D4 server
\end { itemize}
\end { frame}
\begin { frame}
\frametitle { D4 encapsulation protocol}
\includegraphics [scale=0.38] { d4-protocol-encapsulation.png}
\end { frame}
\begin { frame}
\frametitle { D4 Header}
\begin { tabular} { |l|l|l|}
\hline
Name & bit size& Description\\
\hline
version & uint 8 & Version of the header \\
type & uint 8 & Data encapsulated type\\
uuid & uint 128 & Sensor UUID\\
timestamp & uint 64 & Encapsulation time\\
hmac & uint 256 & Authentication header (HMAC-SHA-256-128)\\
size & uint 32 & Payload size\\
\hline
\end { tabular}
\end { frame}
\begin { frame}
\frametitle { D4 Header}
\framesubtitle { Types}
\begin { tabular} { |l|l|}
\hline
Type & Description\\
\hline
0 & Reserved\\
1 & pcap (libpcap 2.4)\\
2 & meta header (JSON)\\
3 & generic log line\\
4 & dnscap output\\
5 & pcapng (diagnostic)\\
6 & generic NDJSON or JSON Lines\\
7 & generic YAF (Yet Another Flowmeter)\\
8 & passivedns CSV stream\\
254 & type defined by meta header (type 2)\\
\hline
\end { tabular}
\end { frame}
\begin { frame}
\frametitle { D4 meta header}
\framesubtitle { Meta types}
2019-02-04 23:10:04 +01:00
D4 header includes an easy way to { \bf extend the protocol} (via type 2) without altering the format. Within a D4 session, the initial D4 packet(s) type 2 defines
the custom headers and then the following packets with type 254 is the custom data encapsulated.
2019-02-04 22:51:05 +01:00
\small
\input { meta.tex}
\end { frame}
2019-02-05 22:19:35 +01:00
\begin { frame}
\frametitle { D4-core server}
\begin { itemize}
\item D4 core server\footnote { \url { https://github.com/D4-project/d4-core} } is a complete server to handle clients (sensors) including the decapsulation of the D4 protocol, control of sensor registrations, management of decoding protocols and dispatching to adequate decoders/analysers.
\item D4 server is written in Python 3.6 and runs on standard GNU/Linux distribution.
\end { itemize}
\end { frame}
2019-02-06 23:42:48 +01:00
\begin { frame}
\frametitle { D4 server handling}
D4 server reconstructs the encapsulated stream from the D4 sensor and saves it in a Redis stream.
\begin { itemize}
\item Support TLS connection
\item Unpack D4 header
\item Verify client secret key (HMAC)
\item check blocklist
\item Filter by types (Only accept one connection by type-UUID - except: type 254)
\item Discard incorrect data
\item Save data in a Redis Stream (unique for each session)
\end { itemize}
\end { frame}
\begin { frame}
\frametitle { D4 server - worker handler}
After the stream is processed depending of the type using dedicated worker.
\begin { itemize}
\item Worker Manager (one by type)
\begin { itemize}
\item Check if a new session is created and valid data are saved in a Redis stream
\item Launch a new Worker for each session
\end { itemize}
\item Worker
\begin { itemize}
\item Get data from a stream
\item Reconstruct data
\item Save data on disk (with file rotation)
\item Save data in Redis. Create a queue for D4 Analyzer(s)
\end { itemize}
\end { itemize}
\end { frame}
2019-03-28 17:43:34 +01:00
\begin { frame}
\frametitle { D4 server - type 254 worker handler}
\begin { itemize}
2019-03-29 09:37:52 +01:00
\item Worker custom type (called Worker 2)
2019-03-28 17:43:34 +01:00
\begin { itemize}
\item Get type 2 data from a stream
\item Reconstruct Json
\item Extract extended type name
\item Use default type or special extended handler
\item Save Json on disk
\item Get type 254 data from a stream
\item Reconstruct type 254
\item Save data in Redis. Create a queue for D4 Analyzer(s)
\end { itemize}
\end { itemize}
\end { frame}
2019-03-29 09:31:24 +01:00
\begin { frame}
\frametitle { D4 server - type 254 - implementation}
\includegraphics [scale=0.3] { d4-worker-2.png}
\end { frame}
2019-02-06 23:42:48 +01:00
\begin { frame}
\frametitle { D4 server - management interface}
The D4 server provides a web interface to manage D4 sensors, sessions and analyzer.
\begin { itemize}
\item Get Sensors status, errors and statistics
\item Get all connected sensors
\item Manage Sensors (stream size limit, secret key, ...)
\item Manage Accepted types
\item UUID/IP blocklist
\item Create Analyzer Queues
\end { itemize}
\end { frame}
\begin { frame}
\frametitle { D4 server - main interface}
2019-03-28 17:43:34 +01:00
\includegraphics [scale=0.18] { d4-5.png}
\end { frame}
\begin { frame}
\frametitle { D4 server - server management}
\includegraphics [scale=0.18] { d4-2.png}
2019-02-06 23:42:48 +01:00
\end { frame}
\begin { frame}
\frametitle { D4 server - server management}
\includegraphics [scale=0.18] { d4-3.png}
\end { frame}
\begin { frame}
\frametitle { D4 server - sensor overview}
\includegraphics [scale=0.18] { d4-1.png}
\end { frame}
\begin { frame}
\frametitle { D4 server - sensor management}
2019-03-28 17:43:34 +01:00
\includegraphics [scale=0.18] { d4-4.png}
2019-02-06 23:42:48 +01:00
\end { frame}
2019-02-05 22:19:35 +01:00
2019-02-04 22:51:05 +01:00
\begin { frame}
\frametitle { }
2019-02-06 23:42:48 +01:00
{ \center Use-case: migrating a legacy network capture model into a D4 network sensor
2019-02-04 22:51:05 +01:00
}
\end { frame}
2019-02-04 23:10:04 +01:00
2019-02-04 22:51:05 +01:00
\begin { frame}
\frametitle { Remote network capture}
CIRCL operated honeybot for multiple years using a simple model of remote network capture.
\begin { definition} [Principle]
\begin { itemize}
\item KISS (Keep it simple stupid) - Unix-like
\item Linux \& OpenBSD operating systems
\end { itemize}
\end { definition}
\begin { block} { Sensor}
\lstset { %
language=bash,
backgroundcolor=\color { gray!25} ,
basicstyle=\ttfamily ,
breaklines=true,
columns=fullflexible
}
\input { tcpdump.tex}
\end { block}
\end { frame}
\begin { frame}
\frametitle { Remote network capture}
\begin { block} { Limitations}
\begin { itemize}
\item Scalability $ \to $ one port per client
\item Identification and registration of the client
\item Integrity of the data
\end { itemize}
\end { block}
2019-02-13 11:32:52 +01:00
\begin { block} { Encapsulating streams in D4}
2019-02-04 22:51:05 +01:00
\begin { itemize}
\item Inspired by the unix command { \tt tee}
\item Read from standard input
\item Add the d4 header
\item Write it on standard output
\end { itemize}
\end { block}
\end { frame}
\begin { frame}
\frametitle { Remote network capture with D4}
\frametitle { Using D4 native client}
\lstset { %
language=bash,
backgroundcolor=\color { gray!25} ,
basicstyle=\ttfamily ,
breaklines=true,
columns=fullflexible
}
\input { d4-client.tex}
\begin { block} { Configuration directory}
\begin { tabular} { l|l}
Parameter & Explanation\\
\hline
type & see D4 Header slide\\
source & standard input\\
key & HMAC key\\
uuid & Identifier of the sensor\\
version & version of the sensor\\
destination & standard output\\
snaplen & length of data being read \& written\\
\end { tabular}
\end { block}
\end { frame}
2019-02-04 23:10:04 +01:00
\begin { frame}
\frametitle { }
{ \center Use-case: D4 analyzer to detect DDoS attacks in backscatter traffic
}
\end { frame}
\begin { frame}
\frametitle { Observing SYN floods attacks in backscatter traffic}
Attack description
\begin { tikzpicture} { scale=0.4}
\node [rectangle,draw,fill=red!80] (a) at (0,0) { Attacker} ;
\node [anchor=west] at (0.93,0.25) { Spoofed requests $ H _ { 0 } ,H _ { 1 } ,H _ { 2 } ,H _ { 3 } ,... $ } ;
\node [rectangle,draw,fill=blue!25,anchor=east] at (8,0) (v) { Victim} ;
\draw [->](a) --(v);
\foreach \x in { 0,1,2,3} {
\node [rectangle,draw,fill=green!25,anchor=east] at (\x * 2+1,-2) { $ H _ { \x } $ } ;
%Horizontal lines
\draw (\x * 2+1, -\x * 0.25-0.5)--(7.0+\x * .25,-\x * 0.25-0.5);
%Links to the victim
\draw (7.0+\x * .25,-\x * 0.25-0.5) -- (7.0+\x * .25,-0.25);
%Links to hosts
\draw [->] (\x * 2+1, -\x * 0.25-0.5)--(\x * 2+1,-1.70);
}
\end { tikzpicture}
\begin { center}
\begin { tabular} { |l|}
\hline
Connections\\
\hline
$ H _ { 0 } $ \\
\hline
$ H _ { 1 } $ \\
\hline
$ H _ { 2 } $ \\
\hline
$ H _ { 3 } $ \\
\hline
\end { tabular}
\end { center}
\end { frame}
\begin { frame}
\frametitle { What can be derived from backscatter traffic?}
\begin { itemize}
\item External point of view on ongoing denial of service attacks
\item Confirm if there is a DDoS attack
\item Recover time line of attacked targets
\item Confirm which services are a target (DNS, webserver, $ \dots $ )
\item Infrastructure changes or updates
\item Assess the state of an infrastructure under denial of service attack
\begin { itemize}
\item Detect failure/addition of intermediate network equipments, firewalls, proxy servers etc
\item Detect DDoS mitigation devices or services
\end { itemize}
\item Create probabilistic models of denial of service attacks
\end { itemize}
\end { frame}
\begin { frame}
2019-02-05 22:19:35 +01:00
\frametitle { Confirm if there is/was a DDoS attack}
2019-02-04 23:10:04 +01:00
\begin { block} { Problem}
\begin { itemize}
\item Distinguish between compromised infrastructure and backscatter
\item Look at TCP flags $ \to $ filter out single SYN flags
\item Focus on ACK, SYN/ACK, ...
\item Do not limit to SYN/ACK or ACK $ \to $ ECE (ECN Echo)\footnote { \url { https://tools.ietf.org/html/rfc3168} }
\end { itemize}
\end { block}
\input { flags.tex}
\end { frame}
\begin { frame}
\frametitle { Passive Identification of Backscatter (WiP)}
\lstset { %
language=bash,
backgroundcolor=\color { gray!25} ,
basicstyle=\ttfamily ,
breaklines=true,
columns=fullflexible
}
2019-02-05 22:19:35 +01:00
\input { pibs.tex}
Early version is available of PIBS\footnote { \url { https://github.com/D4-project/analyzer-d4-pibs} }
with a focus on TCP traffic.
2019-02-04 23:10:04 +01:00
\begin { tabular} { l|l}
Options & Explanations\\
\hline
-r & read pcap file\\
-b & display IPs under DDoS on standard output\\
\end { tabular}
\begin { tabular} { l}
Dependencies\\
\hline
libwiretap-dev\\
libhiredis-dev\\
libwsutil-dev\\
\end { tabular}
\end { frame}
2019-02-07 00:02:06 +01:00
\begin { frame}
\frametitle { Get in touch if you want to join the project, host a sensor or contribute}
\begin { itemize}
\item Collaboration can include research partnership, sharing of collected streams or improving the software.
\item Contact: info@circl.lu
\item \url { https://github.com/D4-Project} - \url { https://twitter.com/d4_ project}
\end { itemize}
\end { frame}
2019-02-04 20:49:09 +01:00
\end { document}