architecture/docs/preso/00-ISday19/isday.tex

390 lines
14 KiB
TeX
Raw Normal View History

2019-05-10 15:55:59 +02:00
% Full instructions available at:
% https://github.com/elauksap/focus-beamertheme
\documentclass{beamer}
\usetheme[numbering=progressbar]{focus}
\usepackage{tikz}
\usetikzlibrary{positioning}
\usetikzlibrary{shapes,arrows}
\usepackage{transparent}
\usepackage{fancyvrb}
\usepackage{listings}
\usepackage{tabularx}
2019-05-16 16:01:17 +02:00
\usepackage{amsfonts}
\usepackage{csquotes}
2019-05-10 15:55:59 +02:00
\definecolor{main}{RGB}{47, 161, 219}
\definecolor{background}{RGB}{240, 247, 255}
\definecolor{textcolor}{RGB}{85, 87, 83}
\title{D4 Project}
\subtitle{Open and collaborative network monitoring}
\author{Jean-Louis Huynen}
\titlegraphic{\includegraphics[scale=0.20]{../../logos/d4-logo.pdf}}
\institute{Team CIRCL \\ \url{https://www.d4-project.org/}}
\date{2019/05/21}
\begin{document}
\begin{frame}
\maketitle
\end{frame}
\begin{frame}
\frametitle{Problem statement}
\begin{itemize}
\item CSIRTs (or private organisations) build their {\bf own honeypot, honeynet or blackhole monitoring network}
\item Designing, managing and operating such infrastructure is a tedious and resource intensive task
\item {\bf Automatic sharing} between monitoring networks from different organisations is missing
\item Sensors and processing are often seen as blackbox or difficult to audit
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Objective}
\begin{itemize}
\item Based on our experience with
MISP\footnote{\url{https://github.com/MISP/MISP}} where sharing
played an important role, we transpose the model in D4 project
\item Keeping the protocol and code base {\bf simple and minimal}
\item Allowing every organisation to {\bf control and audit their own sensor network}
\item Extending D4 or {\bf encapsulating legacy monitoring protocols} must be as simple as possible
\item Ensuring that the sensor server has {\bf no control on the sensor} (unidirectional streaming)
\item Don't force users to use dedicated sensors and allow {\bf flexibility of sensor support} (software, hardware, virtual)
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{(short) History}
\begin{itemize}
\item D4 Project (co-funded under INEA CEF EU program) started - 1st November 2018
\item D4 encapsulation protocol version 1 published - 1st December 2018
\item v0.1 release of the D4 core\footnote{\url{https://www.github.com/D4-project/d4-core}} including a server and simple D4 C client - 21st January 2019
2019-05-16 16:01:17 +02:00
\item First version of a golang D4
client\footnote{\url{https://www.github.com/D4-project/d4-goclient/}}
running on ARM, MIPS, PPC and x86 - 14th February 2019
2019-05-10 15:55:59 +02:00
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{(short) History}
\begin{center}
\begin{tabularx}{\linewidth}%
{>{\setlength\hsize{0.6\hsize}\raggedright}X%
>{\setlength\hsize{0.4\hsize}\raggedright}X}
\hline
Release & Date \tabularnewline
\hline
analyzer-d4-passivedns-v0.1 & Apr. 5, 2019 \tabularnewline
analyzer-d4-passivessl-0.1 & Apr. 25, 2019 \tabularnewline
analyzer-d4-pibs-v0.1 & Apr. 8, 2019 \tabularnewline
BGP-Ranking-1.0 & Apr. 25, 2019 \tabularnewline
d4-core-v0.1 & Jan. 25, 2019 \tabularnewline
d4-core-v0.2 & Feb. 14, 2019 \tabularnewline
d4-core-v0.3 & Apr. 8, 2019 \tabularnewline
d4-goclient-v0.1 & Feb. 14, 2019 \tabularnewline
d4-goclient-v0.2 & Apr. 8, 2019 \tabularnewline
d4-server-packer-0.1 & Apr. 25, 2019 \tabularnewline
IPASN-History-1.0 & Apr. 25, 2019 \tabularnewline
sensor-d4-tls-fingerprinting-0.1 & Apr. 25, 2019 \tabularnewline
\hline
\end{tabularx}
\end{center}
2019-05-16 16:01:17 +02:00
see \url{https://github.com/D4-Project}
2019-05-10 15:55:59 +02:00
\end{frame}
\begin{frame}
\frametitle{D4 Overview}
\includegraphics[scale=0.38]{../../diagram/d4-overview.png}
\end{frame}
\begin{frame}
2019-05-16 16:01:17 +02:00
\frametitle{Roadmap - output}
2019-05-10 15:55:59 +02:00
2019-05-16 16:01:17 +02:00
CIRCL will host a server instance for organisations willing to
contribute to a public dataset without running their own D4 server:
2019-05-10 15:55:59 +02:00
\begin{itemize}
2019-05-16 16:01:17 +02:00
\item [\checkmark]Passive SSL
\item [\checkmark] Passive DNS
\item [\checkmark]Blackhole DDoS
\item BGP mapping
\item egress filtering mapping
\item Radio monitoring
\item ...
2019-05-10 15:55:59 +02:00
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{D4 encapsulation protocol}
\includegraphics[scale=0.38]{../../diagram/d4-protocol-encapsulation.png}
\end{frame}
\begin{frame}
\frametitle{D4 Header}
\begin{tabular}{|l|l|l|}
\hline
Name & bit size& Description\\
\hline
version & uint 8 & Version of the header \\
type & uint 8 & Data encapsulated type\\
uuid & uint 128 & Sensor UUID\\
timestamp & uint 64 & Encapsulation time\\
hmac & uint 256 & Authentication header (HMAC-SHA-256-128)\\
size & uint 32 & Payload size\\
\hline
\end{tabular}
\end{frame}
\begin{frame}
\frametitle{D4 Header}
\framesubtitle{Types}
\begin{tabular}{|l|l|}
\hline
Type & Description\\
\hline
0 & Reserved\\
1 & pcap (libpcap 2.4)\\
2 & meta header (JSON)\\
3 & generic log line\\
4 & dnscap output\\
5 & pcapng (diagnostic)\\
6 & generic NDJSON or JSON Lines\\
7 & generic YAF (Yet Another Flowmeter)\\
8 & passivedns CSV stream\\
254 & type defined by meta header (type 2)\\
\hline
\end{tabular}
\end{frame}
\begin{frame}
\frametitle{D4 meta header}
\framesubtitle{Meta types}
D4 header includes an easy way to {\bf extend the protocol} (via type 2) without altering the format. Within a D4 session, the initial D4 packet(s) type 2 defines
the custom headers and then the following packets with type 254 is the custom data encapsulated.
\small
\input{meta.tex}
\end{frame}
\begin{frame}
\frametitle{D4 server}
\begin{itemize}
\item D4 core server\footnote{\url{https://github.com/D4-project/d4-core}} is a complete server to handle clients (sensors) including the decapsulation of the D4 protocol, control of sensor registrations, management of decoding protocols and dispatching to adequate decoders/analysers.
\item D4 server is written in Python 3.6 and runs on standard GNU/Linux distribution.
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{D4 server - management interface}
The D4 server provides a web interface to manage D4 sensors, sessions and analyzer.
\begin{itemize}
\item Get Sensors status, errors and statistics
\item Get all connected sensors
\item Manage Sensors (stream size limit, secret key, ...)
\item Manage Accepted types
\item UUID/IP blocklist
\item Create Analyzer Queues
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{D4 server - main interface}
\includegraphics[width=\textwidth]{../../diagram/d4-5.png}
\end{frame}
\begin{frame}
\frametitle{D4 server - server management}
\includegraphics[width=\textwidth]{../../diagram/d4-2.png}
\end{frame}
\begin{frame}
\frametitle{D4 server - server management}
\includegraphics[width=\textwidth]{../../diagram/d4-3.png}
\end{frame}
\begin{frame}
\frametitle{D4 server - sensor overview}
\includegraphics[width=\textwidth]{../../diagram/d4-1.png}
\end{frame}
\begin{frame}
\frametitle{D4 server - sensor management}
\includegraphics[width=\textwidth]{../../diagram/d4-4.png}
\end{frame}
2019-05-16 16:01:17 +02:00
\begin{frame}
\frametitle{}
\begin{center}
A distributed Network telescope to observe DDoS attacks
\end{center}
\vspace{10pt}
\begin{center}
\includegraphics[width=.7\textwidth]{eventhorizon.png}
\end{center}
\end{frame}
2019-05-10 15:55:59 +02:00
2019-05-16 16:01:17 +02:00
\begin{frame}
\frametitle{Motivation}
DDoS Attacks produce an observable side-effect:
\begin{center}
\scalebox{0.8}{\input{bsvol.tex}}
\end{center}
\end{frame}
2019-05-10 15:55:59 +02:00
\begin{frame}
2019-05-16 16:01:17 +02:00
\frametitle{What can be derived from backscatter traffic?}
\begin{itemize}
\item External point of view on ongoing denial of service attacks
\item Confirm if there is a DDOS attack
\item Recover time line of attacked targets
\item Confirm which services (DNS, webserver, $\dots$)
\item Infrastructure changes
\item Assess the state of an infrastructure under denial of service attack
\begin{itemize}
\item Detect failure/addition of intermediate network equipments, firewalls, proxy servers etc
\item Detect DDoS mitigation devices
\end{itemize}
\item Create probabilistic models of denial of service attacks
\end{itemize}
2019-05-10 15:55:59 +02:00
\end{frame}
2019-05-16 16:01:17 +02:00
\begin{frame}
\frametitle{D4 in this setting}
Aggregating backscatter traffic collected from D4 sensors:
\begin{itemize}
\item have various points of observation (non contiguous address space)
\item perform analysis on bigger amount of data
\end{itemize}
D4 lookup should provide:
\begin{itemize}
\item backscatter analysis results,
\item daily updates,
\item additional relevant information (DNS, BGP, etc.).
\end{itemize}
\end{frame}
2019-05-10 15:55:59 +02:00
2019-05-16 16:01:17 +02:00
\begin{frame}
\begin{center}
Passive DNS
\end{center}
\end{frame}
2019-05-10 15:55:59 +02:00
\begin{frame}
2019-05-16 16:01:17 +02:00
\frametitle{Problem statement}
\begin{itemize}
\item CIRCL (and other CSIRTs) have their own passive DNS\footnote{\url{https://www.circl.lu/services/passive-dns/}} collection mechanisms
\item Current {\bf collection models} are affected with DoH\footnote{DNS over HTTPS} and centralised DNS services
\item DNS answers collection is a tedious process
\item {\bf Sharing Passive DNS stream} between organisation is challenging due to privacy
\end{itemize}
2019-05-10 15:55:59 +02:00
\end{frame}
2019-05-16 16:01:17 +02:00
\begin{frame}
\frametitle{Potential Strategy}
\begin{itemize}
\item Improve {\bf Passive DNS collection diversity} by being closer to the source and limit impact of DoH (e.g. at the OS resolver level)
\item Increasing diversity and {\bf mixing models} before sharing/storing Passive DNS records
\item Simplify process and tools to install for {\bf Passive DNS collection by relying on D4 sensors} instead of custom mechanisms
\item Provide a distributed infrastructure for mixing streams and filtering out the sharing to the validated partners
\end{itemize}
\end{frame}
2019-05-10 15:55:59 +02:00
\begin{frame}
2019-05-16 16:01:17 +02:00
\frametitle{First release}
\begin{itemize}
\item analyzer-d4-passivedns\footnote{\url{https://github.com/D4-project/analyzer-d4-passivedns}} is an analyzer for a D4 network sensor. The analyser can process data produced by D4 sensors (in passivedns CSV format\footnote{\url{https://github.com/gamelinux/passivedns}})
\item Ingest these into a {\bf Passive DNS server} which can be queried later to search for the Passive DNS records
\item The lookup server (using on redis-compatible backend) is a Passive DNS REST server compliant to the Common Output Format\footnote{\url{https://tools.ietf.org/html/draft-dulaunoy-dnsop-passive-dns-cof-04}}
\end{itemize}
\end{frame}
\begin{frame}
\begin{center}
Passive SSL revamping
\end{center}
\end{frame}
\begin{frame}
\frametitle{A passive SSL fingerprinter}
CSIRT's rationale for collecting TLS handshakes:
\begin{itemize}
\item pivot on additional data points,
\item find owners of IP addresses,
\item detect usage of CIDR blocks,
\item detect vulnerable systems,
\item detect compromised services,
\item detect Key material reuse,
\item detect weak keys.
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Objectives}
History of links between:
\begin{itemize}
\item x509 certificates,
\item ports,
\item IP address,
\item client (ja3),
\item server (ja3s),
\end{itemize}
\begin{displayquote}
``JA3 is a method for creating SSL/TLS client fingerprints that should be easy to produce on any platform and can be easily shared for threat intelligence.''\footnote{https://github.com/salesforce/ja3}
\end{displayquote}
\end{frame}
\begin{frame}
\frametitle{Objectives}
Mind your Ps and Qs:
\begin{itemize}
\item Public keys type and size,
\item modulos and exponents,
\item curves parameters.
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{First release}
\begin{itemize}
\item[\checkmark] sensor-d4-tls-fingerprinting
\footnote{\url{github.com/D4-project/sensor-d4-tls-fingerprinting}}:
Extracts and fingerprints certificates
\item[\checkmark] analyzer-d4-passivessl
\footnote{\url{github.com/D4-project/analyzer-d4-passivessl}}:
Stores Certificates / PK details in a PostgreSQL DB
\item lookup-d4-passivessl
\footnote{\url{github.com/D4-project/lookup-d4-passivessl}}:
Exposes the DB through a public REST API
\end{itemize}
2019-05-10 15:55:59 +02:00
\end{frame}
2019-05-16 16:01:17 +02:00
\begin{frame}[t]{Future}
\begin{itemize}
\item {\bf Mixing models for passive collection streams} (for privacy) in next version of D4 core server
\item Interconnecting private D4 sensor networks with other D4 sensor networks (sharing to partners filtered stream)
\item Previewing dataset collected in D4 sensor network and providing {\bf open data stream} (if contributor agrees to share under specific conditions)
\end{itemize}
\end{frame}
2019-05-10 15:55:59 +02:00
\begin{frame}
\frametitle{Get in touch if you want to join the project, host a sensor or contribute}
\begin{itemize}
\item Collaboration can include research partnership, sharing of collected streams or improving the software.
\item Contact: info@circl.lu
\item \url{https://github.com/D4-Project} - \url{https://twitter.com/d4_project}
\end{itemize}
\end{frame}
\end{document}