D4
/
architecture
mirror of https://github.com/D4-project/architecture
2
0
Fork 0
Code Issues Releases Wiki Activity

chg: [workshop-0] update content, more about pdns

master
Jean-Louis Huynen 2019-09-23 13:44:55 +02:00
parent cf27e4eb25
commit dddd3abdef
3 changed files with 310 additions and 80 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
docs/workshop/0-introduction/d4-introduction.pdf
View File

Binary file not shown.

382
docs/workshop/0-introduction/d4-introduction.tex
View File

@ -10,6 +10,7 @@
\usepackage{fancyvrb} \usepackage{fancyvrb}
\usepackage{tabularx} \usepackage{tabularx}
\usepackage{ulem} \usepackage{ulem}
\usepackage{csquotes}
\usepackage{listings} \usepackage{listings}
\definecolor{main}{RGB}{47, 161, 219} \definecolor{main}{RGB}{47, 161, 219}
%\definecolor{textcolor}{RGB}{128, 128, 128} %\definecolor{textcolor}{RGB}{128, 128, 128}
@ -303,7 +304,7 @@ The D4 server provides a {\bf web interface} to manage D4 sensors, sessions and
\begin{frame} \begin{frame}
\frametitle{} \frametitle{}
{\center Use-case: migrating a legacy network capture model into a D4 network sensor {\center Example use-case: migrating a legacy network capture model into a D4 network sensor
} }
\end{frame} \end{frame}
@ -378,45 +379,25 @@ The D4 server provides a {\bf web interface} to manage D4 sensors, sessions and
\end{block} \end{block}
\end{frame} \end{frame}
\begin{frame} \begin{frame}
\frametitle{} \frametitle{}
{\center Use-case: D4 analyzer to detect DDoS attacks in backscatter traffic \begin{center}
} {\bf A distributed Network telescope to observe DDoS attacks}
\end{center}
\vspace{10pt}
\begin{center}
\includegraphics[width=.7\textwidth]{../../preso/03-PassTheSalt/eventhorizon.png}
\end{center}
\end{frame} \end{frame}
\begin{frame} \begin{frame}
\frametitle{Observing SYN floods attacks in backscatter traffic} \frametitle{Motivation}
Attack description DDoS Attacks produce an observable side-effect:
\begin{tikzpicture}{scale=0.4}
\node[rectangle,draw,fill=red!80] (a) at (0,0) {Attacker};
\node[anchor=west] at (0.93,0.25) {Spoofed requests $H_{0},H_{1},H_{2},H_{3},...$};
\node [rectangle,draw,fill=blue!25,anchor=east] at (8,0) (v) {Victim};
\draw [->](a) --(v);
\foreach \x in {0,1,2,3} {
\node [rectangle,draw,fill=green!25,anchor=east] at (\x*2+1,-2) {$H_{\x}$};
%Horizontal lines
\draw (\x*2+1, -\x*0.25-0.5)--(7.0+\x*.25,-\x*0.25-0.5);
%Links to the victim
\draw (7.0+\x*.25,-\x*0.25-0.5) -- (7.0+\x*.25,-0.25);
%Links to hosts
\draw[->] (\x*2+1, -\x*0.25-0.5)--(\x*2+1,-1.70);
}
\end{tikzpicture}
\begin{center} \begin{center}
\begin{tabular}{|l|} \scalebox{0.8}{\input{../../preso/03-PassTheSalt/bsvol.tex}}
\hline
Connections\\
\hline
$H_{0}$\\
\hline
$H_{1}$\\
\hline
$H_{2}$\\
\hline
$H_{3}$\\
\hline
\end{tabular}
\end{center} \end{center}
\end{frame} \end{frame}
@ -424,60 +405,301 @@ Attack description
\frametitle{What can be derived from backscatter traffic?} \frametitle{What can be derived from backscatter traffic?}
\begin{itemize} \begin{itemize}
\item External point of view on ongoing denial of service attacks \item External point of view on ongoing Denial of Service attacks:
\item Confirm if there is a DDoS attack
\item Recover time line of attacked targets
\item Confirm which services are a target (DNS, webserver, $\dots$)
\item Infrastructure changes or updates
\item Assess the state of an infrastructure under denial of service attack
\begin{itemize} \begin{itemize}
\item Detect failure/addition of intermediate network equipments, firewalls, proxy servers etc \item {\bf Confirm} if there is a DDoS attack
\item Detect DDoS mitigation devices or services \item {\bf Recover} time line of attacked targets
\item {\bf Confirm} which services (DNS, webserver, $\dots$)
\item {\bf Observe} Infrastructure changes
\end{itemize} \end{itemize}
\item Create probabilistic models of denial of service attacks \item {\bf Assess the state of an infrastructure under denial of service attack}
\begin{itemize}
\item {\bf Detect} failure/addition of intermediate network equipments, firewalls, proxy servers etc
\item {\bf Detect} DDoS mitigation devices
\end{itemize}
\item {\bf Create} models of DoS/DDoS attacks
\end{itemize} \end{itemize}
\end{frame} \end{frame}
\begin{frame} \begin{frame}
\frametitle{Confirm if there is/was a DDoS attack} \frametitle{D4 in this setting}
\begin{block}{Problem}
D4 - for data collection and processing:
\begin{itemize} \begin{itemize}
\item Distinguish between compromised infrastructure and backscatter \item {\bf provide} various points of observation in non contiguous address space,
\item Look at TCP flags $\to$ filter out single SYN flags \item {\bf aggregate} and {\bf mix} backscatter traffic collected from D4 sensors,
\item Focus on ACK, SYN/ACK, ... \item {\bf perform} analysis on big amount of data.
\item Do not limit to SYN/ACK or ACK $\to$ ECE (ECN Echo)\footnote{\url{https://tools.ietf.org/html/rfc3168}}
\end{itemize} \end{itemize}
\end{block}
\input{flags.tex} D4 - from a end-user perspective:
\begin{itemize}
\item {\bf provide} backscatter analysis results,
\item {\bf provide} daily updates,
\item {\bf provide} additional relevant (or pivotal) information (DNS, BGP, etc.),
\item {\bf provide} an API and search capabilities.
\end{itemize}
\end{frame} \end{frame}
\begin{frame} \begin{frame}
\frametitle{Passive Identification of Backscatter (WiP)} \frametitle{First release}
\lstset{%
language=bash, \begin{itemize}
backgroundcolor=\color{gray!25}, \item[\checkmark]
basicstyle=\ttfamily, analyzer-d4-pibs\footnote{\url{https://github.com/D4-project/analyzer-d4-pibs}}, an analyzer for a D4 network sensor:
breaklines=true,
columns=fullflexible \begin{itemize}
\item {\bf processes} data produced by D4 sensors (pcaps),
\item {\bf displays} potential backscatter traffic on standard output,
\item {\bf focuses} on TCP SYN flood in this first release.
\end{itemize}
\item
analyzer-d4-ipa\footnote{\url{https://github.com/D4-project/analyzer-d4-ipa}},
\begin{itemize}
\item {\bf processes} data produced by D4 sensors (pcaps),
\item {\bf analyze} ICMP packets,
\end{itemize}
\end{itemize}
\end{frame}
\begin{frame}
\begin{center}
{\bf Passive DNS}
\end{center}
\end{frame}
\begin{frame}
\frametitle{Problem statement}
\begin{itemize}
\item CIRCL (and other CSIRTs) have their own passive DNS\footnote{\url{https://www.circl.lu/services/passive-dns/}} collection mechanisms
\item Current {\bf collection models} are affected with DoH\footnote{DNS over HTTPS} and centralised DNS services
\item DNS answers collection is a tedious process
\item {\bf Sharing Passive DNS stream} between organisation is challenging due to privacy
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Potential Strategy}
\begin{itemize}
\item Improve {\bf Passive DNS collection diversity} by being closer to the source and limit impact of DoH (e.g. at the OS resolver level)
\item Increasing diversity and {\bf mixing models} before sharing/storing Passive DNS records
\item Simplify process and tools to install for {\bf Passive DNS collection by relying on D4 sensors} instead of custom mechanisms
\item Provide a distributed infrastructure for mixing streams and filtering out the sharing to the validated partners
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{First release}
\begin{itemize}
\item[\checkmark]
analyzer-d4-passivedns\footnote{\url{https://github.com/D4-project/analyzer-d4-passivedns}}, an analyzer for a D4 network sensor:
\begin{itemize}
\item {\bf processes} data produced by D4 sensors (in passivedns CSV format\footnote{\url{https://github.com/gamelinux/passivedns}}),
\item{\bf ingests} these into a {\bf Passive DNS server} which can be queried later to search for the Passive DNS records,
\item{\bf provides} a lookup server (using on
redis-compatible backend) that is a Passive DNS REST server compliant to the Common Output Format\footnote{\url{https://tools.ietf.org/html/draft-dulaunoy-dnsop-passive-dns-cof-04}}.
\end{itemize}
\end{itemize}
\end{frame}
\begin{frame}[t]{Common Output Format}
\begin{itemize}
\item {\bf Consistent naming of fields across Passive DNS software} based on the most common Passive DNS implementations
\item Minimal set of fields to be supported
\item Minimal set of optional fields to be supported
\item Way to add "additional" fields via a simple registry mechanism (IANA-like)
\item Simple and easily parsable format
\item A gentle reminder regarding privacy aspects of Passive DNS
\end{itemize}
\end{frame}
\begin{frame}[t,fragile]{Sample output www.terena.org}
\lstdefinelanguage{JavaScript}{
keywords={typeof, new, true, false, catch, function, return, null, catch, switch, var, if, in, while, do, else, case, break},
keywordstyle=\color{blue}\bfseries,
ndkeywords={class, export, boolean, throw, implements, import, this},
ndkeywordstyle=\color{darkgray}\bfseries,
identifierstyle=\color{black},
sensitive=false,
comment=[l]{//},
morecomment=[s]{/*}{*/},
commentstyle=\color{purple}\ttfamily,
stringstyle=\color{red}\ttfamily,
morestring=[b]',
morestring=[b]"
} }
\input{pibs.tex}
Early version is available of PIBS\footnote{\url{https://github.com/D4-project/analyzer-d4-pibs}} \lstset{
with a focus on TCP traffic. language=JavaScript,
\begin{tabular}{l|l} backgroundcolor=\color{lightgray},
Options & Explanations\\ extendedchars=true,
\hline basicstyle=\footnotesize\ttfamily,
-r & read pcap file\\ showstringspaces=false,
-b & display IPs under DDoS on standard output\\ showspaces=false,
\end{tabular} numbers=left,
numberstyle=\footnotesize,
numbersep=9pt,
tabsize=2,
breaklines=true,
showtabs=false,
captionpos=b
}
\lstset{breaklines=true, language=JavaScript}
\begin{lstlisting}
{"count": 868, "time_first": 1298398002, "rrtype": "A", "rrname": "www.terena.org", "rdata": "192.87.30.6", "time_last": 1383124252}
{"count": 89, "time_first": 1383729690, "rrtype": "CNAME", "rrname": "www.terena.org", "rdata": "godzilla.terena.org", "time_last": 1391517643}
{"count": 110, "time_first": 1298398002, "rrtype": "AAAA", "rrname": "www.terena.org", "rdata": "2001:610:148:dead::6", "time_last": 136670845}
\end{lstlisting}
\end{frame}
\begin{tabular}{l} \begin{frame}[t]{Mandatory fields}
Dependencies\\ \begin{itemize}
\hline \item \textbf{rrname} : name of the queried resource records
libwiretap-dev\\ \begin{itemize}
libhiredis-dev\\ \item JSON String
libwsutil-dev\\ \end{itemize}
\end{tabular} \item \textbf{rrtype} : resource record type
\begin{itemize}
\item JSON String (interpreted type of resource type if known)
\end{itemize}
\item \textbf{rdata} : resource records of the query(ied) resource(s)
\begin{itemize}
\item JSON String or an array of string if more than one unique triple
\end{itemize}
\item \textbf{time\_first} : first time that the resource record triple (rrname, rrtype, rdata) was seen
\item \textbf{time\_last} : last time that the resource record triple (rrname, rrtype, rdata) was seen
\begin{itemize}
\item JSON Number (epoch value) UTC TZ
\end{itemize}
\end{itemize}
\end{frame}
\begin{frame}[t]{Optional fields}
\begin{itemize}
\item \textbf{count} : how many authoritative DNS answers were received by the Passive DNS collector
\begin{itemize}
\item JSON Number
\end{itemize}
\item \textbf{bailiwick} : closest enclosing zone delegated to a nameserver served in the zone of the resource records
\begin{itemize}
\item JSON String
\end{itemize}
\end{itemize}
\end{frame}
\begin{frame}[t]{Additionals fields}
\begin{itemize}
\item \textbf{sensor\_id} : Passive DNS sensor information
\begin{itemize}
\item JSON String
\end{itemize}
\item \textbf{zone\_time\_first} : specific first/last time seen when imported from a master file
\item \textbf{zone\_time\_last}
\begin{itemize}
\item JSON Number
\end{itemize}
\item Additional fields can be requested via \url{https://github.com/adulau/pdns-qof/wiki/Additional-Fields}
\end{itemize}
\end{frame}
\begin{frame}
\begin{center}
{\bf Passive SSL revamping}
\end{center}
\end{frame}
\begin{frame}
\frametitle{Objectives - TLS Fingerprinting}
{\bf Keep} a log of links between:
\begin{itemize}
\item x509 certificates,
\item ports,
\item IP address,
\item client (ja3),
\item server (ja3s),
\end{itemize}
\begin{displayquote}
``JA3 is a method for creating SSL/TLS client fingerprints that should be easy to produce on any platform and can be easily shared for threat intelligence.''\footnote{https://github.com/salesforce/ja3}
\end{displayquote}
{\bf Pivot} on additional data points during Incident Response
\end{frame}
\begin{frame}
\frametitle{Objectives - Mind your Ps and Qs}
{\bf Collect} and {\bf store} x509 certificates and TLS sessions:
\begin{itemize}
\item Public keys type and size,
\item moduli and exponents,
\item curves parameters.
\end{itemize}
{\bf Detect} anti patterns in crypto:
\begin{itemize}
\item Shared Public Keys,
\item Moduli that share one prime factor,
\item Moduli that share both prime factor,
\item Small factors,
\item Nonces reuse / common preffix or suffix, etc.
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{First release}
\begin{itemize}
\item[\checkmark] sensor-d4-tls-fingerprinting
\footnote{\url{github.com/D4-project/sensor-d4-tls-fingerprinting}}:
{\bf Extracts} and {\bf fingerprints} certificates, and {\bf computes} TLSH fuzzy hash.
\item[\checkmark] analyzer-d4-passivessl
\footnote{\url{github.com/D4-project/analyzer-d4-passivessl}}:
{\bf Stores} Certificates / PK details in a PostgreSQL DB.
\item snake-oil-crypto
\footnote{\url{github.com/D4-project/snake-oil-crypto}}:
{\bf Runs} weak crypto attacks against the dataset.
\item lookup-d4-passivessl
\footnote{\url{github.com/D4-project/lookup-d4-passivessl}}:
{\bf Exposes} the DB through a public REST API.
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Future}
\begin{itemize}
\item {\bf Sensitive information sanitization} by specialized analyzers
\item {\bf Previewing datasets} collected in D4 sensor network and providing {\bf open data stream} (if contributor agrees to share under specific conditions)
\item {\bf Leverage MISP sharing communities} to augment Threat
Intelligence, and provide accurate metrology.
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Use it}
\begin{itemize}
\item {\bf Create} sensors easily with the generator \footnote{\url{https://github.com/d4-project/d4-sensor-generator}},
\item {\bf Manage} your own sensors and servers, {\bf find} shameful bugs and
{\bf fill} in github issues
\item Even better, {\bf send} Pull Requests!
\item {\bf Share} data to public servers to improve the datasets (and detection,
response, etc.)
\item {\bf Feed} your MISP instances with D4's findings - {\bf Share} yours
\item {\bf Leech} data, {\bf write} your own analyzers, {\bf do} research
\end{itemize}
\end{frame} \end{frame}
\begin{frame} \begin{frame}
@ -485,9 +707,17 @@ Options & Explanations\\
\begin{itemize} \begin{itemize}
\item Collaboration can include research partnership, sharing of collected streams or improving the software. \item Collaboration can include research partnership, sharing of collected streams or improving the software.
\item Contact: info@circl.lu \item Contact: info@circl.lu
\item \url{https://github.com/D4-Project} - \url{https://twitter.com/d4_project} \item \url{https://github.com/D4-Project}
\item \url{https://twitter.com/d4_project}
\item \url{https://d4-project.org}
\begin{itemize}
\item
\href{https://d4-project.org/2019/05/28/passive-dns-tutorial.html}{Passive DNS tutorial}
\item
\href{https://d4-project.org/2019/06/17/sharing-between-D4-sensors.html}{Data
sharing tutorial}
\end{itemize}
\end{itemize} \end{itemize}
\end{frame} \end{frame}
\end{document} \end{document}

2
docs/workshop/1-passsive-ddos/d4-passive-ddos.tex
View File

@ -18,7 +18,7 @@
\author{Team CIRCL} \author{Team CIRCL}
\titlegraphic{\includegraphics[scale=0.20]{d4-logo.pdf}} \titlegraphic{\includegraphics[scale=0.20]{d4-logo.pdf}}
\institute{Team CIRCL \\ \url{https://www.d4-project.org/}} \institute{Team CIRCL \\ \url{https://www.d4-project.org/}}
\date{20190329} \date{20190923}
\begin{document} \begin{document}
\begin{frame} \begin{frame}