and I'll pass over the mic all right
thank you
yeah great, good morning good afternoon
and even good evening for some of you
um so um i'm really glad and uh
we are glad to present about MISP today
and so it's a
double series of workshops so we start
with a workshop of the introduction and
we go
more deeper tomorrow in that second
workshop
um i'm alexander noah I do work for
CIRCL and
i work in the MISP {inaudible}
so today uh the agenda is the following
uh we will do a quick introduction to
MISP
a kind of of one-hour sessions with
all the detail about MISP and then a
more like kind of usage deep dive
of one hour where we do hands-on together
um for the logistic aspect um
in the chat room we will share with you
uh
all the details how to access the MISP
instance so during the sessions in the
workshop
you can connect to a dedicated MISP
system that we set up for you
and this one will be used for all the
hands-on that we will
we do as I just mentioned we have a
small short break of 15 minutes
and then we will continue in the end
{inaudible}
depending of how far we are today uh
we will maybe talk about the community
building aspect but this is a topic for
tomorrow obviously
but if we have some time remaining we
might uh talk about this and then we
have a
q&a sessions uh to discuss about the
different {inaudible} and so on
so don't hesitate to to put your
question to zoom
uh directly and we will try to answer live
all those questions that you are asking
during uh during this session
so first of all welcome all as well from
me so i'm Andras Iklody i'm
also working at CIRCL working on MISP
um to just kick things off um I think
it's a good
good moment to start a little bit about
the history of how this whole thing
started how MISP came about
so just a quick introduction of where we
came from and where where we are
nowadays uh initially this whole thing
for us with MISP
started as part of a of a series of
incidents that we had in
back in 2012 between national and
military CSIRTS at the time
where we were basically investigating umattacks
that were hitting several of the
institutions at the at the time
and one of the interesting things that
we found was that even though we had
something called the malware analysis
working group which was this which is a
group that was regularly meeting and
discussing ongoing incidents
we still had a massive gap in between
those meetings
where everyone was working in their own
silo on basically the same attack and
and doing reverse engineering of the
same attacks without
having the ways, the means or the processes
to directly share with our peers so we
ended up with a lot of duplication of
work which ended
which was obviously frustrating from uh
for many of us
so Christophe Vandeplas at the time he
was working at the belgian
ministry of defense um in his free time
wrote a platform called {inaudible}
that later on ended up becoming MISP so
the initial idea was
really for reverse engineers to share
the output of their work directly with
their peers
in a hosted platform and since then
obviously MISP has evolved and changed
the scope of what we were nowadays doing
with MISP and what sort of information
we're sharing
but it all started with this and since
then it has been an ongoing effort
basically by
a large community of different
requirements and different needs
and that has been building both the
ideas that go into MISP as well as the
software itself
next slide please
yeah so what is the background and why
we are doing
MISP it uh I think like Andras
mentioned it started from a with a kind
of
{inaudible} project for a small set of
CIRCL
CIRCL is nowadays
the CERT for the private sector, the
community {inaudible} in luxembourg
and we basically deal with the
development of MISP not only for our use
case but for many
different users so we are called as
a CERT we basically operate the
development and we operate
{inaudible} communities
so a little bit about our involvement
and
why we're doing this in the first place
so we as CIRCL we're funded by the
Ministry of Economy to basically build
security for the private sector uh and a
lot of what we do involves uh open
source software development so we're
basically
the funding that we get for the uh for
activities also cover our development
focus
we're also uh besides just building the
tools like {inaudible} mentioned we're
also
basically involved in a lot of sharing
activities as well as our day-to-day
operations we're users of the
tool primarily as well we basically host
a bunch of different communities for the
uh national C-Certs for the
luxembourgish private sector community
for law enforcement uh organizations
financial institutions and so on and so
forth
so so we're kind of uh
in the game on both sides so to say that
both as a
producer and as a consumer, also and the
project was
co-financed by the European Union
under the CEF project
uh so this is also one of the sources of
the income that we got basically to
build the tool and as a FIRST member you
have access to a MISP instance that is
operated and
co-maintained by FIRST and CIRCL that
you can get access to
so you just need to to use your
traditional credential or
access at first and you get access to
this instance with information
that you can use and so on we will talk
about that later on
so the main question and I think it's
coming from this story uh
as Andras mentioned from the early days
MISP was
focusing from a very specific aspect
which was malware reversing and so on
nowadays it's a threat intelligence sharing platforms we
are basically sharing any kind of
intelligence through uh through MISP
um because we had an evolution of the
time for the different things so
and our main goals and that's very
important for us it's an open source
software so that means MISP will always
remain an open source project we even
take
some decisions within the project to
keep it as
open source and that's that's really a
key for us so it's really uh
something that you can download yourself
run on your infrastructure and so on
you can really have the full control on
the software stack
when you are using MISP and one of the
goals of the software itself is to
collect information
from other partners, from in the {inaudible}
from automatic tools from different
fields and so on
that was really one of the initial goal
of MISP is like being able to collect
to get all this information into
one place
and then afterwards what you can do with
it is to normalize
correlate this information, extend the
information and enrich
information with more information and then
really benefit from the sharing aspect
of MISP and you can allow
teams and community to collaborate
and we have
seen MISP for example use not only
within different organizations but even
within a single organization you can, for
example run multiple MISP
to collaborate directly on different
investigations
incidents and cases and obviously when
you have all this information and all
these analytic platform into MISP
you are ready to use this information to
for example
feed your automatic protective tools
like intrusion detection systems,
firewalls whatever and to feed
automatically those
information to basically make protective
measures
in your environment
so, start from the starting point that we
already mentioned basically how we
started out with MISP
let's have a quick look at how uh the
user base of MISP evolved in terms of
the different types of stakeholders
within our own organizations and other organizations
the reason for that is obviously that
this drives the development process as well
so the way MISP grows over time really
depends on the type of users that are using
the type of users that are requesting new
features or that are providing pull
requests on the project and providing code
for the project so I said before
initially
the scope of MISP was very limited it
was basically just the output of malware
reversers which meant
raw indicators that we were extracting
during the process and that we're
sharing directly
with our partners, this meant very little
analysis was done on each of these
individual indicators there was very
little information in terms of
of why those data points are relevant in
the long term how they are meant to be used
from detection perspective they were
really just the raw output
from the analysis process or the
reversing process
now one of the side effects of this when
you start building a collection within
your organization of this information
the security analysts are feeding your
various protective tools become
interested in that data set
because obviously whatever is targeting
your organization or your direct peers
are probably the most relevant piece of
information that you can use for
detection
so one of the first steps that we opened
up to was basically involving our own
security analyst with our
own organizations so they can hook
the output of
of the reverse engineering team
up to their SIEMs to their IDSs to
their firewalls
and to feed this data directly into
their protective measures
now one of the interesting things when
you start doing that though is that you
generate a new type of output which is
timeliness for the data, freshness for
the data as well as feedback on how
useful the data was
so we're very often when you're
extracting information
by sandboxing for example a lot of the
data generated will be noise in the end
and this noise will generate false
positive alerts for example
in your detection tools. Now feeding this
information back
to data gave it a whole new type of value
we had freshness so if an older
indicator was reused
over time we saw that that is still
something that is actively to be monitored
and if we saw that something turned out
to be a cleaned up host
in the meanwhile or something was a
false positive from the get-go
we could feed that information back as well
so suddenly once you have timeliness as
well as the the raw data itself
you get the intelligence analyst
interested that are tracking the
movements and the changes of how attackers operate
uh over time so that means that usually
back then especially in 2012 in most of our
organizations the people that are doing
intelligence and the people that were
doing operations and security for the
operations were usually working in their own silos
so while there was obviously interaction
between the teams, it was not as
ingrained to work together
between those type of roles but this
changed over time and one of the changes
that we saw happened was that the output
of what the security analysts
and the reversers and the analysts were
outputting basically from the operation side
became more and more interesting for the
intelligence analysts that meant that
if they were tracking a certain threat
actor and they could attribute certain
actions that they were seen in the
network of the organization
to the certain threat actor they could
monitor for example how
the given actor was changing how fast
they were changing infrastructure
whether they were switching up their
methodology and this
gave them a lot of idea of useful data
of improving their libraries of the
threat actors that they were tracking
so suddenly we got this group interests
as well and they were obviously
producing data as well so nowadays if
you look at MISP
going from a raw indicator sharing platform
that MISP was initially
nowadays you have a lot of the high
level threat intel information included
with the data as well so you will see threat reports
you will see interconnected information
about threat actors modus operandi
infrastructure impact and so on so forth
these extra layers of information that
we were missing initially
so this was the biggest change that we
had over time within our own
organizations but
obviously as a CSIRT that has
different constituencies
we're also interacting with the security
teams of other organizations and one of
the things we noticed early on was
there's a lot of the issues that other
types of organizations had internally
with information sharing are very similar to ours
so initially the first use case that we
got that was different and from our
normal security use case
was basically the various financial
organizations reaching out to us saying
that their fraud teams were
running into similar sort of issues with
sharing between their teams,
sharing with other partner teams,
information about mule accounts and
about other fraud related information
so they reached out to us and
basically their security teams reached
out to us and said
can't we just try to help them also
to share that sort of information
through MISP directly, I mean we already
had the tooling in place
it was just a question of changing the data model
so we we started doing that for uh together
with the financial sector initially
where we expanded the data model of MISP
when we allowed for modeling of new
custom data types
and it's even surprising to us at the
time turned into a success
very rapidly so nowadays we're involved
with quite a few different types of
organizations out there
replicating the same scenario where for
example law enforcement, where we initially had
mostly contact with their security teams
and helping them build data sets for bootstrapping their forensic
investigations nowadays we have all
sorts of information sharing involving
uh for example uh seized goods information sharing from
border control agencies uh law enforcement agencies
sharing information about passenger information
so a lot of the type of data
sharing that was very unusual for us as
a CSIRT initially
now once you get all this data in a
system and you
started building a data set from your community
you start to see trends in the data set
and this is what gets
for example our risk analysis team
interested in it the moment that you're
seeing how attackers are changing their
trends over time
you can better advise your constituency your customers and so on
about the different risks that they might be facing and the
different risks that they should be preparing for
and preparing the organizations for
based on what the same sector is facing
perhaps in the same geographic location
so suddenly you get a lot of knowledge
out of the collected data as long as
data is well contextualized and I think
this will be one of the main topics that
we're going to be talking about quite a bit today and tomorrow
is contextualizing the information and
making the information actually usable
and turning data really into knowledge
yeah so like Andras mentioned
we have a pretty large set of different
communities using MISP
and over the time it became I think more
complicated to handle all those requests
from different organizations
um so we came with a model of governance
even if it's a very lightweight one we
decided to have this kind of models to
still benefit from the open source
community model and then
bring all the experience from a
different community into systems where
it allows us to develop and extend the
software so we decided to
create this kind of models where we
basically
take care of all the features and
requests that we receive from different organizations
so we use this kind of priority list of different features
and we get that feedback from {inaudible}
one of the {inaudible} one I would say is
Github so we get the feed from
the different issue that we receive from
Github I mean on the if you look at on
Github you'll see that we have a
significant number of issues and those
issue are usually for us a way to track down
all the different requests of features
in MISP and that's one way to get it.
Another way and this one is a quite a common one
is basically a training or session like
this where people are providing feedback,
bug reports, future requests and so on
directly during the training and for us
uh I think really practical and we can
get all this information needed for us
to improve the software. Another thing
that we do and
that's maybe for the audience some
people are interested in that one
we know that there are plenty of
different MISP of groups that we don't
control and that we don't manage that's
great we have for example ISACs, ISAO
doing really those kind of things where
you have those kind of user groups
and what we do we participate on a
regular basis to one of those groups for
example on a quarterly basis on a yearly basis
and we do a collection of requirements
from those different groups
during one session and that's really
i think useful for us because it's a way
to to gather information so for example
Andras mentioned those
financial groups where people are
sharing information about bank account
detail and so on
and that's where we basically gather all
those requirements
so if you are setting up a group
somewhere in US or
in the world about sharing information
and so on and you want to
invite us at some point in time it's a
way for us to gather those kind of requirements
we do a summit which is a yearly event
usually it was physical but nowadays
it's virtual trainings
so it's basically every user or
organizations using MISP presenting what
they are doing
it's a way for us to to see the
interactions and see what
can be improved in MISP and see {inaudible}
in the community behind
and then we have a kind of 20%
project around in MISP
where we design new functionalities and we test
them out for example one of those is the
detailing of indicators
which was a request from different
organizations but it was kind of
difficult to design
and with this kind of models where we
designed first as a kind of prototype
multiple iterations uh we did a multiple
research paper on that and then finally
this become part of the MISP core software
we will show that later on but so we
have a lightweight governance model
but really the goal is to gather as
much feedback from the user so don't
hesitate if you have any bug reports, ideas and so on
either open an issue,
get in touch with us.
You are more than welcome to basically
share such kind of information.
Yeah, now addressing the elephant in the room
when you bring so many different
organizations together and build a large
community of sharing with different
needs and requirements
you're obviously going to have to run
into conflicting requirements as well
so one of the most obvious ones that
that we're dealing with very often with
information sharing and something that
that we're working on tackling ,
basically since we started with MISP, is dealing
with a different requirement of
of what you count as valuable
information depending on your use case
so this is different also within
different analysts, different roles
within the same organization as
well so for example, for us as a CSIRT in general
uh detection is the most important
matter so we're interested
in in using indicators to detect if our constituency
is affected by something that the
information is being
shared about or whether uh any of the
the infrastructure that we're
responsible for is infected
on the other hand if you're talking to an isp
one of the large
requirements from an isp basically will
be able to protect their users
from potential harm so that means that
if there are any urls, websites, and so on
that they should block for their users they
need to be able to generate a block list
out of the data
that is considered to be malicious
enough now if you compare these two use cases
with each other detection versus blocking
you will immediately see that the effect
of having a false positive in the data
set or data that is no longer fresh
has a completely different impact
sure for us when
that are mostly in the detection game
it's annoying we get a false positive
alert it has to be handled
and it takes time and effort it also
introduces something called alert fatigue
that i'm sure many of you are familiar with
if you're getting a lot of false
positive alerts you're more likely to ignore the
next alert that you get but besides that
it has no real operational impact on us
on the other hand for an isp that ends up blocking
something that is uh
potentially a false positive might have a catastrophic impact
imagine if someone accidentally, for example shares
facebook.com as an indicator that might
basically cause a riot with their users or it might {inaudible}
but it's a different story
but with that in mind, you see that these
two use cases are already conflicting
now if you also take the perspective of
intelligence analysts that are tracking
threat actor movements in to account
that's an even more lax use case
where you care about whether something is a fresh indicator still or not
even less than the other two groups.
The reason for that is you're interested in
the historical movements of a threat actor, for example.
So even if something is no longer
an indicator because and an infected
website was cleaned up
since the time when the indicator was
shared they still want to see
how long, for example a threat actor was
using that infrastructure,
how quickly they changed to something
else and what methods they used
back when they were exploiting it.
So if you bring these different requirements on board on
the same platform is difficult and there are some
things that we can do to alleviate these
issues. For example what we do with
MISP
we have a system called warning list
system that allows us to filter out
obvious false positives
so we maintain these lists of
most common websites, empty hash lists,
public dns resolvers and all these
typical things that end up in
the sets while doing automatic extraction for example
that end up being false positives but
with that said this is just one part of the story
So if you're looking at the different
use cases up there that doesn't solve our issue
of having different requirements
from the data set based on what you do with it
and this is where contextualization
becomes more important again
if we can supply the information together with the data,
why this data is relevant and what context you're
supposed to be using it
then the consumers of the data can make
those decisions for themselves based on
whatever they want to use
the data for in any of those different use cases
so one of our main efforts with MISP has been
to be able to provide these different
structures together with the data and to
be able to label data well enough. Back to you.
Yeah so and that's iI think important regarding the
different kind of use cases and so on
and we try to support those different use cases and
that's sometimes challenging for us but luckily
we are at the same time
part of various community so we can see
the different use cases, especially
regarding the handling of false positive which is
an ongoing challenge but we will show
you how to handle that
and at the same time we basically
operate those different communities.
So for example we operate a pretty large
one for the private sector
where we have a lot of organizations,
more than 1200 organizations are basically connected there.
It's pretty large and we see an active
community sharing information and
there is plenty of different communities
some that we don't know even about
because you can even run your own
private communities without telling anyone, that's fine.
That's part of the system but if you want to have different kind of communities
you can connect those automatically then you have I would say
different kind of model you have those kind of
fully island mode communities.
Those kind of trusted groups so for example for the
intelligence community it's very common for them to run MISP
in an island mode so having air gap system and so on
sometimes they are partially connected
with third parties to share partial
information so for example we know some organizations
or for example border controls or customs
are using MISP but they still need to
share some small information and that
partially connected system.
MISP freely supports those kind of models
and then you have community that are
more broad and more large
for example in the financial sector and
I think the CSIRT Luxembourg has some banks
we are involved in various sharing communities
at European level and worldwide level
where for example we know some ISACs that are dedicated to
the financial sector are using it as a sharing mechanism
you have some organizations that are really
dedicated to a payment processing system
that are using this to share automatically
information and so on or analysis
One of the I would say pretty large community too is
with the military organization and international organizations
FIRST for example, you have a lot of FIRST members using
MISP for sharing their information
but there are plenty of networks, national governmental networks
a military one intelligence, one or even NATO for example are using
using MISP so maybe some of you are eligible to access those ones
so we have on the MISP an interface a way to connect to those
community and you can reach out to the
different community by asking for access for example
then you have very specific communities
that are set up by security vendors it's not uncommon
tp see for example a security vendor
services their own MISP
we have seen for example some
{inaudible} agents vendors running a dedicated MISP
or even some operators of specific cloud
services running a MISP instance
to share information amongst
their different customers.
Then you have communities that are
i would say very specific on the topic
for example you have about sick information uh false news
and stuff like that you have communities doing that
for example we cooperate one called the COVID-19 MISP
which is really targeting COVID-19 as a topic
and then you have 10 different subtopics like
cyber security, health related topics and so on.
So you can see that MISP can be really used on
different model of communities
you can bridge those communities,
you can interconnect those with together,
you can keep it for yourself, so it's
really a matter of models.
Worldwide there are I would say a lot of
communities that we are not aware of
but we as CIRCL operates
around 20 communities nowadays, that you
can basically get access
and Andras just sharing in the chat the
access to the COVID-19 MISP and if you want to get access to
that one you can connect on the main page and self-register and
you can request access to that community
so you see that MISP has different groups different communities
and it's up to you at the end to decide
which kind of community you want to {inaudiable either be/visit}
So, a little bit besides all the technical things
that we talked about, that we do with MISP,
and that we try to solve with it.
In terms of sharing, there are obviously
going to be other hurdles that you have
to overcome whenever it comes to information sharing
one of the the toughest things to
overcome and this is where no tool can really help you
is to get enough trust in a community to
be able to share your information with them
So the only way to facilitate this is really social interactions
so sadly though we're living in times
where social interactions are tougher than usual
but for example events like FIRST conferences
are great ways to get to know your community and to
build this trust and build those
social relationships that you need
to be able to really exchange meaningful
information with the community
so I really encourage everyone that
wants to partake in information sharing communities
to be social, to reach out, and to get to know your community
because that's the biggest facilitator for sharing in the first place.
Other than that, there are obviously some
legal restrictions that you have
that might come up in the entire process.
We see this very often with organizations where the first
questions that they ask us when they join
in our communities okay how does this
fit into GDPR for example.
If my legal team asks me why I am sharing
an information out what can i
give them as an explanation of why i'm
supposed to or allowed to do this.
So if you need any help with that we
have a bunch of compliance documentation
and that we've been working on together with a bunch of partners
and so we have descriptions for how
MISP fits into the GDPR, the NIS directive
and some other frameworks so just
have a look there and if you have any
questions or if you feel that anything is not covered
let us know and we keep updating our documentation
based on on feedback of what's missing
or ideas that we should be incorporating in there
but generally , once your legal team is more
familiar with the process and why this
is {inaudible done/tied}
why ensuring security for your
organization and for the data that you
have to secure is important then it's seen more as a
benefit than a hurdle really
but it obviously takes time to get
this into your processes to define why you're
doing what you're doing
your retention periods,
describing how you're going to handle data and so on
so this obviously has some ramp up time
but we have a lot of documentation that will help you with that.
There are also some practical restrictions that we hear from
organizations so very often when
organizations reach out to us
the first thing they say is we don't
really have any information to share,
we don't have the capability for example
to build those highly vetted threat reports that we're so used to
from feed providers and obviously very few organizations do.
With that said information sharing comes
in many different shapes and sizes for example going back
to the initial use case about
providing feedback from your analysts
about the data that you receive from
your community is already valuable
information sharing so if someone for
example provides sightings
I've also seen this indicator at this given time
that can already help you tune the data set
for what goes into your working data
sets for detection and blocking and so on.
Also providing information on false
positives and some information that
you provided to the community turns out to be false
or something that is no longer relevant
getting information that is valid as
well so pretty much everyone has
information to share by just using the information and running
into frustration with the data by itself.
Also besides not having information to share
there comes also the issue of time.
Most of us are overburdened with
the different tasks that we are facing nowadays
so taking extra time out of the day to
encode information and to share it out in the community
is obviously going to be an extra burden
there is no way around it.
What we try to do with MISP
is to make this process as minimal as
possible but it is going to be a time investment in the end, after all
especially if you want to vet the data if you want to ensure that
the right data reaches the right recipients
This always has a time drain on you as well but in return this
is offset by what you gain by sharing that information we're
going to talk about this a little bit
more during the community building part
about what effects you're going to see
if you're sharing information and why it is relevant for you
but to basically sum it up in one sentence
and whatever affects your organization
is probably the most important information for you and if you get
feedback on that, what you're seeing in your network
and more eyes on it, more perspectives
and perhaps more sophisticated methods of
research from other organizations
then that will probably just improve
your own security posture the best way it can.
Now, besides timeliness and basically having information to share
there's also the issue of different
classification models so classification
not just in a sense of of deciding who we share information with
but how we classify information really
in terms of contextualizating it we are all used
to naming things a certain way in our organizations in
our communities and we've probably
been doing it for longer than digital information systems exist
so we're probably using a lot of those
vocabularies that we've been using for decades
and what one of the things that we
wanted to avoid with MISP is to
get these communities to switch to a
different way of describing things so if you already
have your set methods, your set processes
how you define things, we don't want to alter that so one of
the things that we do with MISP and we are
going to talk a fair bit about, tomorrow mostly
is that you have ways to describe your
own taxonomies and your own vocabularies
to use those in your community so very
often when you're spinning up a
community and when you're starting out
with the sharing community,
a national sharing community, sectorial one, whatever
then one of the first tasks is basically
defining those common vocabularies
that you're going to be using
now apart from the vocabularies
themselves there is also the issue of
of us speaking many different languages
in terms of of our tools using different formats
so that means even within our own organization which is
rather small we have a set of different tools
that will ingest data in different formats
or will prefer to ingest data in given
format so one of the things we also try to do with MISP
is to act as a hub for all your different tools
that will get their data translated into
the format that they can best ingest.
Obviously this is something where we cannot be completely
100 percent covering all the other
things that exist out there.
So one of the things we try to do with MISP
is make it as modular as possible and
it's easy to encode your own formats as possible.
We're not going to go deeply into how to do this during the training
but if anyone is interested about that just
let us know and we'll point you in the
right direction where you can find
documentation on how to modularize and
how to build import and export in MISP.
So just one side note, all the training
materials are available online
like {inaudible} mentioned we have a Github
repository with a pretty extensive README files with all
the material that we provide, there is a MISP book too which includes a
lot of reference to MISP as you know MISP has a
pretty large topic coming from technical aspect and
you will see that in a minute about the project overview.
So don't hesitate to go there on the MISP training
page on Github this one is a good
reference because it's really pointing
to the different elements
that we have. We have a huge slide deck of
close to 500 pages of slide deck on the
MISP book we have close to 500 pages. I
would not mention the number of pages
for taxonomies, galaxies and so on. It's quite large too
but really look at this as a kind of way
to shape it to what you like.
So it's really there to help you and if
you see something missing
let us know but we have slides,
for example system requirements, things like
for example building community that
we'll talk tomorrow, that's
part of it but for more the
programmatic aspect, API
how to integrate with MISP {inaudible JSON/taxono},
how to extend it too
there are plenty of slides regarding that
so it's really a good reference
and thanks to {inaudible} to share this
information on the chat
so to just give a quick overview of the MISP project and really to show that
the project is quite large nowadays
we basically have like four pillars of things in MISP
one is obviously the open software itself
so the initial version in {inaudible} it was
the small first small block there
the MISP core software which is like just the software
mainly for the LMAP aspect where
you have the backend, the web interface,
and so on but over the time the project extended
with multiple things so if you look on the Github
repository of mid project we have around 50 repositories so
it's pretty large. Just to summarize what
are the different one
we have for example the MISP modules um
which is an easy way to extend MISP so the behavior of MISP
on the expansion side on the import, export and so on by just writing
python modules it's super easy to develop and use
and the idea behind is obviously to
extend MISP without knowing
the core details about the system
then we have a library called PyMISP and this
PyMISP library is basically a
python library to expose the new MISP platform API
so MISP has a large REST api this one can be quite large but
by MISP is really helping you to for example {inaudible jest/Get} events
create feeds and stuff like that so it's
really important if you want to
extend MISP to have a look at PyMISP that
is not the only library for extending
MISP some in golang you have some
other in python too, you have others in java and so on
but the PyMISP one is the one that
is maintained by the author of MISP so it is maintained by us
and you can have a look at this one it's
really the one that's up to date it's
really core and part of the system too
because we use it for our own tests
within MISP then we have different
repository I will just mention one which is
dashboard, the dashboard is an extension module
in MISP using what we call the ZeroMQ feed in MISP
so we have a kind of way to have kind of a real-time feed
in MISP you can {inaudible} and
so on but we wanted to show an example
application for that and the MISP
dashboard is exactly that
is a way to really get all the
information that you have in MISP
into a very nice dashboard and so on
this is really to have a good example of what you can do
with information within MISP and how you can use it
so that's the main pillar you have
plenty of other projects but those one are the main ones
on top of that you have
what we call the intelligent and knowledge database of
MISP and just mentioned about the difficulty
sometimes in some organizations to use a
{inaudible proper/corporate} classification and so on
and we try to ease this in this
different organization by having a kind
of library of all the taxonomies that exist
so we started as a very simple one where
it was just including for example a
taxonomy like the traffic light protocol one, FIRST is using it
and it's a commonly used classification but over
the time we have seen that many
organizations have different
classification and so on. So we already
in advance we prepare all those taxonomies in
possible information {inaudible} expose MISP
and you can enable the one that you want
so we have around 150 libraries now
ranging from classifications, specific one for
intelligence communities and some
other activities so this one is our
really useful label and you can just
{inaudible share/pick} the one that you want and we maintain those one
so we have some that are coming from
third party, some that we are collecting
as each projects are creating.
It's really usually a good source to see
how other communities are using
classifying and contextualizing the
information, there nevertheless the
taxonomy itself was like kind of labels, those labels were quite small
so it was not like completely extensive information so
over the time we maintain a kind of more extensive one called the galaxy
you will hear the term very often
those galaxies are defining many things
for example one of the most common one is the threat actor
we have a huge database of threat actors but a lot of
times it was extended
for example, Microsoft is not using
threat actors for example there is this activity group is part of the
galaxy, it's really one that we
use for different and you can represent whatever
galaxy you want so you have a predefined
set of existing one but you can create your own
so if you have your own threat actor database
you can create your own from scratch or
you can reuse and fork existing ones so
that's really those kind of things that we manage in
the project is not only code and software
we manage those kind of knowledge base
for intelligent {inaudible} organization
we have some specific one like the notice list
this one is a pretty small one that you use for the GDPR aspect
but this one can be used for anything you want,
It's for informing the analyst or the user of MISP when he
touched some specific information in MISP
that could impact for example the legal framework and so on
it's actively use in the intelligence community,
law enforcement and so on maybe less in
security operation center but it's
coming more and more due to the legal
side of information sharing and
especially storing information that might contain personal information
then we have another one called the
warning list and Andras quickly mentioned this kind of recurring problems or false positives
and the one in MISP are basically list of existing potential false positives 1305
for example we have lists of well-known IP addresses from Microsoft, for example.
We have list of things like domain names used by Google and so on
that's already helping users to find out
if something might be a false positive
and we do that automatically and we
maintain those libraries because one {inaudible MISP/reason} they're automatically updated regularly
I think we have around 50 lists nowadays
It's really useful when you do on a day-to-day basis and creating events and
so on you can really find and spot things that might be a false positive in advance
by having those warning lists enabled
and again it's up to the user to select
one {inaudible} warning list or to enable everything depending on the different use case
so that's one of those {inaudible} pillar
knowledge base I mean a lot of
contributions coming from threat parties
are coming from that aspect so it's not really programmers
or coders that are contributing there
but it's more analysts and people doing really threat intelligence
or classification and so on
is really something that is useful for everyone
without being your direct contributions on the code
then over the times we we we became a kind of de facto standard and
uh nowadays is even more than a de facto standard, is a standard
We published as an interesting engineering task force draft
all those documents especially the core format
and to ease that for the development of external tools
integration and so on.
So if you're interested you can go to the
MISP platform website where we describe the different standards that we
published. We even co-host standards that are for people
integrating with MISP
and we have specific standards for example for the object template
and that's something that we will talkabout but that's something that was
really a need for us from the early beginning of MISP
a lot of organizations want to have their own structure of information
about objects and so on and we have a flexible model in MISP to
really create your own data models and this one is standardized too
and it's really helping sharing communities to
extend MISP as they wish and their models without breaking the
the standards itself so that's really interesting for for showing you new models
and then next to that we will do everything possible to help community
and Andras just mentioned the question of the
legal aspect and I think maybe some of
you already have this order to
seek legal team about the information sharing policies and so on
we try to make it easier so we publish this kind of compliance document
and so on it's part of the MISP project
everything is open source again so everything we do is open source and
on open access. You can reuse it and so on. We have for example a specific
document about building communities which is something that we do within the X-ISAC project
and it's containing kind of best practices what are kind of agreement that you can
use when doing a setup of sharing communities.
Up to things about how to do contextualization and so on
so that's that's maybe something that
for an organization that wants to boost up an ISAC or sharing communities they can
look at those documents and so on so it's again a thing that we try to help
for example we produce kind of OSINT feeds of existing reports and so on to
not only have software ready but to have some content and to show what kind of
information can be shared within different MISP communities.
So let us some get some of the naming conventions out of the way
before we start with the hands-on stuff and
just a quick explanation of the different uh data points and uh and naming conventions
that we use for them so it's a bit easier afterwards
this can be a bit overwhelming, don't worry we'll go through everything step by step also 1424 00:45:04,639 --> during the hands-on part
So basically all the data that goes into MISP, we separate into two main layers
one we call data layer which is really everything it has to do with
individual data points their compositioning and so on
So everything that we share in MISP in general in this regard
starts with something that we call an event, these are our general
envelopes for information so that means that
whenever we're describing an incident, we're describing a threat report
we're describing a watch list that we recurringly update
and they will all be grouped into something that we call an event
So the name is a little bit controversial at times, we try to pick a name
that is the least amount of a loaded term that we could find
but obviously even with that there it can be a bit confusing but just consider 1450 00:45:56,239 --> it as a generic container for data that has some contextual linking
then each of these events is populated with lists of attributes.
So attributes are the most basic data points in MISP
an attribute can describe for example an IP address
you can describe a file hash or it can describe a car plate number for example
It's basically just an individual data point with some basic context around it
such as describing in what context this was seen in
what type we're using to describe the attribute,
for example that we're using an MD5 hash to describe the hash of a file
would be one of those descriptions, hopefully not used as much these days
but that's just an example and then we can take these individual attributes
and composite them into what we call objects that are describing multifaceted concepts.
For example, a file object would be described by a list of attributes
including a file name, different file hashes, maybe file entropy and so on and so forth.
Each of these individual objects and attributes can then be further interlinked by what we call references.
So that means that most of the time when we're describing data in MISP
we're trying to tell a story so we're thinking graphs instead of individual data points
that means that we can for example, describe the entire flow of an attack
from the initial attack vector all the way to the exploitation
using the interconnected graphs using these references so we could say
initially it all started with an email that was received
that contained for example a malicious sample which then had to send this effect
in our infrastructure so all of these different steps can be then described
via different references there then to aggregate this information
into and aggregate the sightings of this information via structure
that basically captures sightings from our different information sources
that means if you have an IDS that is generating alerts
you can feed information back on when individual attributes were seen
in your network, in your premises or at your partners and so on
so this is basically it for the data layer, these are our main building blocks for that.
now in order to contextualize this information, we have different tools at our disposal.
The most simple one is what we call tags these are basic text labels that
we attach on individual data points or entire events
and these can either be created freely or most commonly they come from what we call taxonomies.
They're basically standardized vocabularies and that are either shared
by us so the MISP project at large or by individual communities to their members so
these vocabularies can include anything from for example something is simple and
and commonly used as TLP to national classifications
to various different sectoral classifications and so on
now if you wanted to provide more high-level information instead of just simple
labels for the information we can use what we call galaxy clusters
so galaxy cluster is basically a knowledge based element that we use as a label
these can be either coming from standard libraries
such as the ones that we maintain or you can create them ad-hoc in MISP.
That means if you're describing a threat actor
you could create create a threat actor galaxy cluster that describes the various metadata
about the threat actor and then use this to label your data whenever you think
that whatever you're describing is associated with a threat actor
you can also create for example a galaxy cluster describing the different
target sectors and then interlink using cluster relationships
the threat actor galaxy clusters with target sectors with exploited TTP and so on and so forth
So these are the high level structures that you can put on top of your data
basically to further contextualize it. Alex
Yeah, so just to summarize it and that's always a lot of people are asking about it
how do you summarize it about, for example in easy way you have to see really
MISP {inaudible environment/development} as an envelope and then you
have information inside and then what Andras describe is basically
different component that you have within that envelope and then you have
contextual layers on that envelope and relationship that are basically based on on that.
So, another thing that is very often and I think it is good to explain it, is about the
terminology between indicators, attributes, and so on that is
a different especially indicator of compromise and so on
In MISP, an attribute is close to an indicator
and we have this kind of flexible models where
maybe some of you are familiar with observables in MISP
we call it attributes and those observables are basically depending on the type
So, we have a specific flag in attributes which is basically defining
if information can be used automatically for detection
and that's I think one of the most important aspects when we talk about attribute in MISP
an attribute can become an observable or become an indicator of compromise
depending on the simple flag and this is quite important because
a lot of analysis and so on will depend on that and especially all you will use that afterwards
if you plan for example to use the data into a protective systems and so on
the IDS flags need to be set so the thing is if I take an example
you reverse the malware and this malware is connected to google.com for testing the connectivity
obviously you will have an attribute for example www.google.com
and this one is an interesting indicator for information for the analyst
so like that you can for example maybe cluster those kind of malware together as in this kind of behavior
nevertheless you are not really interested in that information as an indicator of compromise
because it will generate a huge amount of false positive
but if for example at some point you have an IP address that is really dedicated to that malware
then you will set the IDS flag, so the thing is when you define in MISP these flags
and we will show you later on it's very important because it will define what you can do
with information later on if you're going to automate and so on like
In MISP, what we try to do too instead of having just indicators
it's very common and I think many of you know about it you might see for example
a list of hashes so like for example MD5 hashes without any context
and sometimes it's difficult to know exactly what we are talking about
Are we talking about MD5 of malicious sample, are we talking about md5 of legitimate software,
are we talking about the MD5 value of the X.509 certificate,
are we talking about an MD5 as a mutex in memory
we have plenty of way of seeing those kind of MD5 so we try in MISP to have what we call the
kind of I would not say {inaudible keep shine/kill shine} but at least contextualization a category that
help to see in which context this has been seen
and as for example if 1 MD5 might have a payload delivery, telling that in which scope this has been set
So that means in MISP we have always and complementary type
so for example for an MD5 files you can say that this one is from a file or is an md5 of a fingerprint thing
So that means, always in MISP try to have as an indicator all those three information together
so it is giving at least more context and if you cannot set this context MISP will try to automatically set it.
So attributes are equal to indicators but with a bit more of information which is useful for you
At least being in a way to understand what is in a position to understand what you have in front of you
when you have to treat those attributes.
So this is just a brief view of what this looks like.
We're going to see this more in practice basically the idea is that all the data that we have in MISP
if it's well defined allows us to draw a graph out of the data and allows us to tell a story more easily
So here we see a simple example that basically shows the bank account that is associated with the threat actor
With all the various different data points with it and then we can basically relate these
different data points to each other and give the relationship a term as well
So in this case we see from the chart immediately there that that person is the owner of that
bank account with all those different data points for us as humans it's it's generally much more
easily understood if we look at a graph like that and tell the story that way
then if we look at a tabularized view of the data.
So one of the goals and something that we hope that we get out of
going through trainings like these is to really convert also the participants
to to see the value of producing data in that way instead of just sharing raw indicator lists for example.
And that's again what we think that's really important is the contextualization again.
So I mentioned we have the galaxies in MISP and we have plenty of representation
threat actors and so on and obviously one that is quite important is the MITRE Attack one
so MITRE Attack is {inaudible stored/performed} as a galaxy
and we have this flexible {inaudible mosaic/table} in MISP that you can represent those kind of
matrix-like model which is a case for Attack which is a very convenient way of representing the
different techniques in a progressive way used by the attackers and that's exactly what we can do in MISP.
So you have this kind of model and we have different model formats so again we have an advanced
i would say integration with Attack but you can extend it with multiple different kinds of galaxies which are
similar to Attack or complementary for example we have the Industrial Control System of Attack,
it's a separated galaxy, you can even create a custom one directly in the system
and then you can filter out your data and so on and that's exactly the thing why we are I would say
in bracket pushing people to do more contextualization, it would be useful forthem at the end
because this kind of information is really showing you for example your gap in your defense
your specific things that the techniques that are not used by an attacker
you might ask why, maybe because you are missing a specific detection point that you cannot
detect this kind of attacks or things like that so it's really actively using the data
to show something meaningful with it and I think Attack is one of the way
but if you can combine this with additional information like site links,
contextualization of the relationship between different objects and so on
basically everything in hand to improve your posture and secure it.
Yeah perhaps something to add to this as well,
some of the additional advantages is the moment that you encode all this information along
with the data you can start asking those those questions from your tool basically
for example show me what sort of threats my constituency is facing over the past year
and overlayed over how what sort of threats it was facing a year ago
what are the trends that have evolved since then
the other thing that it really helps with is it also gives you a high level overview of individual reports
that means if I'm looking at an event in MISP and it has 800 different attributes described in there
making any sense out of that quickly is very difficult, but getting a high level overview using MITRE Attack
where you say oh okay this has to deal with spearphishing, it has to deal with information exfiltration,
so these immediately tell me the story of what i'm dealing with without having to dig deeper into the data itself
so it is incredibly useful for an analyst that is trying to make sense of the data that you're sharing.
Also, as for the sharing itself I mean one of the main goals with MISP is obviously to share information,
We haven't really talked about the sharing mechanisms yet, we basically have a bunch of different functionalities in MISP
that we're going to see over the next two days the deal with distributing the information.
One of the most obvious ones to tackle is basically who is to be the recipient of information that we're sharing
so basically MISP, we can basically set the distribution settings for each individual data point individually
or for entire collections of data in one shot so that means if we create an event we can decide who we share the event with
but we can further restrict individual attributes or objects further.
Now, who we share the information with gets decided on using one of two different means,
one of them is a simple system where we tell MISP you are allowed to distribute it to everyone
that has access to this community, for example.
Or to everyone that is directly connected to my community but you can also define more strict distribution lists
what we call sharing groups where you individually name the organizations that are to be the recipients.
Now on top of that, one of the things that we often struggle with is especially if you're in some of those communities
or you're taking part or assisting some of the communities where sharing any information might lead to reputation or financial loss.
For example in the financial sector we have these worries very often
where if a financial organization were to share any information out
it could be misconstrued as a successful attack against them.
So instead, they choose to basically even if it was something completely benign
that they caught in their sandboxes, in their honeypots, whatever
and they decide not to share it out of fear of incurring this reputation loss.
So one of the things that we have in MISP is this system called Delegation
where you can, for example appoint your ISAC,
your central authority for a community, a national CSIRT, whatever
with the responsibility of taking over the data that you produce
and to share it out in their name
so that way, it's basically a semi anonymized information sharing
where you are completely removed from the data that is shared out
so the only two parties that will know who the originator of the data is you
and whoever is taking over the data and taking over responsibility for the data.
On top of that, one of the things that we wanted to achieve with MISP
was basically to build a collaboration with our different partners
so it means that whenever we're sharing information we don't want it to
be a one-way communication, so we don't want to have
this whole feed, provider and consumers relationship
but we want everyone to be able to chip in with their ideas
so while anything that you produce in MISP will only be tied and editable
by your organization, others can make proposals or counter analysis to it.
So proposals are a system where you can basically flag information
as incorrect and provide feedback on how to improve it
or how you can add your own perspective to an event,
so if you receive an event from a third party you can say
oh I can improve it and {inaudible listen/discern} this way
please incorporate these changes in the event
and then the original producer can make the decision
whether to incorporate it or discard your changes.
As for counter analysis, this is what we call extend events.
You can basically create an event that latches onto an original
shared by a third party and provide your own perspective of it.
and then you keep full control of the data and you become the owner
of whatever the extension is that you produce to the original event.
This happens very often for us, for example when a vendor shares out a report.
For example, we get a report from say kaspersky and we have additional information
or we have a different opinion on something,
then we might create an extended event that we share out to our constituency
which if they have access to the original report will latch onto it
and it will show our perspective on top of the original.
Now as for the exchange itself, every organization is free to host their own
MISP instance and then they can decide who they want to interconnect with
if both parties agree, a synchronization link is established
between the two MISP instance and sharing can start flowing between them.
Now this sharing is still governed by those distribution lists
and by some other mechanism that we'll talk about more tomorrow
but basically MISP exchanges information between the individual nodes
in kind of a mesh network way.
We also have feed system that allows us to generate feeds
and to share those feeds with larger communities.
So we as CIRCL we provide another SIEM feed, for example
that we make freely available in our infrastructure
anyone can just point their MISP to it and adjust the data
and keep it updated using the feed system, this is also great
if you have ever have the need of sharing information between air gap systems.
You can just generate a feed based on certain filter rules and basically
share it through say a flash drive or something like that, with an internal system
Now all of these filtering options are basically user defined
and they rely heavily also on the contextualization
so very often what we're doing and especially
if you were ever signing up for the COVID instance that I mentioned before
is you can also make those decisions based on the context,
what data you're interested in, what date you're interested in sharing out.
For example, if you connect to COVID instance, we categorize all of the
information into three categories, health related information
so basically information about the spread of the pandemic,
information about misinformation targeting COVID,
and also cyber security threats that are targeting, that are now basically
abusing the whole COVID situation with remote work and so on.
so if you're only interested in one or two of these three different topics,
then you can set up your filters to only ingest data coming from a subset of the data set
Very often what we do as well is we have these internal MISP clusters
in our own organization as well, where we collect information from different sources
so we have a dedicated MISP instance where we purely collect spam information for example
So for a constituency, anyone can forward their spam to us
and we'll just generate events out of those in that MISP.
Generally this information
is really not interesting for a day-to-day detection use, for example.
But what we do is we cache this information and we can cross-correlate
this information with our operational instance
that means if we start the analysis process and we start an investigation
then we immediately see the moment we start encoding data points
oh this is something that was already flagged once in our spam instance this information that that instance knows about, then we can pivot over to that instance and fetch the information
related to that same data point that we're also seeing in our current incident
and perhaps get more information that is relevant for us from there.
So basically very often we have these multi MISP internal enclaves
that help us basically to separate different concerns and
different collection mechanisms into their own instances.
Just in that scope and I think is linked to the question that we had
regarding the multi MISP internal enclave
someone was asking about synchronizing with an existing MISP and so.
You have this kind of local enclave options, where you can synchronize to MISP
like they behave in the same organization, that's one of the interesting options.
By the way, I would just give the mic to Josh that will explain a bit more about
the question and answer in zoom, to have directly the ability to answer the question answering the Zoom {inaudible}
I just want to jump in real quick, yeah if you have any questions
please direct them at the Q&A board versus the chat room
that way we can kind of keep a monitor of that
and other people can actually see the questions and the answer directly in that area
so if you have questions feel free to use the Q&A board and that's all
thank you josh that's very useful so we can keep track of them and we can answer live or
directly in the chat.
Okay great, so you see that the sharing aspect of MISP is like pretty extensive
and you have different models of of usage of MISP
some people have this pre-conception about MISP being like oh I need to share with MISP
no, it's depending on what you want to do with your MISP instance
and the core functionality of MISP is really to give, I would say the freedom
to each of the organizations to decide what to do with the data, if they want to share or not
and we always design MISP that everyone can be kind of consumers
so that basically getting data from different fields or producer or contributors
Andras mentioned a different way of contributing like sightings, making proposals, things like that
but it's up to the original contributors to decide if they want to share
that's really the thing, with MISP you can set up a MISP for like just pulling data, getting the data and that's it
and if at one point in time you want to like push some data you can just enable it and that's it
so it's really, it's just a matter of just tuning the configuration ,
the filtering really on the synchronization, if you want to share.
So you don't have to change anything in your MISP instance
it's just a matter of of what you decide and what you need to share
and then the thing that is really important in MISP
everything can be in flex. I mean even for example that we were mentioning
so those kind of envelope information might change over time
we have seen for example some past or incident report that has been updated like
two years later because they discover who was the target or the threat actor behind
and that's the thing that's really in MISP that we really want to be flexible
you can really expand the information either internally, add some comment and so on
and to share this information in your different MISP instances and share with partners,
your teams and so on so.
Really, MISP the core functionality of MISP is distributing information
but if you don't want to use it it's fine you just don't enable synchronization
but if you want to use partially, part of synchronization and so on
you just set up this kind of parameters.
So on top of collecting all this information
and synchronizing the information that we talked about before
we basically do a bunch of different stuff to improve
to handle the quality management of the information as well.
So one of the first things we do, this is something we mentioned a bit before
is we correlate information so we're interested in data that we've already seen before
we also have the feedback loop that we mentioned before with sightings
that means we really want to get timeliness to the information as well
so that we can but make better decisions on what we keep in our working data set for detection,
for blocking and so on.
The false positive management is a huge part so the warning list system
where we basically exclude those typical false positives
plays a very important role in the legal equation and this is also a community driven effort
so if you want to get involved with that and build and include your own infrastructure
for example in the warning list and so on,
either do it internally for your MISP or just share it with the open source community as well
so let us know if you want to have that included as well
we haven't talked about enrichment systems yet
but basically one of the things that we do in MISP is
we have connectors to all those different services that you might already be subscribed to
so if you have domain tools, passive total or what way or any of the other services
intel 471, and so on that you already subscribed to, then you can use
those services to enrich the data that you're working on
so if you have an incident and you're encoding information
you go out to all the services that you connect, that you have access to
and fetch the information on what else those systems know about the different data points that you're encoding
so that you basically get a jump start on your investigation.
Now one of the most important things that we have to deal with and this is probably
i think about 50% of the code base of MISP
is basically the APIs and the libraries that deal with integrating MISP with other tools
so everything that we can do by the UI, MISP is also exposed to the api
and one of the most important things for us is to make sure that
you can use MISP as simply a backend for another tool as opposed to just directly using MISP itself.
As for timeliness, we haven't really touched on that yet.
Besides the sighting aspect, you can also encode information about time ranges when something was seen
and you can build a full timeline of the events that occurred during a cyber incident for example.
So if you encode this information together with all your data points
then you get an additional graph out of it that tells you when what happens
and time-based correlations are really important as well.
So very often when you're seeing two things happening at the same time
they might be related with each other and
they might be worth digging into whether there is a link between those two things that happened.
So something else that we will touch on more tomorrow
is we have a full indicator lifecycle management system in MISP.
That means you can define your own rules and tune your own rules
on how you're going to be scoring and decaying indicators
based on all the contextualization that you have,
based on the type of the data that you have,
based on source of the information that you have and so on and so forth.
So we're going to go into much more detail on that tomorrow. Alex.
Yeah, so I was just answering a question and then I will make it public in a minute
that's a question about the API and using MISP.
There are different ways to evaluate the quality of the information that you share in MISP.
One of those is obviously to look at statistics, There is a statistics version in MISP to see
for example, the kind of indicator shared by organization and so on.
In addition to that, there is for example
MISP dashboard which includes a kind of gamification of the platforms
and which is giving badges per organization depending on the kind of information that you share
and that's a nice way to to find out if you are reaching a certain level of capabilities when using MISP
where you basically have for example information like
are you using sightings, do you use objects and stuff like that.
For example, thing that you can really look at if you want to see
if the quality of information that you create in MISP
i would just following the standards what is following the best practices in the different organization
is to compare with the {inaudible} feed
there are some goods even in the OSINT feed and
for example, things that are really a good indicator is to see
are you just using indicators and using objects,
are those objects linked together by using the relationship to it,
are you using galaxies, are those galaxies at the event level, {inaudible} level,
do you have tags, labels on specific objects or specific attributes and so on
that's different parameters and I think the question from {inaudible} is pretty good.
and if you really want to dive into the KPI aspect of MISP and quality of information.
In addition to what Andras just said
there are some other things about the quality of information shared within the community
and there's some good examples in the MISP dashboard about the different badges
there is even a model for sharing such kind of information.
So another thing that is I think quite useful and
it was one of the core functionality of MISP was the correlation features.
This one is, it looks like obvious but it's not always obvious
I mean a lot of tools in the security field exist but they don't do automatic correlations.
For example, at the {inaudible} we are using ticketing system
and sometimes it's very difficult to find if we have two correlating events
and what we decided in MISP which covers the cost.
I mean the correlation engine is maybe one of the costly aspects of using MISP on a database level.
but it's really useful.
For example, here we just have an example of
information about some malware spam that are used to share uh information about
target campaigns for the financial malware
and what we can see there is basically correlation on similar points
and those ones are mainly IP addresses of the infrastructure
but you can really spot interesting things there.
For example, you see that the third {inaudible bone/bin} in Germany share indicators,
you have a polish bank sharing the same kind of indicators
we were sharing such kind of indicators too
and even if they have different names or different contextualization
we can really spot similar infrastructure
so we can see okay, it's maybe the same actors using an infrastructure for different kind of things
or for example we can actually see here that we have different names of the similar malware
so really this is important for example another thing that is interesting is
for example if you have a sinkhole IP address setup by a antivirus company for example,
you can directly spot it. I mean if you have, I don't know, APT 29
and you have like three different criminals malware going on that one and so on
obviously it's usually not the same infrastructures but on the other hand
you can directly spot, okay this one is already take down
it's handled by this antivirus company and you can really handle it
so it's really a way to quickly find if it's a new threat or something that is already known in the infrastructure
So a little bit of the sightings themselves so we're going to see this more in practice
basically we have a very simple interfacing list that allows us to to tell the community
that we've seen an indicator, as well as when we've seen it
and perhaps also include information on what tool we've picked up, what context we've seen,
so sightings can have some metadata on top of just being a sighting.
We can also flag something that we call negative sightings which is a false positive sighting
where we indicate that we've seen it but it produced issues for us so it was a false positive.
We can also indicate that something is potentially going to be expired at a certain date,
so this is interesting, for example, if we're in talks with a provider
and we know that there is going to be a takedown
then we can already indicate that okay, this is no longer an indicator after a certain point in time.
Apart from that if you are ever dealing with bulk sightings,
so if you want to for example just capture any IP address seen in your network or something like that
there is another tool called SightingDB, which is developed by Devo, it's also an open source tool.
It's really recommended to use that, it allows you to to capture massive massive amounts of data
so if you're capturing the entire network flow of your constituency and or your organization
and just dumping all the data somewhere this is a great place to do it
and it's a very fast lookup database that is integrated with MISP
where MISP can automatically just query it for any of the indicators that you're seeing in MISP,
and whether it was seen in your network, and in which time range it was seen in.
interesting thing with that is it's also for historical values
so if you're just doing bulk collection of all the observables in your network
and then even half a year later if it turns out that a indicator is being shared with you
that correlates with an observable that SightingDB from half a year ago
then you know that you might need to launch an investigation into something
that happened half a year ago in logs and so on based on the historic look up.
Alex
Just complementary notes regarding sightings and
that's something that is basically maybe the easiest way of sharing additional information
it costs nothing and if you are connected to a MISP instance
and you can tell someone else that you have seen it
it's really a quick thing that can be useful for many organizations.
So the sighting itself sounds like a very small feature
but at the end, it's a an easy one for contributing and helping the others to know
if an indicator is still valuable and so on, so sighting is really something that
can basically be a kind of of entry-level things to do when sharing information
Something else that we have in MISP, this one is I think becoming more and more important
and we will do a quick demo later regarding that.
It's a timeline, I mean when we do analysis and so on,
it's really I would say common to have a first seen, last seen
to see the evolution of things over time.
In the example that you see on the screen there
It's based on specific threat actors
that sends a significant numbers of spear phishing
and those spear phishing are very well known when we collect those timestamps and so on.
So you can really see and trace the evolution of a specific group and so on.
This can be done automatically, for example passive dns records
I have very often the first seen, last seen and automatically you can really build
and create this kind of timeline because it can be really cumbersome if you have to do it manually
So we have a nice view like that so that means every time you set a first seen, last seen
on any attribute, object, and so on; it automatically populate on the timeline
and it's an easy way to to see evolution, trend and so on for your analysis
and this is a completely interactive so you can navigate over that.
We will show that later.
So for life cycle management, again this is something that we show briefly but
we're going to way more depth about that tomorrow
is basically here we see some examples of some attributes
that have scores applied to them coming from different scoring models.
So we see there an IDS simple decaying model and then a custom model
titled "Model 5" that are basically running on each of those indicators
and they generate a score taking into account for things such as labels that are attached to them,
the timestamp on when the attribute was created
as well as the timestamp of the different sightings that came in so generally
if something is still being actively seen in your network
that is still relevant despite the indicator itself being perhaps older
and then using the score that gets generated from these different models that you define
you can basically then make those decisions when you're exporting data to only include
data above a certain threshold when you're feeding your SIEM for example.
Yeah, and this one is quite interesting because you can really define
so the thing that is really important with the decaying of indicators
you are not modifying that actually you are really just updating and overlapping
and you can just define those kind of models so that means for example
even within a team where you don't agree on a specific model, you can have both models.
It's very common, for example, to have models for intrusion detection systems
and specific models for I don't know, endpoint, {inaudible} or endpoint protection device
and in MISP you have even a models or kind of simulator
where you can simulate the different model that you want to apply
and to see what kind of lifetime you want to apply,
when it decay, when for example you have a specific threshold where
basically say okay you don't use anymore those kind of data and you can do the mapping with
specific taxonomies you can with the different types, attributes, and so on
directly in MISP and it is really a quick win.
So you are not bound, for example, we know that some TIPs for example
have a kind of system-wide decaying models in MISP it is not like that,
everyone has their models, we are sharing some models
and you can define what you want to use without really altering the data.
So that means this kind of of information there is just an overlay
and you actually keep your own data in the systems without having any modification.
And you can simulate that one.
So when it comes to starting out, one of the trickiest things obviously is
when you're starting out with MISP is if you're staring as an empty instance
then getting started and encoding information is really tough
because you don't know what is really expected from the communities out there, you don't know how.
It's a new tool, you don't really know how to get started.
So in order to ease this a little bit there are a bunch of different feeds some of those that we also provide ourselves
which is obviously operational information something that you can use directly
so these are OSINT feeds that we produce as well from our TLP white data set
and the idea is that this will really help with bootstrapping your processes.
Look at the data we consider that generally well-formed
and well contextualized. It should give you an idea of what data generally looks like in MISP.
So don't start out with a fresh instance,
just go to your feed menu in your MISP when you're installing it
and pull in some of these OSINT feeds so that you see the information already.
Also it's a great way to test your internal tooling
so if you want to test the APIs, if you want to test internal synchronization,
it's good to have larger data set already from the get go
so that you already see that the movement of the data is working as expected.
Yeah, the other thing that you can do and where we're going to talk about that quite a bit tomorrow
is basically figuring out which feeds are worth ingesting,
how the feeds compare to each other, running overlap analysis between them and so on
So this is something that this is quite a heavy topic for tomorrow.
You're muted alex.
Yeah just discover {inaudible this/MISP}.
So as you can see for MISP,
it's the development of MISP already done over the years,
based on the feedback of users and that's really one of the key elements for us.
We wanted a tool for us that works
and it's key and based on that we wanted something that works for others too
and I mean the tool is evolving over time so you see that we have plenty of functionalities
On those two days of workshop we'll try to cover a part of it,
we had already some good questions regarding how to customize this and so on.
We might give you some hints how to do it and and so on,
so we won't be able to cover everything in those two days
but you'll see that you can really update MISP based on your specific use cases and so on.
So MISP is there as a tool, what really usually matters and are the successful,
I would say, sharing communities depends on the practices or you do that and so on
and we really want at least at the end, even if it's a complex tool and so on
to be as easy as possible for covering different use case
and that's really the thing that we want to do,
is for example for a lot of things that we have in MISP
and someone just asked the questions about how can you customize MISP
a lot of things in MISP can be customized by just modifying some JSON files.
It's the case for MISP objects so if you want to create a new object you just update the json files,
if you want to, for example, create a new taxonomies or create a new galaxy
you just create those kind of json files.
You have other ways to update and change the behavior of MISP.
it's based for example on MISP modules
so if you want to change the behavior of the expansion and so on
you can just play with MISP modules and we will quickly show you some examples on these modules
but that's really simple I mean there's no {inaudible}
and that's the thing that you have to understand with MISP project is not just a small open source software somewhere
it's really a set of combination of tool, software, packages, open standards,
various best practices, shared knowledge base and obviously the community is using it.
So that's really the thing that we will have we love with FIRST for example.
it is to have this kind of community, learning together, sharing information,
and so that's that's really a key for us.
We have, I think more than 500 contributors on the MISP project with even more nowadays.
So if you want to become one of the contributors it's really straightforward I mean
if you have something, a problem that you want to solve,
for example on an object, you just do a pull request and
it will be in MISP immediately and you become a contributor in the project
so really for us, it's really key in MISP
is to have a kind of tool that is supporting the different use cases
Okay so before that, we do a break I will share you with you
some practical details on accessing the MISP training instance because there were some questions regarding that
and after the break, we will do the hands-on practical session
with an example, so we will with a real example, so you will create the full event
based on some information that you receive.
So first of all, I will give you some details about how to access the MISP instance.
S0, first of all we have a {inaudible acting/active} page
which I obviously share at some point in time and I will share it again.
Yes. So there's a page with some pages that you can even edit.
I will paste the link again in the chat for everyone.
That's the link, so we have a 50 account on the training instance.
Pick randomly one
it doesn't matter if you are multiple one using the same account but be careful
don't change the password because maybe some people will complain,
and then we have a "TrainingFIRST2021" password so super simple
not so secure but that's fine it's a training instance.
Just for the reference, for the one that doesn't want to use the training instance
sometimes for whatever reason you want to use your own instance.
We have different images, virtual box and VMware images for MISP.
So if you want to play with MISP locally and so on.
If you want to play with synchronization too, you can even connect those two
So and during the sessions we will connect to that instance so the instant is "iglocska.eu"
and when you connect, you will get access to the instance.
So you enter your training password so I will enter with my specific account
and we will use that instance for the hands-on that we will do just after the break.
So what I propose now is to do a 15 minute break and we start at 45, if it's fine for everyone
and we will start by with the practical sessions with a specific email
that we will share in the {inaudible} as a practical example
so thank you for the one that join us now and we will start again at 45
to do the hands-on session. Thank you very much.
Thank you.
Okay and shall we get started.
Sure, welcome back everyone.
Okay so now what we're going to be doing is we're going to look a little bit at MISP itself
so we've talked plenty about it but we haven't actually done anything with it yet.
So I really encourage everyone that has a MISP instance to also play along and to create your own events.
What we're going to be doing is we're going to go through a set of fictional little exercise.
Assume that you receive an email from, in this case, luxembourg {inaudible} telecom
the CSIRT of them describing an incident, of a very simplistic incident,
of what happened and what we're going to be trying to do now is to model this in MISP
and to explain how you can further improve it and contextualize this information
So before we start, once you're logged into the MISP instance,
such as the hosted training instance. This is what you're going to be seeing.
So it's a little bit squeezed on Alex's screen
but the idea is that you have a list of events that are listed on the main page.
So we're in the event index, this is our landing page when we load up MISP
and each of these individual lines represents an event so they're describing either an attack,
or perhaps a report, recurring distribution, or a certain type of of indicator lists and so on.
So what you're seeing here is, you have each of these events having an ID and some metadata around it
so these metadata can be either coming from this galaxy cluster system that we mentioned.
For example describing different attacker techniques,
different types of ransomwares in this case or attack patterns that are leveraged
and then if we scroll a bit further right so this is a bit lower resolution here that we see
but you should have it visible on the same page.
You see the information about what this event is trying to describe to us
it's simple to understand text-based representation.
Now this instance is used for trainings in general so it's going to be filled with a lot of junk
interspersed with real data that is coming from our TLP white feed.
So you're going to see some obviously weird events in there.
These are just there for testing, just players playing during an exercise and so on
but also some real events there.
So what we're going to be doing now is we're going to create our own event
based on that email that we received it's also on the hackmd page
so just have a look at the email itself and we need to start encoding that event.
so before we include anything in MISP, the first thing that we need to do
is we need to create a new event so this is where everything starts.
Way to do it is to just click on add event on the left side of the menu
and then you start with a very simplistic form
where we can describe the event in a very high level in MISP.
so here you see the first step is very straightforward
the things that we have to watch out for out here is we have to decide who gets to see the event
so this is the distribution level
and that we need to set a basic description of it.
As for the distribution itself, you have different ways of interacting with the data here already
so one of the decisions that you have to make,
even if you're going to share to the wider community out there is
do I keep this internal until I am ready to share it with the community
or do I already make it visible to anyone that has access to the data in the community.
Now keep in mind that we have a publishing process in MISP, so until an event is published
it is not propagated out to other MISP instances,
that means anyone on the current MISP instance can see the data
but it will not jump to a different MISP instance at this point in any way.
but if you are creating it on a hosted instance for example
if your ISAC is running a MISP instance and you're creating it on that one directly
then this already has an impact on who can see the data.
So the option here, either go with "Your organization only"
and then raise the distribution level once it's ready to be released
or you already involve addressing the process and you pick something like "This community only"
where others can chip in with their ideas from the get-go.
So this is up to you, it's a risk versus efficiency question.
Do I want to share the information and potentially overshare a bit
by accidentally uploading information that is not yet confirmed that it can be shared out
versus losing out on perhaps others immediately jumping on board
and saying okay this is also something we've seen we've already done the analysis of it here you go,
so you have to balance those two things out.
For example some CSIRT was aware of this, so when people are working on a case
by default it is "Your organization only" {inaudible} and at one point in time the team lead
for example decide at some point it's okay, now you can share it to a wider community
and then you change the distribution level.
Yeah, indeed. So let's start with the organization only for now
for different reasons that we'll get back to later on
it allows us to show off another feature afterwards that is handy, so we start with that.
Then we have to describe the threat level so this is a very subjective question.
Threat level will depend a lot on what sort of an organization you are versus who you're sharing it with
so we all have different interpretations of what we consider a high threat level.
We have some descriptions for each of these fields predefined.
If you click on the little information box,
it will tell you that high is sophisticated APT malware or zero day attack.
Please just freely disregard this because
nowadays a lot of information sharing happens in completely different domains,
so if a fraud team is sharing information about fraudster
their definition of high threat level would be very different from those in cyber security for example.
So generally it's just a subjective first measure
but a lot of organizations users use this to briefly filter out what they should tackle first
so still use it with care. If you don't want to use this field, picking undefined is fine too.
Analysis is the next field, describes how far along you've come with the analysis process.
So basically with this what you're telling the community is
I'm just starting out with the analysis these are my initial findings
versus for example saying that my analysis process is already complete
i'm not going to be digging more for now, I consider this complete,
if you have additional information then obviously start collaborating with us.
So just pick whichever is most appropriate for you. Let's just go with "Initial" for now.
and then comes the most important part of this form which is describing the event info,
so this is a brief description for analysts that are looking at the data
that best described the event that you're basically sharing
Now be brief here and be careful about including very domain or organization specific information.
One of the mistakes that people often make here is
for example typing a ticket number or ticket id in there
so if you have a ticketing system and you basically start your investigation from your ticketing system
sharing out something like what Alex has typed there is not very handy for anyone else
nobody will have a clue what you mean with that.
Another mistake that can happen here very often is
especially if you're starting out small and in turn initially you're only keeping the events for yourself
and then perhaps later on you decide that you want to maybe perhaps after all share it out to a community
then one of the things that can really hurt you at that point is if you've used different language
for example to describe the event info so we've seen this very often
instead of describing the things in English, we choose our own languages
Both Alex {inaudible} and myself is hungarian, we are pretty prone to doing this in general
and this is generally something that will hurt us in the long term
because once you share it out with a more international community
you either have to go through the effort of translating it
or basically make it illegible for the recipient
So stick to something simple, simple phrasing, be as concise as possible
but make sure that it's still understood what you mean.
Okay, once you are done. In this case we are doing a {inaudible} spearphishing email,
we know that it's targeting the telco sector in luxembourg and we know that we have a malware sample
so that's a pretty nice short explanation of what the event is about.
So once we click submit, we have our event created and we already see that that
our event suddenly has a lot of data that we didn't intentionally put in there yet.
So we see a bunch of tags that are applied to the event,
we see that the event already has information about
who created the information, who the local owners is information and so on.
So MISP basically takes a lot of {inaudible local/global} settings from the instance
and it fills in the event when it is created with these basic datasets.
A lot of these also involve the contextualization that we start out with
so it might seem a little bit pointless to immediately label something
that we have not even started working on yet
but also keep in mind that very often what we do internally in our organizations is
we have several MISP instances that already are domain specific
So, for example we have our spam collector instance, we have our our sandboxing in MISP instance and so on.
These already define the scope of the information that go into them
so we can already decide okay if we are on our spam collector MISP instance
anything that goes in there will be related to spam
so in this case we can remove these tags because we don't actually want to include those just yet.
Maybe we can keep that one because it's still a draft,
so that means we will do an evaluation of this spam email accuracy
and then, so we have some defined taxonomy in MISP on this instance we enabled
for example the workflow one, this one is maybe of interest from different organizations
is a generic one about workflow, what is the current state of things.
So don't forget, in the initial event when we created the event
we have information about the state and stuff like that
Now with this what we do is recommend to have taxonomies
and you can really set up whatever you like in the misp event
to define the current state of this event so we keep "Draft" for this case.
Yup indeed, so we keep it at this
and we scroll further down and we see that MISP warns us about a few things,
first of all, data is not published
and second of all if we scroll a bit further down
we see that MISP also tells us that there are no attributes in here.
So this is still an empty envelope that we are about to share
so MISP tells us, don't share this just yet, fill it up with data first.
So at this point, we can start populating the information
So if you look at the initial document that we use as a starting point,
we see in there that we have a lot of information in there described
we see for example that we are dealing with spearphishing,
we see that we have an email that was received at a certain point in time
and we also see that we have an attacker that pretends to be working at the CEO's daughter school
and sending the email from a spoofed email address.
So we can start by describing this information, by including this information.
So perhaps one of the things that we can take here is,
let's start with the most basic thing we're describing, an email, so let's start with an email object.
So we're going to add an object and we're going to select email
So here we see that this is coming from the templating system
where you can define different concepts with different fields
that have to be then populated using this object templating system.
So we have a bunch of information that we can fill out here
we see the spoofed address so we see a "From" address that we can encode
Okay. we also have a sample that I don't know if I've uploaded it anywhere. Alex?
If not just pick any file for now because I think that's something I forgot to do.
Yeah I don't know where the sample is. Yeah maybe we should add it.
Yeah just put putty.x or something if you have it
Oops
or we can do it as a separate object, we can just do this separately
Yeah, we can do a separate object.
Yeah, indeed, indeed. Okay so what we can already describe here is
we can still add the name of the attachment that we had in there
just to fast track it a bit
good
so we have a timestamp too which is interesting
um
so this one has been received at a specific date so it was the third of....
so the "First Seen" is basically something that you can already set up
so it was the third of February, we had a specific time if i'm not mistaken.
So in this one has been sent on, received on 15, 16,
we also see that that basically the attachment was spoofing the document
about the report about the CEO's daughter's progress in school.
So we can pick the file name for the attachment and that is under the attachment section in the object
Good.
I'm just clicking it. Yeah
It is called report.doc.exe, I mean maybe it's not in the text right now.
Ah okay, it might not be in the text, might be just in the original file.
Sorry about that. So yeah report.doc.exe
Ok yeah perfect.
And then we also know that it was received, that we have the received header ip 325 6
so we can include that as well that's also stated in the email. It's 137.221.106.104
and we even have the hostname ,if you want to include that, that was also included in the email/report
Perfect, so as you can see here we did not fill everything out
because we don't know everything based on the report but we knew some of the fields.
We also see that each of these objects basically have some requirements and we satisfy those in this case.
So if you scroll all the way to the top you will see that that this object had a requirement
any of those fields have to be filled, we've definitely met that so we can just click submit
and we can create our object in this case
so here we see MISP telling us if we create this object that's what it will look like.
So we have in this case created our object and now it is attached to the event
and suddenly stuff happened here
so we see that each of these attributes already start correlating with existing events.
Now we ran this little exercise before so it correlate with some of those previous events
but normally if this was a real case if you get a correlation
there is either something very similar that already happened before
or is it something that simply might be a coincidence
but it's still cause for investigation to check
is this something that might help me bootstrap my investigation or is it just noise that is not relevant.
Maybe a side note because we have often the questions
when you create such object in MISP, you see that can be cumbersome to create it manually
so don't forget that everything that we do right now can be done through the API
so you can use PyMISP, automatically do it and so on.
So what we show there, if you think on the API level it can be done automatically
so if you have tool that are extracting emails automatically from the PC mailbox, whatever
you can automatically do it in MISP.
We just show the complete process manually but you can have a mix for some event
maybe some might be created automatically and then update it manually and so on.
Something else that might be interesting here at this point is we've encoded this object
and we look at it and perhaps we might want to change the distribution settings
based on the different data points that we have in there
so most of these such as the malicious host that email is sent from
are technical information that we can share with the broader community
but perhaps the name of the school that our CEO's daughter attends
is something that we don't need to share with the entire community
so we could reduce the distribution of that individual attribute in this object
so that we keep that, for example only for our own organization and for our own internal records.
So one of the things you can do in this case is you can edit that individual attribute
so the from address in the object and you can set a distribution level to "Your organization only".
In this case, once we release the event to a broader audience
it will keep this individual attribute for an organization
and it will not share it out with with other constituencies
Okay, so some other stuff that happened at this point
we see that several of you are creating events so that's great.
The correlation count really went up all of the sudden so it's good to see.
Something else that happened at this point is the event itself got correlated to other events as well
So if you scroll up all the way, we see that the attributes that we've added
are also showing us what other events we're correlating in.
So this is a summary of all the individual attributes, correlations from the event
that means that if this object is correlating
or these attributes within the object are correlating to it with a certain event
and certain other objects are correlating with other events
then this would be a full summary of all the events that you're correlating with.
You can also draw a graph out of that.
If you click on the correlation graph you will see how the events are interlinked
and you can further explore this by selecting any of the nodes
and pressing x on that to further expand it with its own correlations.
Okay. Let's go back to the event
Yeah I don't think we have a lot of correlations there, for the other events they're all the same
Okay now going back to our little example we have now created four attributes all together
out of the object template but we could have done this differently as well
what we could have done is we could also have created those attributes individually
and added those to the event directly.
So one of the things that we can do now
is we can go back to our report and tackle the next thing that is described there
and let's do it slightly differently.
So we also see that basically the person that is impersonated is also described
so that is basically, in this case John Doe the teacher of the student
So let's just create a person object and describe that.
So what we can do now is instead of directly describing it as an object,
we can first add those different fields, at least a name as individual attributes.
So let's see how adding individual attributes work
so we click on the little plus icon above the attribute list
and we simply select category "Person"
and from "Person" we select "first-name", "first-name" is John.
and here we can already define, is this an indicator?
Do we want to set it for intrusion detection system flag?
No, definitely not, this in itself is not an indicator
In fact we want to also disable correlation on this as this is a pretty common name
that is definitely not something to...
We don't need a comment for now, we're going to convert it into an object anyway
So what we can do is we can also disable correlation on this we don't want to correlate on John.
Okay, doesn't matter.
We can do the same thing for the last name Doe and we can basically say that this is now "last-name".
Now we've added these two things in there now the problem with this is
if we just had attributes instead of objects is we don't really see that
"John" and "Doe" in this case are the first name and last name belong together.
So if I were to describe several people in the same event
you would have a list of first names and a list of last names with no connection between the two things
so it's better to use objects in general whenever you're describing multiple aspects of the same thing.
Obviously if you just have a list of file hashes that you got from a feed
and you just encode those and you don't have any other information with them
you might as well just create flat attributes out of them
because there is nothing else to describe from your perspective.
But even in that case it's arguable whether you don't want to start with an object
from the get go but what we can do in this case if we did start with this way
or if you receive information in this format or your tools parse the data out in this format is
you can select those two attributes by clicking the little check marks next
or little tick boxes next to them and then clicking on "Group selected Attributes into an Object"
and here MISP will propose, okay these are the different object templates
that satisfy the list of attributes that you've selected,
there's a person object that we can use so let's just pick that one for now.
So here we see if we were to combine these two things they would be merged into an object
that is fine with us we see first name will become the first name of the object,last name, the last name
so let's merge it,
Now we basically have a person object, now we also know that this person that we're dealing with here
is impersonating the teacher of the CEO's daughter
so the impersonated person is a teacher of the CEO's daughter
so we added the object and we also see that we can add just another text field.
Yeah, just text field works, where we can describe it.
i just want to first disable the correlation because different {inaudible}
Okay yeah sure.
Yeah that works and we just add a text, description of the identity of the person
we can just say teacher of the ceo's daughter.
Okay, now we're done. We have now added the additional attribute
and now we know what this object is actually about without having a description in there
but we still just have an email and a person described in here
but we don't know anything else, we don't know that this email is spoofing to be that person
so we should add a relationship between the two.
Mow for this we can switch over to the event graph view so that is a little bit further up.
This one allows us to create connected graphs out of our individual data points
so we see that we have two unreferenced objects, so we explode that node by pressing x.
and we can draw an edge between those two nodes by clicking edit and add reference
and drawing a line between the two from the email to the person.
When you do, that MISP will propose a list of relationship types between these two different nodes.
There is also a custom one there so if you don't want to select anything from the list that is fine too
but for now we can just use the "impersonates" relationship which already exists in the default library.
Just click on submit and now we have a relationship set between those two.
So we started telling our story by basically having a connected graph between these two points.
Now let's further look at our original email and see what else we can get out of the text from there.
We also see that the malicious file was contained in the email as well as an attachment
So let's upload an attachment now to MISP.
I hope you have put in the text there or something because I forgot to {inaudible} it.
{inaudible} Perfect .
So as an attachment and this is where things become a little bit tricky.
There's a quick question there on the chat i'll just quickly answer that then we can get back to this.
Where can I create reference?
If you go above the attribute list there is an event graph button if you click on that
you get the event graph and on the top left side you click on edit and then add reference
Like I can show it again now.
Yeah that's a bit better, yeah.
So you have this kind of gray bar there with event graph so you can basically collapse or expand it
and then there you can select one of those reference objects you press x
to expand all those reference objects then you can just select one object that you want to add
and then you can edit, add the references and then you can add specific references.
In this case it doesn't make sense to make a second reference but that's basically how you do it
then you select your relationship type and you can add your reference.
It's not the only way to do it, there's a, I would say current-based representation
where you can do it, we can even show it so you have to go
It's much more difficult to understand what happens.
Yeah, so so there the reference that you created through the event graph is represented here
so you see that this object has a reference so from email to impersonate
and here's the opposite relationship that even describes the "Referenced by"
and you have the "Referenced by" on this object
so another just mention is I think less, I would say {inaudible} and so on
but sometimes you just when you are in the object
you just want to see if you have any reference or a sign and you can quickly see that.
So let's add an attachment now and upload the sample that was included in the original email.
Sso we just click on add attachment we select the file that you want to upload.
yeah so for the attachment in MISP you have really two models,
you have the model that an attachment is basically something
completely benign, safe and you can basically share it directly.
So for example you have attachment like reports and stuff like that.
In our case what we want to share here,
it's a malicious sample so and that's I will take a sample somewhere.
Press on one {inaudible} interesting there {inaudible} windows executables
and then you have to select if the sample is malicious
if you don't do anything, what it will be, it will be something
like saving report, a pdf report, something that's like supporting you in contextualization
could be a screenshot for example things like that.
But if you share a sample you have to select "Is a malware sample"
because like that MISP will encrypt and hash a file
so that means you have a zip file encrypted with a default password "infected"
but I got to avoid classical mistake of clicking on a link
executing binaries on your analysis machines and so on.
you don't want to do that, so if it's malicious always click malware samples
then you have one below "Advanced extraction", MISP can do a lot of things behind the scene
when you receive a file, in this case it's a window portable executable files
so we have particular advanced extraction for those files
and we can expand completely the files including resources, code segment, and stuff like that.
So I will upload the files.
Okay in this case this one was just like a very simple one.
So in this case, what do we have? We have an object
with the file name, the size-in-bytes and then the hash file,
so automatically MISP will do the hashing of the different files
the sample itself is attached so you can basically use it
and some additional ones like ssdeep for example, mimetype are automatically extracted
just maybe for the sake of it I will just take maybe another binary
just for showing you what could happen with other binaries.
Maybe that's for later for different events so we don't have the objects in this one
because it's easier to see for the graph
That's fine too.
You can show it afterwards
Yeah, okay so now we have this again this kind of object attached
and there's a relationship to create obviously.
Indeed. So in this case the relationship is to the email itself so we know that the email contained
this file so what we can do is we can just create relationship between the email and the file
and see that email contain that file
So you see, it it's again the same model.
contains
There we go.
So now what we can do is if you look further in the email
we see that there is a bunch of other stuff still described
so what we can do is we can just, now for exercise sake, just take
at least the next few lines or the next paragraph
and drop the entire paragraph into something called the free text importer.
What this will do is it will try to parse this text blob
and it will try to extract anything that looks like an indicator out of that
So this is another method of basically entering attribute into MISP.
So "Freetext Import", we just paste it in there and we just hit "Submit".
So MISP will tell us in this case it didn't extract everything actually,
so we need to still go back to it and refined a bit more
but it extracted some of those things that were in there already.
So that's fine, we can just already add those to the event
so how does it work behind the scenes is we have a bunch of regex in MISP
automatically extracting information from natural text, it's one way to do it.
There's another tool for doing it which is part of the event report
but it's usually a quick way to automatically extract information
and to see if it's already known for example.
So what we see here already is that evil provider was basically,
according to the email text, the place that was used to download the secondary payload from
so we can take evil provider and we also know that we got an IPv6 address to it.
So we're going to add that to it as well and convert this into an object again.
So we're going to to just select that one convert to object
and the object that we're going to convert it to is going to be a "URL" object.
Yep. all the way down there. Perfect.
Let's just do the conversion and then we edit the object afterwards
and we add the additional information that we have about it.
So we have an IPv6 that it resolves to. We also have a port.
So once we're done with that IP destination, perfect.
We can also add the port. It was communicating on port 443.
and again everything i'm currently doing there can be done through the API obviously.
Yeah and and finally we also have a domain evilprovider.com
now let's deal with referencing this to the other objects later on.
We can still still add the additional information that we have in there
and then we do the linking afterwards.
Again, we have the same problem here on this
because for example you see that the comment has the port
so that means we can just convert it as an object again.
Yeah and the IP belongs to that one as well by the way
Okay great, it's even better.
Yeah exactly.
Just my screen that is a bit small. Okay.
So in this case, it's again a url.
And the things that we have this time the port is actually a high port.
So while in the other one we do not correlate on on the port because port 443 is common
this is one of those ports that we might want to correlate on already.
So we don't want to disable correlation for this one.
Where as for the other one we we should disable the correlation for the other port 443.
Okay. Now the other thing that we have at this point is we have a secondary sample
so if you have a second one that you can upload now.
Yeah I just just add the domain so I get {inaudible}. Okay.
so what do you want {inaudible}
So we still have another file to update and we have a CVE that was also mentioned in the email.
Okay. CVE is an interesting one, we have single attributes for CVE
but sometimes you want to have some more information
so what you could do there is to create a simple attribute
so the CVE is "Payload delivery" in this case.
We have "Type" which is "Vulnerability" and usually a "Vulnerability" is defined by CVE
you can use other value but the best practice is the obviously to use CVE.
It's a very old CVE, those kind of attackers are always reusing those kind of old things
but you know, it works. You know people never patch their Windows.
This one is interesting because you know it was exploited
so I would add the IDS flag because it may be interesting to look into your system for additional ones.
So in this case, what do we have?
We have again, a single attribute, which is not the nice thing that you want to have
is basically you want to have as much context as you want for those kind of investigation.
Luckily on this instance, we have one of those expansion modules
and why the "CVE Advanced" is empty, that is quite interesting.
Ok, great. So and then you have some additional information
in this case we have some description, so what I can do in this in this one is
so you see that we have either the overlay thing
so in MISP modules, someone was asking about extension of MISP.
This is one way. You have this overlay things where you can basically just do expansions
and see okay contextual information
but sometimes you just want to have a bit more than just contextual information.
You want to have the associated object there
so there you have this kind of explosion there and you can add the enrichment
I'll give a try on that one
Okay great. So there's something wrong on this machine
That's great. I'll take the other one but this is not an object but that's fine we can just like...
Yeah, we got some attribute in this case so we have basically the description
that coming from the enrichment and what we can do is to then make an object called "Vulnerability"
and "id", it's not a "credit" in this case is the "description"
and make an object of it. Usually you should have a full expansion there
but I didn't test it on the training instance maybe something is broken on that instance.
Okay so now what do we have is more contextual information.
We start with a story and we see that we have an email we have a first url,
a second one which is a download and a specific CVE.
So maybe now we can go back to the...
We still miss one thing which was a secondary file that was downloaded.
Oh okay. The secondary files, yes.
Yeah. So according to story what happens was the initial sample was when executed
was downloading a secondary sample and that one was basically
then used to exfiltrate data from from the system.
Yes, so this was a {inaudible} malicious file okay then I will add...
Yeah just another file and we just pretend it's the one that we were supposed to use.
Why is this one makes sense, it's an Emotet one {inaudible}, that makes sense.
so now we have all these different objects in our event
and it's time to build the story out of it as Alex has mentioned.
So it's time to go back to our event graph.
And basically so far the story is that we got an email.
The email was impersonating a person and we basically got a primary sample out of it.
That primary sample then reaches out to evilprovider.com to download a secondary sample.
So we have a relationship between the file which "downloads-from". "downloads-from" yeah.
Perfect.
from "evilprovider" and then "evilprovider" downloads the secondary sample
which is in this case "index.html.1" and this one then exfiltrates to another evilprovider url.
Now there's one thing we missed in the story here is that the first one try
so in this case "trilog.exe" was actually abusing the CVE that
that Alex has already expanded so we have an abuser's relationship from "trilog.exe" to vulnerability.
So and once we're done with this we already see the entire story in this graph.
So even if you have no idea about what happened in the report and you don't read the original report
by just looking at this graph you can clearly read it out in simple sentences.
We see email impersonator a person, John. Email contains "trilog.exe"
and that exploits the vulnerability. Downloads from evilprovider.com, "index.html.1".
which exfiltrates to a url.
So it's a very simple story to comprehend without us knowing the original data information
and without us having even having to look at the individual indicators further below.
So this is when we're talking about information sharing we're basically sharing on two layers.
One of the layers is sharing with machines so informing an IDS about things to alert on
and at the same time we're sharing with analysts
that want to really understand what the threat actor was doing in this case
and what happened during the incident.
However at this stage, we have described our event but we're still missing something at this point.
We still haven't actually contextualized the information with everything else that we know about it.
So we have vocabularies at our disposal, we have the ATTACK matrix at our disposal
So let's start going through the individual attributes
and let's start to attach those different labels to the data
so first of all if we look at perhaps, which one should we start with?
Let's not do everything. Let's look at the original email for example.
We know that the original email deals with phishing.
Now ATTACK has a pattern that describes phishing
so we can just attach the galaxy cluster of attack to the attributes in there.
So we use cluster yeah.
and we can just use ATTACK. mitre-attack, perfect.
and we can click on "Attack Pattern" then we get the attack matrix
and here we can select "Phishing". It should be in...
Yeah there it is perfect.
So we attach it. We refresh and there we see it is now attached to the attribute
and if we generate a heat map now out of the events if we scroll up.
We have an attack matrix view next to the event graph.
If we click on that one now we now see that as a first overview already we know
without looking at any of the details we see that we're dealing with phishing here.
So this is one of the attack patterns that we've described.
let's see what other attack patterns from attack we can describe
You also see that there is automated exfiltration happening.
So if we go to the secondary url, so another evilprovider.com.
We can attach the pattern there as well.
now we can choose to do a single attribute what we're doing or we can just select all 4
and attach the cluster to all 4 for let's just do one for now, it's enough.
Watch out it's... Yeah, perfect. And just pick "Automated Exfiltration".
It's the first one on the... Yeah.
Okay so now we've attached some attack patterns, we could attach it to the sample as well.
What the sample is doing but we're not going to go through all that effort
Let's look at some type of contextualization for example...
maybe this then it's a matter of taste again regarding
at which level you want to attach the galaxy
there is really the topic is a matter of phishing at a global level, usually we can add a galaxy there
and then for example add MITRE ATTACK directly there
and select the pattern phishing then the techniques there directly.
so you have different options usually we recommend to make it at attribute level
but in some case you don't even know which attribute level it applies then you select the event level.
Exactly. So that's indeed a good point if you know that the entire chain of what you're describing
references to a single contextualization, be it label, be it a galaxy cluster then indeed
what we assume is anything that you label on the event level is inherited by all
data contained within unless explicitly overwritten by the opposite tag basically.
So indeed that's the case. In this case we're kind of in a weird situation
because we're describing the full chain of the attack which includes initial phishing attempt
but also includes the secondary payload and the exfiltration and so on
and if we do this on the attribute level as oppose to the event level
then you're really only describing which part deals with the phishing
which part deals with the actual exfiltration and so on.
So this is really up to you. What we generally recommend is don't just do it on the event level.
So if you're describing more concepts in a single event
make sure that you contextualize individual parts of it
because one of one of the things that we use these labels for as well is Searches.
so if I were to search for all indicators that relate to phishing
I might not want to get the secondary payloads artifacts included in that response.
because that was just the initial vector of getting into the network of the victim
whatever happens afterwards is not directly related to the phishing.
So keep that in mind as well. something else...
Yeah so just something that you have to keep in mind too it's about
which classification to choose or which contextualization source you want to use.
On this instance, we have already a lot of things enabled
and if for example you go for taxonomy. You have a lot of taxonomy that is describing phishing.
For example you have even a complete taxonomy about the kind of phishing you have and so on.
So when you install your MISP instance and you start to make it operational
you really have to decide what kind of taxonomy you want to use
in this case we have already a lot of things are available by default
so the phishing taxonomy itself is a complete one coming from I think {inaudible} academic paper
where we have all the techniques that are used.
So for example you can say that this one is coming from a spearphishing
which was described there and you have the different techniques.
So in this case it's email spoofing
and you can go deeper there into the description of what is exactly the phishing.
and you can mix match both. I mean Andras selected the ATTACK phishing techniques
at specific indicator level
Maybe another analyst would want to classify it and maybe the objectives might be different.
Maybe on one, for example it's more specific for tools.
But if you want to run out statistics at the end of, I don't know, quarterly meetings
and say okay, how many spearfishing that you receive or maybe emails spoofing.
For example if you can control better emails spoofing, the SPF record and so on
you can just look at the current technique that are used by by the attacker.
So you see that those kind of, it's full of taxonomies that can be used
and obviously we usually recommend to not enable everything
but just pick what you really want and some are very generic,
some are more advanced but that's maybe something that we dig into more afterwards
but just be careful of which kind of taxonomy you want to use because it will be the language
that you use with the community and your partners for sharing this information.
Maybe something interesting to look into the email and that's linked to classifications
but there's this comment there
"Please be mindful that this is an ongoing investigation
and we would like to avoid informing the attacker of the detection
and kindly ask you to to only use the contained information to protect your constituents".
So this is kind of your language describing to you what kind of classification it is.
And so now which one should we use so if we are FIRST members, if we are using the FIRST community
obviously the classification that we will use is not the NATO one,
or the Ministry of Defense in whatever country, it's really TLP.
So then again based on that, we will look into different taxonomy that we have
we can look for TLP and I should not do that like that.
I go for the TLP library and then I have the specific taxonomy TLP
and then you have the different one.
In this case they say you have to share it with your constituents only
so TLP amber seems to be the most appropriate one.
We say that the TLP Amber information is given to organization
sharing limited within organization to basically act upon.
If we have the extended classifications from FIRST it includes the constituent too.
So I will just use a TLP but I mentioned something else that is interesting.
In the email they mentioned that this is an ongoing investigation, to avoid informing the attacker.
In this case, how would you inform the attacker but if you do actions on specific indicators and attributes
you might want to restrict that so there is another classification,
I don't know if this one is enabled.
It's called PAP which is exactly that, it's similar to TLP
but describing what you can do with this information.
If we don't want to at least notify the attacker that we are doing some further investigations.
Maybe we want to restrict that and the PAP is really telling you
what are the permissive action that you can do.
In our case, "PAP: RED" for example, non-detectable actions only and that's really what we wants
because the reporter say okay we have an ongoing investigation
so you don't want the other parties are informed.
So in this case, I will use RED and again this is used at event level
and that's something quite important because MISP will take care of that.
You don't need to set "PAP: RED" on every single attribute.
Behind, it's really at event level so it's automatically allocating on all attributes
We don't show it on the interface because it will be too cramp.
You know, overloaded with information but we do it in a way that's on the API level.
If you do {inaudible} search. for example on event level or attribute level
"PAP: RED" will be included there if you have an attribute containing some information
automatically tags like "PAP: RED" will be then included into the information.
So that's something to keep in mind when we have information from third party
is to to wonder okay what is the classification scheme
so sometimes they don't say a specific classification to use that you just use natural language
or just a normal sentence to describe all the information should be shared.
So the interesting thing here is that what we've seen now is
we've contextualized information in many different aspects
and this is just scraping the top layer basically
we could go much much further with contextualization.
Imagine for example describing how this information is relevant to whether it's used.
What sort of mechanisms they should have in place to be able to block this information?
How can you make this useful? Think of different maturity organizations as well
when you're sharing information. You could also describe information about who's behind it,
what the motivations are? So we did not describe the threat actor
because we haven't done any analysis yet.
This is the initial information we got from a CSIRT that just reported an incident to us
but we could go further and if we did our analysis we would find who's behind this
we could go for information with the threat actor, we could look at target sectors
we could look at a lot of different information in regards to to further contextualizing the information.
So in this case we could also, for example say that's in the {inaudible} where
because we know that this is something that was targeting an organization in Luxembourg.
We know there is also a sector taxonomy that you can use
so that's not a galaxy but the taxonomy.
So we can also add for example information about the financial sector
We know the CEO is a CEO of a financial sector organization.
So we could also say that it's probably has to do with that as well.
Maybe it's not enabled, sorry about that.
Yeah this one never. But if you search... Yeah, exactly, there.
If you just search for "sector", it should be there.
Yeah but I'm... Yeah there's something that you can do talk about it later but it's...
"sector" so we have different one.
Maybe you find that so if... there was one for finance you can just pick that, yeah.
Yeah, something else you can do and this one is important too,
it's going a bit further than the email. So for example, as a source
we receive emails from various people, I mean if I receive an email from,
I don't know, from an analyst from, I don't know, he is at Mcafee and so on,
that I'm working with them for years my confidence on this information is quite high.
On the other hand, if I receive an email from someone unknown maybe my confidence will be a bit different.
So in MISP, you have plenty of taxonomies to express confidence.
For example, the one that is actively used, for example, the military network is Admiralty Scale or NATO scale
where you can basically define the credibility of the source.
In this case, we can say that we know the source and is usually reliable so that's the source itself.
and we can say for this specific information that is probably true because they send us some evidence.
Now if I have like three emails talking about the same case maybe my level of credibility will increase
because we have multiple people that have seen exactly the same kind of thing
So in this case I will have those kind of information there
Again it's a way to really contextualize information and the quality of the information
and you have, for example, additional one like, for example, we have one called "estimative-language".
so this one is more coming from DNI and the CIA.
It's like the likelihood of probability that this happen.
So we can say that this one has been almost certain and then we can even qualify
or own analysis judgment on this and I can say that it was like quickly done
and it's not perfect, I will just say low for example.
So then you can have this kind of information.
And you can either use it as an event level again or a specific attribute.
So for example, if one of the emails it was like, not properly collected or it was skirts,
or someone modified the headers and so on, maybe you can reduce
the summative language of the confidence level that you have
in the analytic judgment of the specific evidence or element by tagging that at the attribute level.
So again, those kind of information that we are putting there are factors and so on and more like event level.
But if you have really specific things that need to be changed
or that are specific to the attribute or object then you can change it at the attribute level.
Just some other thing on the user interface that might be useful too that we skipped.
On the metadata of the event you have plenty of information there.
Why that is interesting regarding "organization only" and "distribution",
In this case, we just distribute to the organization
but if you have pretty large event at some point in time and you want to distribute.
You have this kind of overview there which is helping you to see at which level you share this information
In this case, it's super easy. We just distribute it to the training organization, that's fine.
But if you have a pretty large instance with a lot of organization and so on
It will display you a full graph of where the information will flow and will be distributed.
Okay. Now going back to our event, basically the reason why we went so deeply into the contextualization part
is looking at this event we can already use this right away when feeding our tools,
when doing our searches, to basically search for anything targeting the financial sector for example.
we can search for anything related to phishing and find the data contained in this particular event
So this already helps us with our filtering mechanisms.
As for PAP and TLP, those tags we can use when we make decisions on which tools we feed the data to
or which partners we share the information within the case of TLP.
So we're going to see that more tomorrow when we're creating synchronization links with other instances.
We can for example set restrictions on TLP when we're pushing data to another node and we can say
okay, no matter what distribution setting don't send anything TLP Amber in this direction for example.
Yeah, as an example there's a very good open source tool called TheHive for threat hunting
and they use PAP to know which kind of actions they can do on the data.
So if you synchronize MISP with TheHive instance you can really be sure that
what you set as PAP for example "RED" on the MISP instance will not generate issues
when you are starting to expansion within Cortex on TheHive
to be sure that the information is not basically flowing somewhere else.
So at this point something that we didn't do so far is we did not include the initial email.
So what we're going to do now is we're going to use another functionality of MISP
that we haven't talked much about called the report, the event report.
We can also include clear text information such as a report description and so on together with the event.
So what we're going to do now is something very simple, we're not going to write our own report.
We have a report already available from the original source
so we're just going to paste that entire email in.
Okay. Just submit for now.
So now if you look at our email report we just have a simple report here in clear text.
We're going to see an example what you can do with this. So this is all in markdown
so you could go into edit mode and pretty it up, add additional information there.
We're not going to do that now because we're going to just look at an example that already has that
but before we do that let's get back to our event and let's assume that we're done this entire process.
We have our report, we have our event we have contextualized all our data
and let's publish it now to the community.
So when it comes to publishing we have different ways of achieving that in MISP.
By default, when we create an event like this, at this stage
we have all the data contained that we want to share out and that we want to use
however MISP considers this to be non-final, it is not to be used by automation tools connected to MISP,
it is not going to be synchronized out to other instances and so on.
And what we can do now is first of all we need to decide how we shared it out
it is the organization only for now so even if we were to publish it
would still only be pushed to our own tools that connect to our MISP
but it would not be made visible to other organizations but we want to change this, in this case.
However, let's assume that we are an organization that does not wish to reveal
that we were involved in this entire incident. We just want to entrust the third party with doing it.
So as you see there, where Alex is hovering. We basically have several options here.
We can either publish the event which means we initiate the entire exchange with other instances
if the distribution allows it. It will alert everyone that we have published this
or alternatively we can delegate the publishing to third party and stay anonymous ourselves
So let's do that option for now. So what we're doing now is
we're entrusting a third party to take over this event for us
so let's say that we would entrust, say for example CIRCL, to take over this event
and we tell CIRCL that we want to share this event, to be shared with the entire community.
so we've collected..
Yeah and you can see "This community only" for example or a sharing group whatever you prefer.
Okay, so this is again a suggestion to the other organization saying
okay, we want you to share this out and we want you to share this to this community.
Once we click "Yes", even though the event was your organizationa and only visible to us.
It now becomes visible to two organizations, ourselves and the other organization
that we entrust in this case CIRCL. So CIRCL would get an email saying
Okay, there's this delegation request someone wants you to take over their event
are you willing to take it over and publish it under your name.
This will look something like this with slightly different text we're cheating here now
since we're doing a training, we're site administrators so we see both sides of the story,
so we can either accept or discard this request. Keep in mind if you accept such a request,
the event becomes your event a copy of it is created under your name
and basically you are taking responsibility for the event from now on,
so also make sure that you're not pushing junk under your name.
So in this case let's just discard it
but we could have accepted it and then it would have become our event.
Okay. Let's go back to the event.
Okay, so now the other alternative is if you want to publish it under our name,
what you would need to do is you would need to raise the distribution level first
if you wanted to involve any other parties
So we need to edit the event in that case and raise the distribution level
to say "This Community" or "Connected Communities". Let's go with "Connected Communities".
connected communities means anyone that has access to my MISP instance
and all the directly interconnected instances including all their members as well.
So in the case, for example, of us publishing something like this in the FIRST instance
we as CIRCL have our instance connected to it
so all the members of the CIRCL instance will automatically also be included in the exchange.
Here we see a graph of that so we see the event would
be also visible to all the directly connected instances which we only have one of
which is a loopback connection to "iglocska.eu" so not that interesting
and to everyone that has access to this current instance
Okay. Once we're done we can click publish and then the event gets synchronized.
So what happens at this stage is first of all the event will jump over to directly connected instances.
MISP will send out a bunch of emails to everyone that subscribes to publish alerts
that there is a new event with all the data contained within.
it will push the event down various local channels to other tools using ZeroMQ, Kafka and so on and syslog.
So if you have any tools that are subscribed to these published feeds
and they will ingest the data and it will also make it available to the API
and to make it available to all the integration tools out there.
so if you have your SIEM connected to MISP, it will now be able to fetch the data contained in this event
So this is basically the publishing process however if at this point we noticed that
okay we've now shared this event out but we've actually made a typo in the title we wanted to include
I don't know, a trailing period at the end of the sentence {inaudible} something like that.
in the title and we edit the event. wWat happens now is, there is a modification to the event
so even though it was published, it becomes unpublished again and it needs to be to be republished.
now the reason why we do this is whenever there is a change
we need to synchronize it out to the other instances out there
and if you have a publishing process in place
where so only certain users have access to publishing rights for example
then anytime your organization is pushing out information
it can go through the irregular vetting process so any change will unset the publishing of the event
now in this case this is a very small change that we've made
so we don't want to actually send out emails to all the other users.
We don't want to spam them with data that is pretty irrelevant for them
so we can do the publishing again but this time using the "Publish (no email) option.
So it will also synchronize the data
it will again make it available to all different means of ingesting the data
but it will not spam our users with emails.
Okay so that's basically it for the publishing and perhaps one thing that is interesting
and that we didn't talk much about is
we have now raised the distribution level of this event to "Connected Communities".
So the event is synchronized out but we actually had an attribute if you look further down
that's had a different distribution level. So that one is actually going to be removed
from the synchronized button.
So we had one that the impersonated person's email address that was set to organization only.
So whenever we're talking about synchronization that thing will,
in this case, not synchronize out. So that will be redacted from the event.
Okay something else that we can do at this point once we have created our event is
we can also extract it in different formats, so if you click on "Download as..." on the left side.
You will see that we can basically convert this automatically to a bunch of different formats
and extract it in those formats directly. This is also what we would be accessing by the API
if you were to search for this event. We can also mark whatever response format we want.
just very briefly we won't go very deeply into this. These formats are coming partially
from our predefined hard-coded list of formats that we support in MISP
but some of these formats also come from the different export modules that we have
So if you want you can either build your own native modules for exporting and converting data
or you can build modules that are sitting in another tool called MISP modules,
side by side with MISP that will ingest the data and then convert it to other formats.
So here's a PDF report that was created directly out of the event
and that you can just share out directly for the event.
Something else that you can do is anything that we do in MISP, so all the process of adding attributes,
all the process of viewing data, you can also do that in a machine {inaudible} way
by just appending .json at the end of any of the url.
So in that case in this event we're going to get the json representation of the event
Okay.
So that's basically for creating an event.
Just maybe one thing that is interesting we have a very good question from Martin {inaudible}.
It's a quite complex one but maybe we can already partially answer it.
So when you create an event and in this case the creator of the training.
People can contribute on that one but if you have an ISAC and you want to distribute back the information and so on,
one of the options that you have is to try to create extended events for example out of it.
So you can out of an event, you can create a new one which would be for example with additional information
like validations, additional things that you want to add. So you have this kind of extend event
and you will create automatically a new event based on that.
A thing that is interesting there, the thing is you can really create something completely new out of it
and then see, so for example, for this case I can say that we did a kind of session with additional information.
There the distribution is "Your organization only"
and I would add for example a specific attribute which is for example "Targeting data"
and I can say "target-user", the son of the prime minister.
So it may be information that you really don't want to share with others.
So this one is basically a normal event with additional information there
and it's only shared within your organization.
Nevertheless, if you go to the original event, you have this kind of extended view there
and we can have what we call an extended view and not an atomic view
and the two information is combined and you can see there
that we have one with the information about the son of the prime minister
which is the extended event there. So just to answer the question of Martin
about the question about adding information on existing event, this is one way of doing it.
So using extended event is a way to qualify or extend event with additional information and so on.
It's actively used, for example, for when you have two different view of the information
because one is distributed and another one is like the private information
like the forensic evidence that you cannot share for example,
you can create this kind of thing. It's one way of doing it.
It's not answering completely the question of Martin but we can even go deeper later on that
but it's one way of... because tomorrow we talk about synchronization
there are some specific options for ISAC like unpublishing events if we do synchronization and so on
that can be used in some some cases for ISAC. There are many options but that's one way of
partially solving this kind of issues of not owning the data. It is to extend the information.
So I know you have something you want to add on Alexandre.
No, no, that makes sense.
Again for the collaboration on this one, we can do various things so
in the case of, you have a typo for example in the specifications and so on.
you can make proposal, another thing with on the interface here you see that
you can basically make either an edit or you see that you can make a proposed edit.
So what is the use case for that? It's not like for fundamental changes
but for I would say minor challenges on a specific report.
Imagine that you don't agree on this IP Address, there's a typo
and it's not D5 but E5 in the ipv6 address, so you propose the change.
in this case i'm playing both roles here but what do I have here,
it's basically an attribute with a proposal of the change and I'm playing the both roles
contributor roles and the original creator then I can say, okay I accept the change there indeed,
this proposal makes sense or basically discard and this is a way to get updates
from supportive other members, other organizations and so on.
It's one way to to update the information in this case I will discard it because it's not correct.
We were talking about contributions, this is another way of contributing is the sighting itself.
So for example for this indicator if for example we have an intrusion detection system
and we have seen it like three times in a row
we can add on the interface, with the API, through the user interface, and so on
that you have seen that multiple times and like that you can share this kind of details about the sharing aspect
So what we have seen that at this specific amount of times we have the three counts
saying this is a sighting and you have seen it and you can do it per organization
or it could be even anonymously you get different configuration in the model of sighting in MISP
but it's a way to see that an indicator has been seen or not.
If one specific {inaudible} for example a false positive you can see the negative one,
negative sightings which basically tell others that okay this one is generating a lot of false positives
sometimes not every organization agrees on the false positive
because they have different views coming from different networks and so on.
That's a way to provide feedback, so one is delegations proposals or another way is to basically get sighting.
Okay.
I'm just trying to go through the questions
yeah maybe there are some...
Yeah there are some that are repeating so perhaps it's good to call them out
So there was a bit of confusion about how to add the email object.
So it is a little bit confusing so when you're in an event and you click on add objects,
first you need to select the scope from which you choose from
So it's going to be climate file and so on.
Just click on all objects if you're unsure and then you can search for whatever.
So after you click on all objects and you type email, it's going to show email object.
Here the first step is more like finding out the category of an object
Yeah so some sometimes you just know the category but you don't know what is really available there for you
So you want to see okay what sort of objects can I use in network contacts
and I would click on network first
and then you get a list of of all tangently related objects
that will have to do with network connectivity but not necessarily describing the same concept at all
but if you don't know which domain you want to pick it from
or if you know exactly already what you want and you just want to search by name
Just click on all objects first and then you will find what you're looking for
by just typing it. So email is easy to find that way.
Okay so just type email and that's it.
Okay, other questions that were there. Okay perfect.
And there there were a few other questions that I answered in the ...
Meanwhile, maybe it's a good idea to read some of them out.
Yeah, indeed there was a good one about correlation graph and filtering on it.
Indeed, we don't have a way to filter the correlation graph
but it's something that we've discussed for a while already and we want to do it at one point
so that you can add some filter rules in there.
Yes, the only way to to do it here is through the api
so that means you done creating one and then you have to do a filtering {inaudible priority/proactively}
but it needs something there to be added
I don't know if you have an issue on that one.
Yeah I think we do. Yes, yes, I think we should be good.
So maybe you know what sometimes what we do is just to add the one on this one
If i can find it back so I guess you can see what kind of issue that we have and so on.
A lot of the issue that we have is more like
I mean around 25 percent, 30 percent are basically installation problem.
That's something that you can discuss maybe tomorrow about recommendation on the system
We don't need a lot of requirements but at least you need to have a LAMP system working
So MariaDB, linux systems and Redis running.
So obviously, for example, an Ubuntu distribution out of the box is working without any problems.
Now if you try to install a MISP on the MAC OS, you might run into problems obviously.
But what we recommend is we have automatic install script for ubuntu for example
and this one works works quite well.
I wanted to search the issue for...
You can just search the correlation graph, for correlation.
Yeah. correlation filtering.
Nah, that will be a bit too specific I think. No maybe not. Maybe yes.
We're filtering by correlation on {inaudible}.
Oh this one maybe. Yes. Yeah, yeah okay, this one, okay. So... One sec.
So that's how it works so if you see a component issue that
or a feature that is really interesting for you
don't hesitate to take an existing issue about specific requests and add some comments there.
Like for example, it's really the feature that you want, is it important for you, why and so on
and then we use that as a source of of doing a feature request
As an example, we do a release of MISP every three weeks usually
and there are many new features on each release
As an example we had a request like that someone just fixed two days ago about the events...
Oh we didn't even show it. Event timeline and then
we wanted to have something that is easy to set the number of days
and then he added the new feature. So sometimes it makes a lot of sense
so don't hesitate to create an issue and and propose a new feature.
Which remind me of showing you the event timeline because we didn't really show it.
So you see that on this one we have nearly everything on the same time
which is basically the time when we create the different object we just set the time for one so...
and then I can basically look at this one and this one is like the..
Yeah, I don't know why we don't have the expansion on that one.
so for example if you have a specific time we can expand it and even change it in the graph
so that means if we have for example this email,
a thing that we can do is expand it and change when this has been seen
and we can even change at which time this specific model has been seen.
But that's again a good point to do is to automatically create a first thing last seen on your element.
Because every time you do that you will get an automatic timeline and actually a quick
I would say quick win when you do analysis.
So if there are no more questions about event creation perhaps one of the things we can do is show the searching.
How to search for stuff in your MISP.
okay so this is something that we're going to show very briefly now
and we're going to go in a bit more detail into this tomorrow
when we're also going to look at the api but generally whenever you're searching in MISP
the main question you need to ask yourself is what scope am I searching on?
Am i searching for individual attributes or am I searching for events?
The search filters very often overlapping or are almost the same
but one of the things you need to keep in mind is for example
if i'm searching for bitcoin addresses in my MISP instance, bitcoin wallets.
Am I searching for any event that contains at least one bitcoin address
or am I searching for just the bitcoin addresses themselves?
So this is when we decide between different scopes
so generally attribute scope will only give you the individual attributes that match the criteria
and the event scope will give you everything that contains at least one matching value.
So here what Alex did, he just searched using the attribute search for all the bitcoin addresses in the instance
and we see we get a bunch of them from different sources,
we see which events they're from which organization has created that information and so on and so forth
If we're happy with the search results and we've set up all our features and we're getting exactly what we were looking for
maybe even several pages of it like here.
We can download the results in any of these supported formats
so we could say okay now we have all these bitcoin addresses out there
generate the csv out of it and it will generate a massive csv
with all the attribute information for each of these.
I hope you're not running my time instance out of memory.
There it is.
So if you open it. Just to see the results quickly. There we go.
So in this case, we now downloaded our search results as csv.
Now keep in mind whenever you're dealing with integration of MISP
with other tools or exports keep in mind that certain formats don't really cater
to exporting certain types of data.
So if you're searching for ransomware payout wallets.
you could, for example, specify as a tag, all the different ransomware related tags that you have
and as a type select BTC like what Alex has done here and export the information.
Now when you're deciding what format to download in,
again, some don't make any sense so don't download bitcoin addresses in STIX format
because STIX doesn't have a way to express bitcoin addresses for example.
So just make sure that you also take that into consideration when exporting data
so that it's not {inaudible}.
Besides that we can do the same on the event level, we can also do searches on the event level.
If we go back to our event index, we have a little magnifying glass icon
where you can add additional filter options to the index and filter the database on that.
So let's just do "CIRCL", we're going to just filter on events coming from CIRCL
and we can also add, for example, events that are not published.
If you wanted to do some final checks on whether...
We need to add the organization again.
whether we have any events that need to be vetted.
For example for our own organization, then we could use this filter for it.
On the event index, all of these search filters that you apply generate a specific url
and you can bookmark it, so if you have recurring queries that you want to monitor
then you can just bookmark the url and you can go back to it later on
and see if there is anything that popped up that matches your search criteria.
now generally like I think 90% of our searches do not actually happen via the UI
they happen via the API, so very often you have tools that you search through
So if you have a tool that acts as a front-end for your MISP for certain searches that works as well.
We're going to talk more about those type of searches and how you integrate with other tools tomorrow more
When we go into the api
There is a question about soft delete attribute search
I just lost the Q&A page.
So Martin asks "is there a way to do a global search for soft delete attributes?"
Yes, sorry, where is it? For soft delete attributes? Yes there is.
So not via the UI but via the API which you can also access via the UI by the way.
We have a tool. We have a built-in tool. We can even show this example there.
we didn't show the delete
Haha, that's a good point.
Let's start with the question first and then we go to the delete S296
so we have this built-in tool called the "REST client" that allows us to run searches directly from the interface.
So generally indeed we have a soft delete mechanism in MISP
that allows you to to not fully remove an attribute but mark it for deletion.
The reason why we do this in general is whenever we're synchronizing information
and we delete an attribute we want to inform all the other instances
that an attribute needs to be removed, it is revoked.
So this is why we do the soft delete where we hide it from the interface, we hide it from the exports
but we still keep the data and we inform the other instances that they need to also mark it for deletion.
Now if the question is how do we do a global search for all the soft deleted attributes.
So first of all what we need to do using our little REST search tool is
by the way we have the Modern APIs here, so to create a new API unless you know yours by heart
so alex because... So just quickly, so in the meanwhile what Alex is doing now is
he's going to generate a new api key for himself so that we can actually test the api queries.
Oh that won't work.
Yeah you can add another key from here as well, this will work.
Yeah that works.
Global action, my profile, by the way if you want to find your profile.
Okay, so now we have our api key.
Now we go to REST client we just paste it in there now in the authorization field.
Here we go and now what we're going to do is we're going to run a search
for all soft delete attributes so we're going to search for attribute restSearch.
So that is a scope that allows us to search on the attribute level.
we'll see more of this tomorrow, just a small example.
For return format, let's pick something like JSON
and perhaps set a page under limit or take one limit 100 or something like that
I don't know how much was deleted here but it might be a lot
and then just add another key deleted, there we go
and then deleted set to 1
and we don't need anything else
and this will return the first 100 hits from the instance of attributes that are deleted.
There we go And now if you wanted to paginate through all these attributes
you would have to just raise the page number.
Go back and and get page 2, page 3, page 4, and so on
or if we have enough memory certainly my training instance definitely doesn't
then we could just say give us everything in one shot.
Okay so I hope that answers your question Martin.
There is also a question, is there an official MISP docker image?
There are actually several, they're not maintained by us but by contributors
that are very active and working closely with us.
So I've pasted one example in the zoom group chat.
I don't know if maybe it's not visible to everyone. I can just drop it as an answer here.
Yeah, it's better.
So this one is done by coolacid, so why are there are so many docker MISP?
That's I think the specialty of docker, not everyone agrees on a model with docker
so there are at least as far as I know four or five different dockers
there's one managed by DCSO, one by CoolAcid, one by Xavier Mertens and
one by HarvardSecurity and I'm sure I'm missing some.
So the thing is for the docker images it's depending on I would say your taste
so have a look at what the different contributors are doing
and you'll see that you pick the one that is matching what you really want to do with docker.
Some are really more separated container wise some are more like one single container with everything
Again it's a maker of taste and how you want to operate one
we don't maintain one as MISP project
but there are some that are under our MISP project {inaudible} position.
Someone is asking about API key to invoke cortex analyzer.
For the cortex analyzer, it's a separate toolset part of the HIVE project
and then you have specific API keys.
Cortex extension is like MISP module so it works for the expansion services
Be careful, cortex analyzer are not supporting objects and stuff like that
which is the case for MISP modules so you might have expansion on the interface
but if you want full-blown expansion with relationship and so on
then you can use MISP modules. A lot of organizations are mixing both
so you can have cortex-enabled and MISP modulus enabled on the same MISP instance
But going back to the question if you already have the Cortex API encoded in your MISP
and you want to invoke a lookup through the API through MISP
then you can use your MISP API to tell your MISP to run a query against Cortex.
But with the new api key models usually it's better to have dedicated API key per {inaudible}.
Okay, there is something else, is there a way.. we had that already
Could you touch on how we could use one event to add multiple attributes
and how would correlation work here?
Configure event one to fetch all records from a feed
Would this work with correlation?
Show all instances where any of those attributes match with other events from other organization events
Well okay if I understand it correctly, indeed so if you do that
you create an event for a phishing feed and you have those attributes in there
and you have cached other instances, then within that that feed's event
you will see correlations both to other events created locally on your instance by other organizations
as well as links to other instances that have the data as long as you have cached those events
so we're going to talk more about that tomorrow about the synchronization
but when you're interconnecting with another instance you can do it in two ways,
one I want to start exchanging data, pushing data, pooling data
or two I can just tell my MISP to go crawl that other instance
hash all the values that they have and if I ever get the correlation
then it flags it for me then it shows me that the instance already knows about this value
and I can pivot over to previewing the data. So I hope that answers that.
Yeah, and then the correlation of feeds for example if you just enable the caching
you just see that it's correlating with specific values without providing the full feed
sometimes it's quite handy when you have for example feed that
you cannot show the data but you can show the correlation,
There's another one, do you recommend using MISP alone
or using the Hive MISP Cortex integration
I mean generally if you need a case management tool then using the Hive for that is great
So it makes absolute sense to use them together and integration is really smoothly done
so that means that no matter where you start your process,
whether you start by creating an event in MISP
or whether you start by creating a case in the Hive
you can basically propagate the data to the other tool
and work on both tools and data. So yeah, so absolutely
Yeah, absolutely it's pretty smooth, just be careful if you use the expansion on MISP
and you have MISP modules enabled I would prefer to have MISP modules enabled
because you basically have all the features of MISP like relationship, objects and so on
with the cortex integration is basically just a layover with the Cortex
Yeah but one of the things that you can do is
if you start for example from the Hive perspective and you push the data afterwards to misp
you can then go through this process like what we've done here with enriching the information
creating objects that affect attributes so you can do it as a secondary step
before you share it out to community to refine the data in MISP
that you've created in the Hive for example
and the same thing if you've used cortex to fetch additional information in the Hive
you can then take that data and further enrich it with MISP modules once it's in MISP.
Yeah this is a good question from Muhamad Junaid about
when I try to import the data from STIX to MISP it's called lossy
like can you please explain that a bit and this one is interesting
because it's... I was saying a long long long discussion
and that even influence how MISP evolved and the standard behind MISP
so STIX is really focusing on cybersecurity and cyber threat intelligence
and the problem is you might have at some point in time
data that are basically not defined anywhere
so it's more for the export of data so for example if you export in a MISP event
and you have for example an object with the person and stuff like that
it won't be in the STIX to export for example.
So it means that in MISP even if you get all the information
but it's bound to the limitation of the standards and the format
where you export and it's exactly the same for any format
I mean if you export a person in Suricata format
obviously you don't have any field or things like that with person and so on
so that's why we call it lossy because sometimes when you import data
it's bound to a specific set of fields that are supported and so on
Another thing that is quite important with STIX,
you might have a lot of peculiarities or specialities depending on the vendor
some vendors are adding some specific custom objects things like that
that are not bound to any existing one
so we are importing them as kind of you know generic one but it is basically like lossy again
so you have to be careful when you use a specific format
to be sure that you properly map an existing different one.
So it's more for the export, MISP quite flexible on that
so you can basically have any object you like but when we export for example in STIX one
we just export what is existing in STIX even if we start we added some custom objects too
some which are on to the MISP object but some tools will not recognize obviously the custom object
because they are just having a profile for a specific set of known objects.
Yeah I think that's exactly the point that maybe is different from
when we described the text in those import and export fields.
we say lossy but in reality what we do is we do try to capture everything
and we do try to map everything but a lot of it will end up in custom objects.
Now what Alex mentioned is the problem even if we export bitcoin addresses for example
whenever we're pushing in STIX2 format as custom objects
no other tool will pick up on it because we're just using custom objects
unless the other tool specifically looks for them
they will just either store it as is or not know what to do with it.
Yeah and that that's why we recommend a feed provider or vendors and so on to actively support the MISP format
then they can they can really import a full set of objects
and {inauidble, either some already exist in MISP/some are resisting it}.
Yeah in some cases however you don't really care about having the full set
and that's where for example specialized formats are really cool
so whenever we're feeding for example an ids for example we don't care about bitcoin addresses.
So in those cases STIX and MISP both are very expressive exchange formats
but whenever you're dealing with feeding tools for example
you don't care about about losing, 90% even, of the data set
as long as you capture those type of data points that your tool can process in the end.
So this is why generally what we recommend is
if you have the option for example to export data from MISP
is for your IDS for your SIEM and so on and you have the option between
for example STIX or Snort or Surikata go with Snort or Surikata
because those are much more catering to what your tools can actually understand
Yeah, for example for Yara the same you prefer to have like a
good Yara rule set that you can run into another anti-virus or your endpoint protection device
and having a generic one that will not help you to lose the detection.
Are there any other questions?
You already took that one that is there that is longer.
I think we took most of them unless they missed one.
Yeah, perhaps we should show the deletions because we we didn't actually show it, indeed.
Yeah, exactly.
Okay, oh and now we have some more questions but we can take those after.
Yeah let's quickly show the deletions so if we go to an event.
yeah I'll take a hundred events.
So this is a massive gotcha basically in MISP
which we have some protective measures in place to avoid this
but one of the things that you really need to watch out for is
when you when you add data to MISP and you noticed
oh crap I should not have added a piece of information that is either confidential information,
information about the victim that I shouldn't share and so on.
First delete the attribute, that attribute might still be contained in the event
in a soft deleted format. You can always toggle and see the deleted attributes within an event
So I will create an event from scratch as you can see
Okay so before I move forward on that so we have two protective measures in place
to avoid accidental information leakage violation.
One is basically that by default we do not use the soft delete method for
anything that was unpublished at first so we're going to show it as an example
so here's some sensitive information, if Alex were to delete this now
this attribute, this would get hard deleted,
So this will not create a soft deletion, MISP already tells us
are you sure you want to hard delete the attribute? So when you read the text
you will see the difference there in the wording.
The reason for that is the event has not been published yet
we know that it has not probably been propagated to other instances.
There is absolutely no reason to inform anyone that this has been deleted
so we can immediately just hard delete it.
So when we do that, it will get hard deleted.
However, if the event has already been published
this has already been shared out to other instances potentially.
So in this case, if we were to delete it, MISP will tell us
oh are you sure you want to soft delete this attribute
because this is already a published event.
Now it looks like our event is empty but if you look at the deleted flag
you will see that the sensitive attribute is still there
and if I were to publish the event now, this sensitive attribute
would get propagated along with the event.
If you want to avoid this altogether, there is a way to mangle any attribute that gets self-deleted.
What happens in that case is a category will be set to Other
type will be set to Other and value will be set to Redacted.
This is a server-wide setting so your administrator or if you are the administrator
then you yourself can set this setting in the server settings.
The downside of that is if you are mangling attributes that you're soft deleting
it will still inform the other instances, they will still remove the data
soft delete the data because the uid is reserved.
However, you cannot recover the attribute anymore.
So in this case right now we deleted attribute, Alex could now click on the recover button
and the attribute will be recovered as a normal attribute so if you made a mistake you can recover it.
So there are two different mindsets, I want to make my data recoverable
and that I want to always inform others
versus I want to always hard delete data that I delete.
Both of them have a setting so just pick and choose whichever makes sense for your community.
Whether you prefer secrecy or prefer convenience basically so that's it's basically.
-Yeah that's delete for attribute so if we delete an event that's another story. -Yeah
And this one is interesting because now we have these options where we say
I want to delete this event and obviously it will be deleted on your instance.
Nevertheless this even has been already synchronized, copy on different MISP instances
so that means at the next synchronizations the event should be pulled
but to avoid such kind of of issue
MISP is automatically generating a block list of all those deleted events
so if you are the administrator you can see at the "Blocklist events",
you can see the the one that I just deleted. So why we do that?
It's very simple. We don't want to re-import the event that has been deleted
because locally we don't want this event.
So it's a blocklist of all the {inaudible} but there is a catch there.
Sometimes we have people, oh i'm doing some tests and so on
I'm synchronizing with MISP but I can't seem to get my event back
and obviously yes because it's there in this block list.
So if you have some tests and you're running some tests, don't forget to look at the block list
and maybe you want to just remove the event from the blocklist
and then you can synchronize back the event.
there's something to keep in mind it's there, it's done automatically
but in some cases you want to manage the blocklist so that's something to keep in mind.
Yep something else that we perhaps should touch on here is
for the event deletions besides just a blocklist part
there is one thing that comes up as a question very often is.
how do I inform others that an event needs to be removed?
We don't have a mechanism in place for that so while we can revoke attributes for events
We don't have that and there's a reason for that.
In general whenever it comes to events we don't want to give the power
to just outright delete events remotely this way so this might change in the future.
We're having discussions on that whether we want to enable that or not but currently that's not the case.
Yeah and usually we take as an example emails I mean you can remove emails from your personal mailbox
but from the remote mailbox if someone already receives the emails
you want to have the control over third parties on the mailbox
that might be one of the drawback I would say.
So there are two new questions one of them is basically can you demonstrate
the progressive enrichments of events by the shared communities over time with correlations.
This one is tough I mean i'm not sure how we could demonstrate that because
we're not dealing with live instances with live data sets and active sharing
but perhaps for tomorrow we will prepare an example where we can show it off
and choose an event that we can show on one of the operational instances
but I can show one on, you know what I can go on just one second
I'm going on an instance. So it was.. oh we are flexible
So it's not simple to do it no but so it's maybe something interesting there.
So I'm connecting an instance where I have more expansion services active and so on.
I'll just keep it for "My Organization Only" so i'm creating an event there
so what happens on progressively enriching event by shared communities
I mean it's going back and forth to different communities
but I can imitate what the community is doing usually.
so if I'm creating an attribute for example I will create a hostname with some network activity.
So we have specifically a test that we created with this kind of...
so what would be your community and sharing?
So it could be for example in the same organization
in my case it's just shared to the organization so if I publish event here
it will be shared with all different instances maybe the different members of CIRCL
and one of my colleagues is taking one of the indicators there
and then he's going on the Faresight database doing a full-blown expansion
so that means he's basically doing a full-blown expansion.
What do I have here? I have a a complete set of objects for a specific domain
so you see again I'm going to the event graph now I enter my domain name
and I have all the passive DNS records associated to that one
and in this one I think I will have the event timeline
I have a completely different timeline of the different expansion and so on.
So then I will have one of my... it will be published again with the data.
If it's a collaboration I would say in the same team that's a thing
so it's sometimes people are working on the same event and publishing it
Sometimes they are sharing it and doing additional expansion on the things
until to reach a specific point that is like I would say
accessible or at least publishable in a publishing state that is acceptable by various people.
Now we can make proposal too. So that means if we are again with a different organization,
i don't know if in this example it will work but I can take...
do I have something interesting there
Yeah, for example I see an interesting IP address, this one.
So what I could do is i could add..
What's going on here?
{inaudible}
Okay just a demo effect, typical.
What's going on? Okay, just going back to this one I just want to add a proposal.
Yes I cannot just...
you wanted but you're admin.
You can cheat if you really want, you can do it.
Yeah, wait, fine. It's just like okay so I don't know for {inaudible} if we answered your question
but I mean a full-blown step would be like that, if you work on an event it's not a single person obviously.
When you do an investigation, you do like multiple steps but the question is more like
if you do it within a team usually you edit the current event in the same organizations
if you do inter-team, you do proposal, extend it even like we showed before
and then you start to work on this thing, so it's really depending on the case.
so I hope you can see what are the capabilities there
but it's really the progressive approach of collaboration usually depends on how people are working together.
If they are really external it's more proposal, extended event.
If it's within the same team it could be extended event or within the same event,
that's usually the two way of working. Andras, if you want to add something on that?
No yeah, that's perfect. Perhaps another question if you're okay with switching.
Yeah when speaking of feeding tools what would be the automatic way of doing it?
So normally when we're talking about feeding tools there are two separate ways of doing it
and we'll go way way deeper into this tomorrow when we talk about integration
but generally tools can either fetch data from MISP
so this is a more common way where a tool would use REST search API
that we mentioned before where you define your search patterns
for example give me everything that is newer than 30 days,
everything that is not coming from say OSINT sources
or perhaps not something, nothing that comes related to a certain topic
for example I'm not interested in Ransomware when feeding my tools, just a stupid example.
So you set up your filter options
and then your tool would fetch data from MISP every 60 minutes, for example.
and then replace the data set there. You can also do sliding time window searches
where you say give me everything from the past 60 minutes that is new
and then you keep concatenating your dataset on the SIEM side, IDS side, whatever tool you're feeding.
the alternative, if you want to have the data push automatically as it comes in
you have different channels in MISP that your tools can latch on to.
The downside being that you still need to do the conversion in those cases.
So if you were not using the APIs to fetch the data from MISP
then MISP can push using the MISP JSON format data down
via different channels ZeroMQ or the Kafka channel or this blog and so on
and then your tools automatically feed on that data so you have these two different ways of interacting with it
There's also a third way where you can basically
either build an export module or an enrichment module where an analyst can trigger
a direct push of a certain data point to another tool, so that's another option.
We'll talk about these different strategies when to use which and how to mix those tomorrow more.
So I hope that answers it in a brief fashion.
Yeah what i'm showing here is it's just like on the REST search client
for example you want to feed your Suricata and so on.
Just take page and a specific limit. So what you can do is
if you have a python script and so on you can pull directly the data.
So the rest client, so you see in this case I have the Suricata rule set
but if you want to feed your specific tools and and so on,
automatically we are generating curl and python card so it could be a bootstrap to see okay,
how should I create my own tool for feeding my IDS and so on.
For Suricata, for example a lot of management interface have already MISP connector
So you can even like feed the data directly from the interface if they have the ability
Splunk for example, there's a specific application
which is an external tools part of the app store of Splunk
that you can install for doing the connection
and some other people are using their own python script to feed other SIEM.
So again it's a matter of taste if you are curious about the different kind of integrations
or you can do it in Python for example on PyMISP itself there are plenty of examples.
So if you go in the example directory of PyMISP
you have quite significant set of default scripts that you can use
and that's I think usually a good basis
if you want to start to write your own custom custom tool set for
feeding your systems or existing software in your infrastructure.
Yep, I don't know if we should jump on a new topic
or we just push the Copalos example for tomorrow?
yeah I think we can do maybe tomorrow.
I think that would be stretching it a little bit if we were to start with that
So quick summary of today, so today we showed how to create an event
the basis of MISP like what is an attribute, an object and so on.
How to create it, how to make proposal, delete and stuff like that, so it's really a simple example.
Tomorrow, we want to show you more the event report aspect
and automatic imports into into MISP with a practical example of an original report.
and we will discuss tomorrow about how to build sharing communities
and especially we will share our experience of things that worked
and things that didn't work in the past years when creating sharing communities.
So if you are ISAC members or creating your own sharing community even within your organization
It's something good to participate because we will share with you
some of the things that are interesting of building and bootstrapping such kind of of community.
I don't know Andras if you want to add something
No and that's basically it thanks for everyone for sticking through this
through it's a very condensed session so we said we didn't make as much progress
as we hoped so we have quite a bit left for tomorrow
and hope to see you all here tomorrow.
Thank you very much, take care and don't hesitate to ask questions
either later on directly contact us, thank you very much, see you tomorrow.
Thank you all see you tomorrow