(high pitched chiming)
(applauding)
- For those that were here last year certainly know
that I'm a World War II history geek.
My company Digital Shadows started in London
and I just hit my two year anniversary last week
and I've been to the UK nine times.
And so when I'm there over the weekends I always go and do
all kinds of military history things to see.
Anyone been to RAF Duxford before?
Any of the Imperial War Museums?
They're awesome, RAX Duxford is a live museum,
you can be in Spitfires, they're quite expensive,
you can go up for 45 minute flights.
But I geek out there.
And some of you may have seen in previous talks
this analogy I'm gonna use about analysis
but it's really relevant for here.
They have a US Aviation Museum at Duxford.
So of course we have SR-71
and we have a U-2 kind of positioned
like Powers would've potentially been positioned
when he was shot down.
A little bit when I was in the army I worked with the U-2
as opposed to some of the other assets
we also worked with too but total geek out here for
an intel person, a military history person
and general geek, I love Duxford.
They also have a land warfare museum.
Does anyone here, maybe targeting people,
know if this is a BMP1 or a BMP2?
So this is a BMP1.
And you can tell, as a former imagery guy,
the size of the turret on top is one of the
indicators for it, but I use this to tell an analogy
of a young specialist Holland who went TDY
or to the Eisenhower when I was in Kuwait
and I was working with the ISs there,
and we were doing imagery from the TARPS platform
and we were looking in Southern Iraq
and specialist Holland made a mistake
and he called the BMP1 a BMP2.
Does anyone know the significance of
what this would'a been between the two Iraq wars?
Only the Republican Guards Divisions had BMP2s.
So specialist Holland basically called out
an Iraqi Republican Guards Division
further south than they should've been and you know
it hit the fan, things were spinning up.
There wasn't a lot of analytical rigor
in specialist Holland at the time.
And so I use this as a really good example of
making a mistake, maybe not having enough
peer review as well.
A lot of analytical failures didn't really turn out
to be that big of a deal, but for a little while
it was a problem.
But I really think it's a good way to lead into
analysis matters.
Certainly if you're in the intelligence community
it might mean something different
than it would be in the private sector.
But in the private sector, could be your livelihood
depending on what type of work you're doing,
commercial sector, competitive intelligence,
whatever the case may be.
So I always like to use this as a lead in on analysis.
And my personal failure, I like to make fun of myself a lot.
And there's a lot of things to make fun of.
Of course, I'm not gonna go into Dick Heuer Jr's book here,
I've had this on the slides probably for three years now.
But what I wanted to try to do is evolve,
is to look back, I spoke every year at the conference,
I've coordinated the conference
with the advisory board as well for the past three years.
And so I've had this slide up
and what I was thinking about this year is like,
OK, enough about just showing this slide
and telling you to read this book.
I want to try to move the conversation forward
on better analysis.
This book, how many people have read this pdf?
And you can download it for free right?
One takeaway if you haven't read this pdf,
highly recommend that you do,
it's probably one of the top five things
you should read in the space.
Also Thinking Fast and Slow, but I think the first time
I saw Kahneman talk, she also recommended that book.
And I read that as a result.
So here's, I'm not gonna read all this to you,
I was gonna highlight some of the important parts but
it's really about externalizing
and decomposing our thinking.
I think we have a lot of lone gunmen,
maybe that's not the best analogy, people that are
champions, they want to do it, heroes,
that's what I was looking for, superheroes.
They want to do it themselves, they want to save the day,
they want to get all the kudos for this
incident response analysis or this
threat intelligence analysis.
And really what I like about structured analysis
is you're exposing what you're thinking about
and you're getting more analytical rigor
into what you're doing.
There's a number of cognitive biases,
and Carmen mentioned some of them.
And I'm gonna go through a couple.
You can do a whole session on just cognitive biases alone.
But here's confirmation bias.
Facebook right now, if you looked on Facebook,
it's probably the number one source of confirmation bias
in these hyper political times.
Another one is mirror imaging.
Anyone familiar with mirror imaging or Into the Dragon?
So if you've seen this movie, it's probably
one of the top five movies ever made.
He's being attacked, he's being scratched,
and essentially he has to break the mirrors.
So you actually need to do this for your analysis,
you need to break through your mirrors.
Mirror imaging is basically when you look at,
and you make an assessment through your lens
and not the person who may be targeting you's lens.
You're not taking into context their social situation,
their geopolitical situation, them,
there's actually a technique called red hat analysis.
I'm not gonna talk about it today really
but they can be used to overcome mirror imaging.
Anchoring.
I think Carmen also mentioned anchoring in one context.
Is anyone familiar with anchoring?
We actually see this a lot with university students.
At my company we recruit a lot of people
from King's College in London,
graduate programs in intelligence studies and
there's a tendency with students that go through programs
that maybe the first analytical technique they stick with.
So you can anchor on the techniques that you know,
and then you can also anchor on maybe
the first assessment that you have.
And then perhaps, it becomes, oh this is the way
it's always been so it's gonna stay this way type of piece.
So we don't want to anchor either our techniques
or our assessments.
And then of course groupthink.
And I think everyone's familiar with groupthink.
I think this is a pretty good illustration
of groupthink as well.
And this is what
psychologists who coined the term groupthink,
Irving Janis talked about these conditions
that make groupthink possible.
I'm not gonna read through them all
but if you look at your org structure,
if you're maybe a student,
a fan of organizational behavior, some of these things
might reside in your organizations.
But there's one in particular that I highlighted in red,
those are situational factors that contribute to groupthink.
And I think it's really important
for this audience to be aware of.
How many people would describe themselves
as this individual in their jobs?
Are you ever tired?
Are you ever stressed out?
Are you ever working long days stuck on a console
for ArcSight or looking at packet captures?
The very nature of our role, no matter if you're an
incident responder or intel analyst,
anyone in cybersecurity today
in the threat landscape we have, is you're tired.
So we really need to be aware that fatigue
can cause groupthink, and really try to take steps
to protect ourselves or take care of ourselves
to try to avoid that.
Sleeping more than four hours a night is highly recommended.
One of my New Years Resolutions was try to sleep
seven hours a night, I'm not doing very good at it.
It's ACH, how many people have done ACH
in their careers at some point?
There's a fair amount of hands.
With this audience I would expect it.
This is an ACH that we did
at Digital Shadows on WannaCry and actually
made the Internet Storm Center.
But I don't want to talk about ACH.
I was not exposed to actually doing ACHs
until I've been in my career at Digital Shadows
and I actually think they're a pain in the ass.
They take a lotta time.
Even understanding, as you're setting up
the assumptions that you would have in it,
the scenarios you would have in it takes a lot of time.
But ACH is probably one that gets the most headlines
these days, people are talking about them and using them.
So I want to think beyond that.
So I'm not gonna talk about ACH
'cause I actually think it takes too much time
and there's other techniques that you can use.
There's a whole lot of techniques that you can use.
So just a small, and I have the reference
to this book at the end of probably 40 different
structured analytic techniques on this slide.
Is anyone familiar with this book?
I've got it at the end,
it's like a 70 or $80 book,
it's probably not a bad thing to expense for your teams
that are out there and they break down
all these structured analytic techniques.
I think it's very overwhelming in trying to know
the right ones to use in the right situation
is pretty important.
So I was gonna talk through a couple things
that you might do in your organization.
As you start to incorporate structured analytics
into your programs, and I think you really should,
I'm gonna go through a scenario and go through
a couple here, but you could kind of,
and what I've been trying to do here is line it up
to the tactical operational and strategic goals
that you might have from an intel perspective
on different types of structured analytic techniques
that you can use.
I'm a newbie on this.
Rob in his intro said no one's a pro here,
no one's an expert.
I am by no means an expert in
structured analytic techniques.
But I have been working with them,
I've done structure brainstorming, I've done ACH,
I've done SWATS, red hat analysis, cone of possibility.
But you probably are gonna have some
that are gonna be in your stable, and you're gonna use them
and it's important to refresh them from time to time.
But what I would recommend is you start to try to map
different techniques to different things
that you're gonna produce in your environment.
How many people on an ongoing basis in the room
actually have structured analysis built in
to the different intelligence products you create?
I see one hand, and that's an Army hand.
So typically that's what you'll find.
Unless you're maybe a very large
Silicon Valley technology company
with large intelligence teams, or you come from
the intelligence community you're not doing this.
But the whole point of my talk today is
that you don't have to do these crazy ass ACH
that take a lot of time, potentially take a lot of time.
Once you have them down you can do them quicker.
There's an easy way to do it, or easier.
And this is the common, this is the third time
over the past six months at a conference where I've
talked about this topic.
And I'll talk to people afterwards and really
this is what comes back, is we don't have time for this.
Our hair is on fire, we don't have time
to add new things in.
What I would say is you probably already doing
some types of analytics, structured analytics
that maybe you hadn't thought about and maybe
you just need to formalize a little bit more.
So I want to go through a scenario,
and in this scenario you have a recently promoted CISO,
let's call him Mick.
Mick came from the intelligence community.
So he's really into the intelligence trade craft
and of course like all CISOs or C-level folks,
he reads the Wall Street Journal.
And he just saw this headline this week
in the Wall Street Journal and tell me if
this has ever happened to you.
The CISO comes and knocks on your doors
and wants to know about this article in New York Times,
Financial Times, Wall Street Journal
and starts a fire drill for your organization.
Has that happened to anyone before?
Yeah yeah, well now I'm the one doing that now.
So this is Mick our fictional CISO's response
to this and he really wants to know what's gonna happen
for the organization.
So I'm gonna walk through this hypothetical ish situation.
I'm gonna do two techniques, we'll walk through,
it's really to give you a flavor of
how you could try to apply this to a real world scenario
from your CISO that's zomygod-ing the news.
So we won't go into detail on some of these things,
others we will, but I just want to give you a flavor
of how you can actually incorporate
something like this into your program.
And you could use other techniques as well.
Just a flavor.
So one that I think is really, really important
and actually doesn't take a ton of time
is a key assumptions check.
I think this is really important
for a lot of different things.
For those of us that are on the vendor side
and we're producing our super cool intelligence reports,
this sort of stuff, if you're on the vendor side,
you're writing research that you're using for
thought leadership marketing purposes,
you really should be doing a lot of structured analysis
inside of your research.
But key assumptions check is really really important.
It's the foundation of your intelligence product.
And if you're making faulty assumptions,
and in intelligence there's no certainty right?
We don't know everything,
so it's really important to do this.
This goes back to
what Carmen was talking about is you need to have a culture
that will allow you to question other peoples' assumptions.
Right we have to not have hubris, we need to be humble,
we need to recognize that we don't know everything,
that another perspective could be a better perspective
or complement your perspective.
So you really want to understand the evidence
and reasoning behind something.
So in the scenario that we're walking through
on cryptocurrency fraud, Mick our CISO
works for a financial institution,
so he's definitely concerned about the implications
of cryptocurrency in general and then fraud in particular.
We've got a number of key assumptions.
And so these are the assumptions there.
A couple of new alternative coins and exchanges will emerge,
cryptocurrencies will eventually be adopted
by major retailers and financial institutions.
Some of these are trailing trends anyway,
but we're making those assumptions
and we're making them clear up front.
So whenever you start any type of analytical process,
whoever the team is working on writes these assumptions down
and then if you're fans of Silicon Valley,
you get the stick it notes out.
And actually Carmen referenced this
with structured brainstorming, another way that
you can do this exercise as well,
you get the sticky pads out.
And you start to ask yourself these questions
about the assumptions that you've made.
What would make an assumption untrue?
What if it was true in the past but it's no longer true?
Assign a confidence level to the assumption,
and then basically, and there's some others
that you can add to the mix here,
but that's a high level of what you would do.
You ask these questions about each of these
key assumptions and then you rate them.
This is solid, we feel really good
about this assumption, it's valid.
It's caveated, meaning something must happen
in order for this, or it's unsupported.
And then if it's unsupported, throw it out.
So I would say for any kind of intelligence product,
this is the type of activity that you'll want to have.
And this is not something that's gonna take a long time.
You could do this in an hour.
And I'm gonna talk about some remote and collaboration tools
you can use as well.
For my organization, I have people from Greece to London,
East Coast, Central, West Coast.
So it's a large distributed team.
So there's some challenges there as well.
So now we have the assumptions.
And one of the things about the key assumptions check,
even if you were to do an ACH, there's many of these
techniques where assumptions
are gonna be a component of it.
So this is using one structured analytic technique
to build upon another.
Has anyone ever worked with scenario planning or futures?
Has anyone ever done a cone of plausibility?
So this is looking forward.
And what I like about this, especially as a CISO,
what I'm trying to accomplish here in this example
is how you could take some intelligence assessments
and actually then use them in your program.
We talk about this panacea of intelligence
informing business decisions and giving you
decision advantage over your blah blah blah blah blah.
My goal here is to try to start thinking about
how we could actually do some of this stuff.
So cone of plausibility.
This is a forecasting technique.
Often times you'll see it in larger
intelligence organizations and may do them further out,
you know 2025, what can we expect.
I like to make fun of forecasts.
Not the forecasts but the predictions.
As Rob said I was a Forrester analyst and every year
in October we'd do these predictions.
And I always like to talk about forecasts
and not predictions because there was no
analytic rigor really in the predictions that I would make
as an industry analyst.
I don't know that I was blind to that fact then,
but what I like about forecasts is at least you have
some rigor that's going into your thought process
that you're using here.
So essentially what you're trying to do is project trends,
events and their consequences,
and you look at scenarios that occur.
And I think from a business that is trying to plan out
the implications of something,
in this case cryptocurrency fraud in 2018 and 2019,
you can start to talk about things in the business
that they might care about and then you start to,
I'll walk through it, let me not get ahead of myself.
I have a reference for all the things that
I'm talking about I've got references at the end.
So you have one slide that you can just look at,
it's similar to what I did last year.
If you look at that one slide you can just
hyperlink out to all these things.
So there's a lot of things in the cone of plausibility
as far as the framework goes.
Understanding the current conditions,
so the drivers and the assumptions.
How many people are familiar with the PESTLE model?
So you can use the PESTLE model
to help do the drivers an assumptions but
what's driving the current state?
What are the key assumptions that you're making
about the state, basically new alternative coins
are gonna come, new exchanges are gonna come,
fraud's gonna continue what we went through
on the previous one.
And then you build out the scenarios.
And so you have a preferred scenario
which in my experience in doing these preferred
really never happens, a probable,
and then you have wild cards.
Now the wild cards, I think one other thing that
we need more of from analysis perspective is creativity.
So wild cards is you just pick something that's
super random that you may not have thought of,
completely out of the box,
and you have that as a scenario.
So it's really just to kind of challenge
the thought that you have, and basically
what you then do is you monitor for these scenarios
and then you come up with countermeasures
for those scenarios, it'll be the process.
We'll walk through it a little bit more.
But I really think the wild card one,
in the ones that we've done internally at Digital Shadows
it's kind of fun to do that one.
One that we did about 12 months ago was around
DDoS botnets and we were trying to think about
how extortion actors might change,
I'm sorry with IoT botnets, doing DDoS,
and so one of the wild card scenarios we had,
and we've seen a little bit of this but
not on a grand scale was gonna be grand scale of
crowdsourcing extortion so instead of extorting
the brand, it's Christmas and Rick wants to get on
Xbox Live or PlayStation, and we take it down
and now I'm like OK Rick, if you pay me
$100 and we crowdsource that, then we'll bring
the service back online.
So that was just one example of a wild card
scenario that we did there.
So this one, it's not as bad to read on that screen,
it's kind of what it looks like from a large scale.
You've got the drivers that you've come up with.
And here the drivers for this scenario is
the accessibility of the technology, the anonymity
that you have, or lack of anonymity
as some of the cyber criminals are finding out,
popularity and high reputation, the opportunity,
there's all kinds of opportunity for fraud, the users,
the exchanges, the platforms,
regulation or lack thereof regulation.
And then the security of individuals,
consumers and organizations.
Again, this is just to be illustrative of what you could do
for this and your organization may not have
anything to do with cryptocurrency.
Then we have the assumptions that we talked through
before on the previous slide, and then we have scenarios.
And really at a high level in scenarios preferable is
law enforcement gets their act together,
regulations come into place and the risk level goes down.
Pretty unlikely I think for most things in our area.
The most probable one, and I'm gonna walk through that
in subsequent slides, is the cyber criminals
continue to innovate, they target the cryptocurrencies,
the exchanges, they're having a lot of success,
they're making a lot of profit.
You might even see
other types of networks that may have been used
for ransomware that are moving over to this
particular piece on crypto jacking as an example.
And then in wild card scenario here, heavy regulation,
decrease of anonymity, the fraudulent attacks...
(mumbling) Water.
The number of targets greatly decrease
and this just goes away as well.
So unlikely on that scenario as well.
So here we're a financial institution.
Mick the CISO he's concerned about the exchanges
and the customers, the employees there.
So you have your scenario so you go through,
you understand the drivers, you have the key assumptions.
You think about two or three scenarios
and maybe you just plan for the probable
because that's all you have time for.
And then you start to monitor for that.
You want to be able to monitor changes over time
that indicate this is about to occur.
And then you want, you develop courses of action.
Of course military people will understand
courses of action but this would be your plan.
So for me, Mick the CISO trying to understand
what do we need to do practically about this threat.
So several things that I would, trying to take
a change in the threat landscape and tie it back
to my security program and how I might make investments
or de-invest in certain areas.
So things that I would want to do in this particular case,
we know that criminals are trying to use your Amazon Compute
for crypto jacking, so you're paying for them to mine
Monero, Bitcoin, whatever the case may be.
We see more Monero, a lot of cases.
But here, do I have a capability to monitor GitHub?
Now GitHub does a really good job of take down requests.
There's a lot of misconfiguration.
Let's talk about something that in our programs
like shadow IT, GitHub is a really painful component
of shadow IT, so here I want to understand
what's out here on GitHub, what misconfigurations
did someone do that shouldn't be there
that's exposing risk to my organization?
And then you start to see your Amazon bill going up.
So this is one practical thing that I might,
I may not have a capability to do this.
How can I do this, start monitoring that?
Another one, now this one probably is
appropriate for everything right?
This is web browser so this is crypto jacking
of your users' browsers themselves which
crypto jacking in and of itself,
using your browser resources it reminds me of,
people remember SETI@home back in the day
where you could my computer to, that's what
actually I kinda think about crypto jacking.
I actually wouldn't mind a SETI like
if I was gonna let someone mine my computer.
It was a non profit, let's say it's
a cancer research center or something like that,
I'm not necessarily opposed to letting them
mine my browser if it's gonna help them get Monero
that they can use for research so it's not to say
that all of this crypto jacking is necessarily bad.
There is some good out there potentially.
But here we want to look at our browser security
and the controls that we have extensions.
And I would guess for a lot of organizations
this is probably a weak area of their configuration
and patch management in general.
And this obviously helps out way more
than just a crypto jacking scenario right?
This is everything scenario since the browser
is such a prevalent attack pattern.
Another one, Have I Been Pwned, this is something
that you can do for free.
You can go out and look at your organization.
You can also subscribe to it as well,
but Troy Hunt puts together these databases.
So Mick the CISO wants to know if any of my employees
have been using their corporate email address
to register on one of these exchanges
that they're on, that exchange gets popped,
they drain all of the coin from them,
transfer it out, and then they have the email addresses
of everyone out there.
So what kind of risk does Mick the CISO's team have
from employees that are reusing their corporate credentials
even just the email address, right,
and these exchanges get compromised?
So this also then becomes a multi-factor authentication
conversation as well.
So again what I'm trying to do is map this threat
that you get from the Wall Street Journal
taking it down to how do you actually change
your security program as a result?
Another one, and I think this one is an interesting one
we were talking about, if you think about Millennials,
Millennials are more apt to invest in cryptocurrencies.
I actually see it as an opportunity, one is
I need to cover this in my security awareness training.
How many people have run security awareness
programs in their careers?
They're tough, and the way you get people to connect,
you talk about threats to their family,
to their children, well here,
for the Millennials that are interested in this,
this might be a way to connect with them,
and then you talk about hey look.
There's all kinds of fake exchanges set up,
fake platforms for trading, just understand
the overall risk, if you want to make the investment
that's fine, that's your business.
We're not gonna give you investment advice
but just know the risk there.
Oh by the way, here's some other things
that you want to think about.
So this could be an opportunity to take again
this Wall Street Journal zomygod article
and actually do something in my organization
to try to level up the security a little bit.
So what can you do?
There's a couple things and these aren't gonna be,
some things that'll be interesting,
some are more obvious.
But one, has anyone read this
Cases in Intelligence Analysis?
This is almost like a compendium book.
It's really good, we just started in the Dallas office
with our intel analysts, we're starting to take an event,
and basically it could be something like
attempted assassination of
anti-Putin journalist in Russia,
things that have happened historically and then
this is a workbook and they'll give you two to three
structured analytic techniques to work through.
What I have found to be the most valuable is
Harrison's one of the guys I do it with
in my office in Dallas is just us talking.
We're in a room, we're discussing these things
and it's just, the mere fact that you're having
communication maybe goes back to Carmen's point about
it's not as much about the structured analytics
as it is about better analysis.
This has been a really good way for us to do that
within our intel team out of the Dallas office.
So especially if you're a history buff as well,
you'll like this 'cause you're going back to
all sorts of, there's 20 of them in there.
In the books, maybe 60, about something like that.
So recommended take-home assignment for you.
Google Docs, this is one, now I will caveat this with
I realize Google Docs is in the cloud.
So you could use another solution but conceptually
either you host your own solution but
what I like about Google Docs, we used to use this
at Forrester, we'd have all the analysts,
I think we had 12 analysts on the team,
and we would be brainstorming
at the beginning of the year about
the threat landscape, customer pain points,
what do we want to write about?
And it would be ready set go and then everybody
just starts throwing up ideas on there
and just seeing them pop pop different colors.
Has anyone ever used Google Docs in this context before?
It's a really good technique especially,
especially for remote teams.
And I think that's a big problem that we have
just in general, especially as there's a trend
for more remote workforce, and you want to enable
your own remote employees so I really like this model.
And then essentially what you do is you start to
group these ideas together and you put them here
and then maybe you do a key assumptions check on that.
So it's a way that you can start off with
structured brainstorming, then you do
key assumptions check, then you can move on
to whatever else it may be.
This could be you're brainstorming on the threat landscape
for your organization in 2018 and beyond.
Maybe this is on the vendor side and you're trying
to think of really exciting research and capabilities
that you have that you want to highlight.
Whatever the case may be, Google Docs is one route to go.
Then, you can go big.
Now I do think, anybody here a developer in a previous life?
Developers, it's not the same everywhere,
but developers do a pretty good job of collaborating,
especially when you have remote product management
and development teams.
But there's tools like Google Jamboard
where you actually have a big TV like this
and it's interactive and I can draw ideas on it
and then I can use it on my phone,
can use it on a tablet as well.
So it really helps with both in person collaboration
for those in the office, as well as people that are remote.
The TV is like five grand so this is,
you're starting to (mumbling) a little bit more but
if you think of a threat intelligence team
and you want to have a better collaboration,
and this doesn't necessarily have to be,
this could be other things besides threat intelligence
this could be, I want to draw the architecture
of my environment in a real time view
so that we get a better understanding
of what our architecture looks like.
And Google Jamboard is not alone.
Stormboard has, it's a Canadian company.
They have an OnPrem version as well.
This is one I created last night on my own.
And you can put the note cards on.
So you could put the note cards up,
everyone could do that and then you could group
the note cards and things like that.
So there are some pretty nice collaboration tools out there
that you can use, again, you can have some
that you host yourself so you don't have to worry about
this being in the cloud.
A lot of the collaboration tools,
if the content you're working on isn't sensitive in nature,
they plug into Box, they plug into all the CESS applications
and it's a really convenient way to work.
Again, it depends on the content
and sensitivity that you're working on.
And Carmen actually mentioned this,
cued me up here with IARPA.
So IARPA has funded, I think it's about 50 million across
four universities to do research in,
CREATE is the name of the program,
but Crowdsourcing, Evidence, Augmentation,
Thinking and Evaluation.
If you want to put a Google alert down
on something to track, this, the first,
it's been in place I think maybe it's May of this year,
these universities are gonna present their findings.
Now I don't know how much of it will actually go public
but this is a good one to put a Google alert on.
And here's three of the specific products,
projects that they're working on.
Some of them are more focused on crowdsourcing,
of analysis, using AI with that.
In this case, when it's a university person
talking about AI versus vendor marketing,
I'm more inclined to believe their AI or maybe not
have as much distaste for it.
'Cause I am firmly in the camp of (audio muting)
machine learn and AI all the things is the wrong approach.
Maybe that's my bias from being analyst, it is,
my bias from being an analyst at Forrester
and getting machine learned to death
from everyone out there.
Swarm is a wiki, when you can ask questions
and get responses back.
Trace is another collaborative tool.
I don't know what these applications will actually
come out being, but there could be something
that we could leverage in the commercial sector as well.
And then of course for our friends
in the intelligence community, you may very well
be able to use these types of solutions
in the next several years.
The one thing I would say is this, SATS are not
silver bullets, just because you have
structured analytic techniques
doesn't mean you're not gonna fail.
Intelligence is, at best, a guessing game right?
Hopefully we can do a little bit more than that.
So don't think it could be this false sense of
look at all this analytic rigor.
We had these three techniques that were used
in the production of this particular product or asset
and it's good to go.
No you cannot be complacent with these,
you need to revise the tool set,
Carmen talked about that as well.
And I think this is a great quote.
I often ask if this is an overdone quote.
It depends, is this an overdone quote for this audience?
OK.
I like it.
Tell me what you know, what you don't know
and then most importantly, what you think
is most likely to happen based on that.
I think this is what's really really important for us.
I think if you look at the way we've been focused,
very much indicator focused over the years,
some of this stuff isn't, this is an abstract level above
where we're at but I think just knowing what you know,
knowing what you don't know is the tough piece.
This goes back to the Rumsfeld analogy a little bit too
right the unknown unknowns.
But at least for the things that you know,
what's your confidence level around
that realm that you do know it?
We know it really well or we don't know it that well.
I think this is a really important one to go off of.
That's a picture from last year.
I just was gonna throw up a thank you slide on there
and I found that was our monument walk
that we did last year.
I did put as a reference, a number of different things
that I put on here, and if you want to go through
an actual painful exercise, that second to bottom,
PARC ACH Software, has anyone ever used it?
Do you like it? - It's so old.
- Yeah yeah.
Might just be better to go with the spreadsheet.
It could be a little bit easier but
actually PARC's got some other tools out there as well.
There's different references, some stuff from Rand,
a couple different pdfs that you can download as well.
The one call that I would have on this,
and I don't know if a Slack channel's the right way
to do it but I really want to foster more conversation
about better analysis in our space.
So if you're interested in this,
if you're passionate about it, if you have experience,
please reach out to me.
I'd like to form some sort of group where we can talk
and collaborate on this and then further it for the field.
I don't know what it's gonna look like.
These may not be the right things for your organization
but I do think as we look to mature where we're at
as an industry and practitioners,
we need to be doing better analysis.
Thank you. (applauding)
(upbeat music)
For more infomation >> This is why we play Rust! Why did we ever stop?! - Duration: 8:34. 
For more infomation >> What is The Pattern Day Trader (PDT) Rule in Stock Market Trading? - Duration: 5:24. 
For more infomation >> Meghan Markle Is in Full Duchess Mode Just 3 Days After the Royal Wedding - Duration: 2:37. 
For more infomation >> REVEALED! LORETTA LYNCH SIGNED OFF ON SPY IN TRUMP'S CAMP – THIS IS HUGE! - Duration: 10:23. 
For more infomation >> Homeopathy Is Customized Medicine - Duration: 1:21. 
Không có nhận xét nào:
Đăng nhận xét