This document discusses how to evaluate your organization's security spending and portfolio. It notes that even security experts don't agree on exactly how much should be spent. The document recommends using a Cyber Defense Matrix to identify key assets and risks, and evaluating spending based on addressing real risks and having evidence it is effective, rather than just compliance or benchmarking. It also cautions that spending alone does not equal effective security, and processes are also important.

1. HOW MUCH SECURITY
DO YOU REALLY NEED?
Wendy Nather @RCISCwendy
Research Director, Retail Cyber Intelligence Sharing Center
(R-CISC)
Bogotá, 24 Octubre 2016

2. INTRODUCTION
• The Great Mystery
• “Expense in Depth”
• Even the Experts Don’t Know – pricing out a security
program
• A better framework – the Cyber Defense Matrix
• Trimming your current security portfolio
• Evaluating the risk in a way that works for you

3. MODELS FOR SECURITY SPENDING
• Benchmarking – what is everyone else doing?
• Compliance-driven spending
• Metrics-driven
• Evidence-driven

4. MODELS FOR SECURITY SPENDING
• Spend only what you need to until the next breach
• Keep spending until you run out of budget
• Have an unlimited budget

5. EXPENSE IN DEPTH (RICK
HOLLAND)
• Security is a patchwork
quilt, and you keep buying
things to layer over the
gaps
• Leads to overspending in
some areas and
underspending in others
• Overloading systems

6. EXPENSE IN DEPTH
• Dueling agents
• Prioritizing network
decisions
• Cognitive and effort
overload on your
personnel every time you
add something new

7. “
”
I’M A NEW CISO. IT’S MY FIRST DAY ON THE JOB IN
AN ORGANIZATION THAT HAS NEVER DONE
SECURITY BEFORE. WHAT SHOULD I BUY?
The Real Cost of Security 451 Research, 2013

8. EVEN THE EXPERTS DON’T KNOW
• As few as 4 different technologies and as many as 31
• Everyone said “it depends,” including the vendors
¯_(ツ)_/¯

9. EVEN THE EXPERTS DON’T KNOW
•The minimum baselines pretty much matched
up to PCI, and included both firewalls and AV
•Budget could be off by as much as a factor of 4
•There’s still no guarantee you won’t get
breached

10. CAN WE DO BETTER?

11. CYBER DEFENSE MATRIX
SOUNIL YU, [LARGE US FINANCIAL]
Devices
Applications
Network
Data
People
Degree of
Dependence
Identify Protect Detect Respond Recover
Technology People
Process

12. LEFT AND RIGHT OF “BOOM”
Devices
Applications
Network
Data
People
Degree of
Dependence
Identify Protect Detect Respond Recover
Technology People
Process
Pre-Compromise
Post-Compromise

13. ENTERPRISE SECURITY MARKET SEGMENTS
13
Devices
Applications
Network
Data
People
Degree of
Dependence
Identify Protect Detect Respond Recover
Technology People
Process
IAM Endpoint Visibility and Control /
Endpoint Threat Detection & Response
Configuration
and Systems
Management
Data
Labeling
App Sec
(SAST, DAST,
IAST, RASP),
WAFs
Phishing
Simulations
DDoS Mitigation
Insider Threat /
Behavioral Analytics
Network
Security
(FW, IPS)
DRM
Data
Encryption,
DLP
IDS
Netflow
Full PCAP
AV, HIPS
Deep Web,
Brian Krebs,
FBI
Backup
Phishing
Awareness

14. MARKET SEGMENTS – OTHER
ENVIRONMENTS
14
Threat Actor Assets
Threat
Data
Intrusion
Deception
Malware
Sandboxes

15. MARKET SEGMENTS – OTHER
ENVIRONMENTS
15
Vendor Assets
Cloud Access
Security Brokers
Vendor
Risk
Assess-
ments
Customer Assets
Endpoint Fraud
Detection
Device
Finger-
printing
Device
Finger-
printing
Web Fraud
Detection
Employee Assets
BYOD
MAM
BYOD
MDM

16. See the rest of the slides at
https://www.rsaconference.com/events/us16/agenda/sessio
ns/2530/understanding-the-security-vendor-landscape-using
Or Google for “RSAC Sounil Yu” J

17. TRIMMING YOUR SECURITY
PORTFOLIO
• Why would you need to do that?
• Mergers and acquisitions leave redundant products
in place

18. TRIMMING YOUR SECURITY
PORTFOLIO
• Shelfware
(see Javvad Malik’s research at
https://www.rsaconference.com/writable/presentati
ons/file_upload/mash-t07a-security-shelfware-which-
products-gathering-dust-and-why.pdf
or just Google “Javvad Malik Shelfware”)

19. TRIMMING YOUR SECURITY
PORTFOLIO
• Improving performance
• Simplifying
• Better integration and communication
• Better price

20. BEFORE YOU CUT
TECHNOLOGY …
• Make sure you’re using it right
• Make sure you’re using it as fully
as possible
• Talk to the vendor about its
limitations and roadmap (or ask
peers or an analyst)

21. BEFORE YOU CUT TECHNOLOGY …
•Decide whether you need to replace it
•Is it a greater liability to keep it and not use it,
or not to have it at all?

22. BEFORE YOU CUT
PEOPLE …
• Know what
they’re
contributing both
in expertise and
workload
• Expertise includes
institutional
knowledge

23. BEFORE YOU CUT PEOPLE …
•Remember cognitive workload: just because
they have the time to squeeze in an extra task,
it doesn’t mean they can give it the attention
it needs
•Keep task priorities in mind – response mode
keeps staff from being proactive

24. EVALUATING EFFECTIVENESS AND RISK

25. EVALUATING EFFECTIVENESS AND
RISK
• Is it addressing a risk everyone can believe in?

26. CHEESEBURGER RISK MANAGEMENT
Sure, it might happen – but not for a long time

27. EVALUATING EFFECTIVENESS AND
RISK
•How does it address the
risk?
•Don’t say “it’s blocking
millions of attacks,”
because that makes Dave
Lewis really angry

28. EVALUATING EFFECTIVENESS AND
RISK
•What are you relying on technology to
do, versus what you’re relying on people
to do?
•Are you basing your security strategy on
the hope that people will change?

29. YOUR MANAGEMENT’S FAVORITE
METRICS
Time saved
Money saved
Performance
improvements /
availability

30. MATCHING MONEY WITH SECURITY
• Avoiding loss – but remember the probability
discussion
• Allowing revenue generators to do it faster
• Saving time, which is money

31. MATCHING MONEY WITH SECURITY
• Helping the business make better decisions in other
areas
• Providing a competitive advantage (but you’ll have
to prove it)
• Losses may or may not happen, but other
improvements will show themselves if you can
measure them

32. GETTING BREACHED JUST MIGHT
BE CHEAPER …
• Published research by Sasha Romanosky, RAND
Corporation (August 2016)
• “Most cyber events cost firms less than 0.4% of their
annual revenues”

33. GETTING BREACHED JUST MIGHT
BE CHEAPER …
• By contrast, US firms lost an estimated 0.9% of their
revenue to online fraud in 2013 (Cybersource 2013
Online Fraud Report)
(Which shows that breaches are being treated
separately from fraud, so whatever)

34. GETTING BREACHED JUST
MIGHT BE CHEAPER …
• Calculated that firms were
spending an average of 0.025% of
revenues on cybersecurity
• Half of cyber events cost a firm an
amount approximately equal to its
annual investment in IT security
(i.e. within ±$1 million of
investment).
Wait, what?

35. WHAT IF I TOLD YOU …
… that you may already be spending enough?

36. SPENDING IS NOT DOING
• You can be spending right, but doing it wrong
• You can be doing it right, but spending wrong

37. SOME KIND OF PYRAMID
Using security products
Understanding threats
Controlling changes
Knowing what you have
and what it’s doing

38. SUMMARY
• There are many ways to evaluate your portfolio
• There’s no ground truth
• Identify the risks you can believe in
• Find the evidence that you’re addressing those risks
• Remember: it’s in the way that you use it