Security Metrics and KPIs Meaningful InfoSec Program Measurements

1. Measuring Success
Security Metrics and KPIs
Meaningful InfoSec Program Measurements
Harry Contreras - CISSP, Six Sigma
Phoenix, AZ
www.company.com

2. Why measuring your InfoSec Program matters
Topical overview of this presentation material
• What are metrics and measurements?
• Why use process control methodologies to measure security programs?
• What does this information tell us?
• What are the benefits?
• KPI’s and how they are derived
• How can Information Security Programs be effectively measured?
• Translating business risks to metrics
• How effective is your InfoSec program?
• Resources – The where and how to get additional information
• Summary
• Questions and Answers
Measurement - Best practice in management and control methodologies
Presentation to ISSA – Phoenix, AZ – April, 2009 1

3. Why measuring with metrics matter
Why measuring your InfoSec Program matters
Security is a process
• So why are we not using process control methodologies to measure and
advance our security programs and initiatives?
As security practitioners we should be measuring the value of information security
programs and demonstrating the continuing maturity of the organization.
Why we measure
• Integral to process controls and governance programs
• Integral to measuring deficit areas and where to focus for improvements
• Integral to overall information security program success
Someone once said… You don’t know what you don’t know…
Presentation to ISSA – Phoenix, AZ – April, 2009 2

4. Why measuring with metrics matter
Security Metrics as Established Industry Best Practice
One example from industry best practice for security controls
ISO17799/2005
4. Risk assessment and treatment
4.2 Treating security risks
d) cost of implementation and operation in relation to the risks being
reduced, and the remaining proportional to the organization’s
requirements and constraints
e) the need to balance the investment in implementation and operation of
controls against the harm likely to result from security failures
In the information security practice, the resulting actions of our control
processes is to continuously improve the Company’s business risk profile.
Presentation to ISSA – Phoenix, AZ – April, 2009 3

5. Why measuring with metrics matter
Why measuring your InfoSec Program matters
Measure to Manage
• Converting reporting inputs into meaningful outputs
• Compile data and develop representations of the information
Metrics must be meaningful to the company
• Every company is different
• There is no one metric to fit all
Develop a Security Scorecard
• Regular and consistent measurements from baseline numbers
• Require metrics and regular reporting from security service providers
• Performance assessment to service level objectives & agreements
Someone once said… You don’t know what you don’t know…
Presentation to ISSA – Phoenix, AZ – April, 2009 4

6. Assessing meaningful metrics to report
Strategic Metrics for Information Security - P. Lindstrom, Sept, 2008
Presentation to ISSA – Phoenix, AZ – April, 2009 5

7. Why measuring with metrics matter
Why measuring your InfoSec Program matters
Measure to Manage
• Metrics are measures used to indicate progress or achievement
• Measurements are a quantitative assessment of a circumstance
Metrics can be improved
• Measurements do not need to be as they can be observations of a state
• Primarily inputs and triggering events from an external condition
E.g. number of external scans against the company firewall,
External events that are reported by not influence by internal actions
Presentation to ISSA – Phoenix, AZ – April, 2009 6

8. Measurements and Metrics – Good and Bad
Controlled and Uncontrolled Events
Strategic Metrics for Information Security - P. Lindstrom, Sept, 2008
Presentation to ISSA – Phoenix, AZ – April, 2009 7

9. Why measuring with metrics matter
Why measuring your InfoSec Program matters
What metrics and measurements provide and how not to use them
• Provide basis for continual improvement
• Provide strategic intelligence for management
• Provide fiscal overview for aligning spend to company priorities and business
goals
Don’t measure everything
• Metric or Measurements – Holistic vs. minutiae
• Not for “reward and punishment”
- Participants will learn how to “game” the system
What does leadership want to know?
• What is our level of risk?
• How strong is our security program?
• Are we maintaining appropriate cost control?
Presentation to ISSA – Phoenix, AZ – April, 2009 8

10. Why measuring with metrics matter
Why measuring your InfoSec Program matters
Measure to Manage
• Converting reporting inputs into meaningful outputs
• Compile data and develop visual representations of the information
Reporting Actions
• Regular and consistent measurements from baseline numbers
• Require metrics and regular reporting from security service providers
• Performance assessment to service level objectives & agreements
Rationalization of metrics
• Align with industry recognized statistics to gauge your business risk profile
• E.g., CSI Annual Computer Crime and Security Report
Prioritization for Actions
• Budgeting for Capex and Opex
• Present factual representation of security state with measurements
Reassessment Actions
• Measurable difference in business security state can be identified
Presentation to ISSA – Phoenix, AZ – April, 2009 9

11. Security Program Information–Decision–Action Cycle
AKA, Plan, Do, Check and Act (PDCA)
Actions
Data Apply mitigating security
Monitor threat horizon, review new
controls or changes to
technologies, develop
services delivery portfolio
Effectiveness measurements.
Interpretation
Assess, R&D, review security metrics,
Benchmarking, ROSI analysis and
Key Performance Indicators
Continuous improvement through repeatable process controls
Presentation to ISSA – Phoenix, AZ – April, 2009 10

12. Key Performance Indicators - KPIs
When identifying KPIs to set as measurement targets, select
ones that you have control over and can improve.
Presentation to ISSA – Phoenix, AZ – April, 2009 11

13. Developing KPIs
Key Performance Indicators – 10 Critical Characteristics
KPIs reflect strategic value drivers
KPIs are defined by “executives”
KPIs cascade throughout an organization
KPIs are based on corporate standards
KPIs are based on valid data
KPIs must be easy to comprehend
KPIs are always relevant
KPIs provide context
KPIs empower users
KPIs lead to positive action
Key Performance Indicators are metrics,
but not all metrics are key performance indicators.
Presentation to ISSA – Phoenix, AZ – April, 2009 12

14. Thought Process Map
Thought Process Map for Security Metric Development
Process Identify possible Related to Voice of Customer Is it Is it
Start metrics targets What measure? Input quantifiable? repeatable?
Yes
Are metric Data extrapolation Key Performance
Related to
Data sources
Targets Processes Indicators
Cost dollars?
identified?
Viable? developed Derived
No
Measures
KPIs added to Interpret Quantifiable
Voice of Customer Based
Executive Metric Change End
Input Actions &
Dashboard results Results
Decisions
Continuous improvement through repeatable process controls
Presentation to ISSA – Phoenix, AZ – April, 2009 13

15. Why measuring with metrics matter
Measurement and Analysis - Examples
Assess the viability of your target measurements with meaningful criteria
Analysis Target What does Customer Source Quantifiable Repeatable Derived
it Measure Measures Cost
Mean time to patch Exposure On time Patching Yes Yes $$
application window (SLO) System
Content filtering event Effectiveness Cost SOC Yes $
counts
Percent of un-patched Risk index Patching Yes Yes
systems to asset inventory System
AV events detected and Effectiveness Reliability AV service Yes Yes $$
cleaned
Mean time to AV control file Exposure On time AV Service Yes Yes $
update window (SLO)
Average historical spend per Historical Yes No $$$$
InfoSec Incident records
IDS incident reporting rate IDS Yes $
system
SPAM messages Effectiveness Customer Service $$
suppressed Sat Records
Presentation to ISSA – Phoenix, AZ – April, 2009 14

16. Methods to derive Security Program Metrics
Strategic Metrics for Information Security - P. Lindstrom, Sept, 2008
Presentation to ISSA – Phoenix, AZ – April, 2009 15

17. Methods to derive Security Program Metrics
Strategic Metrics for Information Security - P. Lindstrom, Sept, 2008
Presentation to ISSA – Phoenix, AZ – April, 2009 16

18. Methods to derive Security Program Metrics
Strategic Metrics for Information Security - P. Lindstrom, Sept, 2008
Presentation to ISSA – Phoenix, AZ – April, 2009 17

19. Methods to derive Security Program Metrics
Strategic Metrics for Information Security - P. Lindstrom, Sept, 2008
Presentation to ISSA – Phoenix, AZ – April, 2009 18

20. What actions do you take with your metrics?
Why measuring your InfoSec Program matters
There are costs associated with controls that extend beyond the implementation
of those controls.
• How long will the control be effective?
• Is the cost of the control reasonable, relative to the value of the asset?
How can these numbers relate?
• Align with any in-Company compliance programs
• Align with other recognized industry statistics
• Annual industry published reports
• E.g. CSI’s Annual Computer Crime and Security Report
The company internal valuation process, each company’s approach is different.
• Process through Business Governance path
• Internal business financial valuation processes are different
• Your mileage will vary
Presentation to ISSA – Phoenix, AZ – April, 2009 19

21. Why measuring your InfoSec Program matters
Develop a Security Program Scorecard
• Company risk index (one of may options)
• IT Security metrics and KPIs
• Measure InfoSec program effectiveness
• Regular and consistent measurements from baseline numbers
• Require metrics and regular reporting from security service providers
• Performance assessment to service level objectives & agreements
• Represent this information in a visual form, perhaps an information security
dashboard for leadership to monitor
In today’s information security practice, consider the aspects of combining
reporting information in a “converged” security program for your company.
Presentation to ISSA – Phoenix, AZ – April, 2009 20

22. Why measuring your InfoSec Program matters
Identify – The Reality vs. The Perceived business state
Business risk profile development for measuring and reporting
• Converting reporting inputs into meaningful outputs
• Compile data and develop representations of the information
Derive an “overall” company risk index or set of indicators
• Conducting compliance measurements both internally and externally
• Deriving decision support and governance controls
• Performance assessment to service level objectives & agreements
In the information security practice, the resulting actions of our control
processes is to continuously improve the Company’s business risk profile.
Presentation to ISSA – Phoenix, AZ – April, 2009 21

23. Metric and Measurements vs. Business Value
What tips the scale in the assessment of business value?
Pros Cons
• Costs associated with metrics
• Provides business baseline
• Ongoing activity
• Aligns actions with results
• Staff overhead
• Insight for Governance decisions
• Many variables
• Visual indicators for:
• Information compilation
• Effectiveness measures
• Disparate recording instances
• Risk profile analysis
• Multiple inputs
• Cost analysis
• External influences
• Compliance profile
• Analysis paralysis
• Reality vs. perceived is revealed
• Visibility of poor performance
The overall importance of IT Metrics are the value to the business in representing
the state change associated with the measured activities (Good & Bad results)
Presentation to ISSA – Phoenix, AZ – April, 2009 22

24. What actions do you take with your metrics?
Why measuring your InfoSec Program matters
A repeatable process with consistent results
Analysis Actions
• Converting inputs into meaningful outputs
• Compile data and develop representations of the information
Reporting Actions
• Reports, reports reports…
• Feed into management dashboards
• Presentation to leadership
Rationalization Actions
• Risk impact assessment
• Process through Business Governance path
Prioritization Actions
• Budgeting for Capex and Opex
• Allocation of time and personnel for changes
Reassessment Actions
• The cycle of continuous improvement
Presentation to ISSA – Phoenix, AZ – April, 2009 23

25. Why measuring with metrics matter
Why measuring your InfoSec program matters
Who is watching…?
External Observers
IT Audit practices
Compliance assessment organizations
*Standard & Poor’s (S&P) Enterprise Risk Management (ERM) Analysis for
Credit Ratings of Non-Financial Companies
* Request for Comment (November, 2007)
S&P has proposed a rating criteria for an Enterprise Risk Management
assessment approach.
- How well, or even if companies are proactively and effectively managing their
business risks. Assessment of a Company’s approach and maturity in this
critical business area.
Presentation to ISSA – Phoenix, AZ – April, 2009 24

26. If only it was this easy…
Visualize your information security dashboard here…
Presentation to ISSA – Phoenix, AZ – April, 2009 25

27. Resources – Helpful slides (One of Two)
These important references will aid in developing a security metrics program
Information Week Analytics – Governance Vs. Success: Models and Metrics
December, 2008 http://informationweekanalytics.com/
Available to companies via the publication’s online hosting of this content.
Microsoft – Security Risk Management Guide v1.2
March 15, 2006 Microsoft Corporation. All rights reserved.
Download and On-line Locations for the Security Risk Management Guide
Specifically sections: Measuring Program Effectiveness, Conducting Decision Support
- Download Center: http://go.microsoft.com/fwlink/?linkid=32050
- TechNet online: http://go.microsoft.com/fwlink/?linkid=30794
ISO/IEC17799/2005 - Information Security Standard
- ISO/IEC 13335-3 Guidelines for the Management of IT Security
http://www.iso.org/iso/home.htm
Information Systems Security Association - (ISSA)
• The Use of ROI in Information Security – by Luther Martin (See Resources – ISSA Journal, Nov 2008)
• Security Metrics – Hype, reality and value demonstration – by Aurobindo Sundaram (ISSA Journal, May 2008)
• Ways to Determine or Prioritize Security Initiatives – by Matt Ege (ISSA Journal, Jan 2009)
• http://www.issa.org/ These are just a few of many additional resources to search in this information repository.
CSO Online – The Security Metrics Collection, October 27, 2008
Refer to the Security Leadership section for Metrics and Budget
http://www.csoonline.com/
Presentation to ISSA – Phoenix, AZ – April, 2009 26

28. Resources – Helpful slides (Two of Two)
These important references will aid in developing a security metrics program
SearchSecurity.com A TechTarget online publication
Refer to the Topics section for Information Security Management
http://www.searchsecurity.com/
SearchFinancialSecurity.com – A TechTarget online publication
• Strategic Metrics for Information Security at Financial Services Firms – P. Lindstrom, Sept, 2008
Refer to the Management Strategies section for additional information
http://searchfinancialsecurity.techtarget.com/
International Information Security Systems Certification Consortium - (ISC2)
• Why Security Metrics Must Replace Traditional Risk Analysis Methodologies – by Robert Hudock, Mar, 2008
Available to ISC2 registered members via the organization’s online hosting of this content.
www.ISC2.org Locate in the ISC2 Journal Archives
Security Metrics: Replacing Fear, Uncertainty and Doubt
Author, Andrew Jaquith – 336 Pages
© 2007, Addison-Wesley Professional Publications.
Metrics Management Toolkit
- Implementing Metrics Management Guide, Metrics spreadsheet, Project WBS, 125+ predefined templates
© 2008, Unified Compliance Framework Inc.
http://www.unifiedcompliance.com/ Located in the IT Impact Zones / UCF Toolkits offerings section.
Presentation to ISSA – Phoenix, AZ – April, 2009 27

29. On a final note…
FYI - For Information Security Professionals
At this year’s RSA Conference 2009 in San Francisco, CA
There will be six (6) separate presentations covering security
metrics, measuring security effectiveness and data driven
C-Level decision making approaches featured this year.
Presentation to ISSA – Phoenix, AZ – April, 2009 28

30. Summary
“There are risks and costs to a program of action.
But they are far less than the long-range risks and costs
of comfortable inaction.”
John F. Kennedy
With effective security measurements, risk identification,
assessments and mitigation approaches businesses can
benefit with the following results.
• Competitive advantage
• Security
• Efficiency
• Resilience
• Confidence
Presentation to ISSA – Phoenix, AZ – April, 2009 29

31. Measuring Success
Security Metrics and KPIs
Meaningful InfoSec Program Measurements
Harry Contreras - CISSP, Six Sigma
Phoenix, AZ
www.company.com