SlideShare a Scribd company logo

Minding the Metacognitive Gap - BSides NOLA

•Download as PPTX, PDF•
•
C
chrissanders88

This document discusses improving cybersecurity investigations through increased metacognitive awareness. It presents research showing experts are more metacognitively aware than novices when investigating cyber incidents. The research mapped key phrases from investigation case studies to show experienced analysts rely more on rule-based reasoning and reflection. To close the expertise gap, the document recommends documenting investigation heuristics, framing the investigation process as goal-driven questioning, hypothesis generation, seeking answers, and focusing on metacognition. This will help train analysts to systematically investigate incidents and accelerate the effects of experience.

Technology
1 of 28
COPYRIGHT © 2016, FIREEYE, INC. ALL RIGHTS RESERVED.
Chris Sanders (@chrissanders88)  BBQ Pit Master  FireEye/Mandiant  Former DoD & InGuardian  Founder, Rural Tech Fund  Author  PhD Researcher Copyright © 2016 Chris Sanders
Disclaimer Copyright © 2016 Chris Sanders I’m going to talk about matters of the brain, not just the normal tech stuff. My research for this presentation involved consultation with psychologists. I, however, am not one, ….yet.
Learning Objectives  Increase awareness of:  Metacognitive gap  Investigation process  So you can:  Become a better analyst  Approach investigations in a more systematic way  Get better at training new analysts  Accelerate the effects of experience  Appreciate the value of teaching and learning Copyright © 2016 Chris Sanders
The Metacognitive Gap Copyright © 2016 Chris Sanders
Perception vs. Reality Copyright © 2016 Chris Sanders  Perception  A way of regarding, understanding, or interpreting something.  Reality  The state of things as they actually exist. Perception RealityLearning
How do we do it? Copyright © 2016 Chris Sanders  How did you learn to catch bad guys?  Experimentation  Observation / OJT  Mentorship  KSU SOC Anthropological Study:  “SOC analysts often perform sophisticated investigations where the process required to connect the dots is unclear even to themselves.”
Metacognition  Thinking about thinking  “Why did I do this?”  Understanding your own thought process  Relationship between metacognitive awareness and performance.  Two Components:  Knowledge of Cognition (Understand It)  Regulation of Cognition (Apply It) Copyright © 2016 Chris Sanders
Mapping the Investigation Process Copyright © 2016 Chris Sanders
Experiment Design  Research Questions:  Are experts more metacognitively aware?  What separates novice and expert analysts?  Sample:  Novice and expert analysts  Methodology:  30 case studies  Stimulated recall interviews  Focus on individual investigations of varying types  Perform key phrase analysis Copyright © 2016 Chris Sanders
Key Phrase Mapping Copyright © 2016 Chris Sanders Intuition Experimentation Restructuring Imagination Incubation Metacognition Evaluation Goal Setting Making Plans Reflection Analytically Viewing Data Rule-Based Reasoning Considering Alternatives  Dual Process Theory  Intuition: Implicit, unconscious, fast  Reflection: Explicit, controlled, slow
Results Copyright © 2016 Chris Sanders Novices Experts Intuition Metacognition Reflection
Findings Copyright © 2016 Chris Sanders 1. Experienced analysts rely on rule-based reasoning to a much larger extent. 2. Experienced analysts are more metacognitively aware than novice analysts.
Closing the Gap Copyright © 2016 Chris Sanders  Novice: “How do I do this job?”  Expert: “Here, watch me.”  Expert: “Study this way of thinking. Then, come try it for yourself.” Goal Setting Making Plans Evaluation How can we train analysts to be more metacognitively aware, and provide them with the tools to apply that knowledge?
Rule-Based Reasoning Copyright © 2016 Chris Sanders
Rule-Based Reasoning Copyright © 2016 Chris Sanders  Humans think in if- then-else statements  Rules are heuristics  Shortcuts for solving problems  Derived from experience
Investigation Heuristics  If the process name is made to look like a legitimate system process but isn’t  Then it’s probably malware  If the domain has a bunch of random characters  Then it might have been created by a DGA  Else it’s just a coincedence  If the host is beaconing externally  Then it might be command and control  Else it’s a normal service I should remember for next time Copyright © 2016 Chris Sanders
Documenting Heuristics Copyright © 2016 Chris Sanders  We need an industry wide effort to document these…  If - Then - Else Format  Store in narrative and structured format  Use estimative language  Bonus: You can use these in IR playbooks
Metacognition and the Investigation Process Copyright © 2016 Chris Sanders
The Investigation Process Copyright © 2016 Chris Sanders “An investigation is the systematic inquiry and examination of evidence and observations in an effort to gain an accurate perception of whether an incident has occurred, and to what extent.” Question Hypothesi s Answer Observatio n Conclusion
Goal-Driven Questioning  You should be able to articulate what question you’re trying to answer at any given time.  Focus questioning around uncovering relationships  Questioning is driven by rule-based reasoning  Experience really shines here due to a larger library of Question HypothesisAnswer Copyright © 2016 Chris Sanders
Hypothesis Generation  You already do this, but it’s a passive process.  Expose and Attack Bias  Form an educated guess about the answer to your questions  Consider your “Because” statement  I believe X because Y Copyright © 2016 Chris Sanders Question HypothesisAnswer
Seeking Answers  Key processes:  Finding and Filtering Data  Performing open source intel research  Reviewing evidence  Uncovering additional questions  Hypothesis validation/invalidation Copyright © 2016 Chris Sanders Question HypothesisAnswer
Investigation Scenario 1 Question • Was this done maliciously? Copyright © 2016 Chris Sanders Discovery • SIEM Alert User account added to domain admin group Hypothesis • No – Normal admin activity Answer • Yes Question • What did the user account do afterwards? Hypothesis • Normal admin activities Answer • Accessed mail server and mounted exec staff mailboxes
Investigation Scenario 2 Question • Did the host get infected? Copyright © 2016 Chris Sanders Discovery • IDS Alert Angler EK Landing Page Hypothesis • Yes Answer • No – exploitation failed Question • What type of payload was downloaded? Hypothesis • Flash exploit due to SWF file alert evidence Answer • Hypothesis Confirmed Question • Is a vulnerable version of flash installed? Hypothesis • It’s Flash, so probably Answer • No – Flash is not installed
Further Research Copyright © 2016 Chris Sanders  More case studies  Supporting whitepaper + dissertation  Further experimentation in identified areas  Practical applications  Teaching case studies
Action Items Copyright © 2016 Chris Sanders  Identify and document your rules/heuristics  Start framing through the investigative process  Use the process as a teaching tool  Think about thinking – applied thought has power  Try to teach this stuff to someone
Thank You! Web: http://www.chrissanders.org E-Mail: chris@chrissanders.org Twitter: @chrissanders88 Copyright © 2016 Chris Sanders

Recommended

Developing Analytic Technique and Defeating Cognitive Bias in Security
Developing Analytic Technique and Defeating Cognitive Bias in SecurityDeveloping Analytic Technique and Defeating Cognitive Bias in Security
Developing Analytic Technique and Defeating Cognitive Bias in Security
chrissanders88
 
BSides Augusta 2015 - Building a Better Analyst Using Cognitive Psychology
BSides Augusta 2015 - Building a Better Analyst Using Cognitive PsychologyBSides Augusta 2015 - Building a Better Analyst Using Cognitive Psychology
BSides Augusta 2015 - Building a Better Analyst Using Cognitive Psychology
chrissanders88
 
Threats that Matter - Murray State University 2017
Threats that Matter - Murray State University 2017Threats that Matter - Murray State University 2017
Threats that Matter - Murray State University 2017
chrissanders88
 
Art into Science 2017 - Investigation Theory: A Cognitive Approach
Art into Science 2017 - Investigation Theory: A Cognitive ApproachArt into Science 2017 - Investigation Theory: A Cognitive Approach
Art into Science 2017 - Investigation Theory: A Cognitive Approach
chrissanders88
 
See Clearly and Respond Quickly from the Network to the Endpoint
See Clearly and Respond Quickly from the Network to the EndpointSee Clearly and Respond Quickly from the Network to the Endpoint
See Clearly and Respond Quickly from the Network to the Endpoint
ProtectWise
 
Abstract Tools for Effective Threat Hunting
Abstract Tools for Effective Threat HuntingAbstract Tools for Effective Threat Hunting
Abstract Tools for Effective Threat Hunting
chrissanders88
 
Crowdsourced Probability Estimates: A Field Guide (FAIR Institute)
Crowdsourced Probability Estimates: A Field Guide (FAIR Institute)Crowdsourced Probability Estimates: A Field Guide (FAIR Institute)
Crowdsourced Probability Estimates: A Field Guide (FAIR Institute)
Tony Martin-Vegue
 
Enabling effective hunt teaming and incident response
Enabling effective hunt teaming and incident responseEnabling effective hunt teaming and incident response
Enabling effective hunt teaming and incident response
jeffmcjunkin
 

Recommended

Developing Analytic Technique and Defeating Cognitive Bias in Security
Developing Analytic Technique and Defeating Cognitive Bias in SecurityDeveloping Analytic Technique and Defeating Cognitive Bias in Security
Developing Analytic Technique and Defeating Cognitive Bias in Security
chrissanders88
 
BSides Augusta 2015 - Building a Better Analyst Using Cognitive Psychology
BSides Augusta 2015 - Building a Better Analyst Using Cognitive PsychologyBSides Augusta 2015 - Building a Better Analyst Using Cognitive Psychology
BSides Augusta 2015 - Building a Better Analyst Using Cognitive Psychology
chrissanders88
 
Threats that Matter - Murray State University 2017
Threats that Matter - Murray State University 2017Threats that Matter - Murray State University 2017
Threats that Matter - Murray State University 2017
chrissanders88
 
Art into Science 2017 - Investigation Theory: A Cognitive Approach
Art into Science 2017 - Investigation Theory: A Cognitive ApproachArt into Science 2017 - Investigation Theory: A Cognitive Approach
Art into Science 2017 - Investigation Theory: A Cognitive Approach
chrissanders88
 
See Clearly and Respond Quickly from the Network to the Endpoint
See Clearly and Respond Quickly from the Network to the EndpointSee Clearly and Respond Quickly from the Network to the Endpoint
See Clearly and Respond Quickly from the Network to the Endpoint
ProtectWise
 
Abstract Tools for Effective Threat Hunting
Abstract Tools for Effective Threat HuntingAbstract Tools for Effective Threat Hunting
Abstract Tools for Effective Threat Hunting
chrissanders88
 
Crowdsourced Probability Estimates: A Field Guide (FAIR Institute)
Crowdsourced Probability Estimates: A Field Guide (FAIR Institute)Crowdsourced Probability Estimates: A Field Guide (FAIR Institute)
Crowdsourced Probability Estimates: A Field Guide (FAIR Institute)
Tony Martin-Vegue
 
Enabling effective hunt teaming and incident response
Enabling effective hunt teaming and incident responseEnabling effective hunt teaming and incident response
Enabling effective hunt teaming and incident response
jeffmcjunkin
 
Estimations in Project Management
Estimations in Project ManagementEstimations in Project Management
Estimations in Project Management
Intaver Insititute
 
Crowdsourced Probability Estimates: A Field Guide
Crowdsourced Probability Estimates: A Field GuideCrowdsourced Probability Estimates: A Field Guide
Crowdsourced Probability Estimates: A Field Guide
Tony Martin-Vegue
 
Estimates in Project Management
Estimates in Project ManagementEstimates in Project Management
Estimates in Project Management
Intaver Insititute
 
Threat Hunting with Splunk Hands-on
Threat Hunting with Splunk Hands-onThreat Hunting with Splunk Hands-on
Threat Hunting with Splunk Hands-on
Splunk
 
Threat Hunting with Splunk
Threat Hunting with SplunkThreat Hunting with Splunk
Threat Hunting with Splunk
Splunk
 
I am my worst enemy — A first person look at Insider Threat
I am my worst enemy — A first person look at Insider ThreatI am my worst enemy — A first person look at Insider Threat
I am my worst enemy — A first person look at Insider Threat
Ahmed Masud
 
Resistance Isn't Futile: A Practical Approach to Threat Modeling
Resistance Isn't Futile: A Practical Approach to Threat ModelingResistance Isn't Futile: A Practical Approach to Threat Modeling
Resistance Isn't Futile: A Practical Approach to Threat Modeling
Katie Nickels
 
Threat Hunting with Splunk
Threat Hunting with SplunkThreat Hunting with Splunk
Threat Hunting with Splunk
Splunk
 
What is the story with agile data keynote agile 2018 (Magennis)
What is the story with agile data keynote agile 2018 (Magennis)What is the story with agile data keynote agile 2018 (Magennis)
What is the story with agile data keynote agile 2018 (Magennis)
Troy Magennis
 
Threat hunting workshop
Threat hunting workshopThreat hunting workshop
Threat hunting workshop
Megan Shippy
 
NTXISSACSC4 - Hacking Performance Management, the Blue Green Game
NTXISSACSC4 - Hacking Performance Management, the Blue Green GameNTXISSACSC4 - Hacking Performance Management, the Blue Green Game
NTXISSACSC4 - Hacking Performance Management, the Blue Green Game
North Texas Chapter of the ISSA
 
From Theory to Practice: How My ATTACK Perspectives Have Changed
From Theory to Practice: How My ATTACK Perspectives Have ChangedFrom Theory to Practice: How My ATTACK Perspectives Have Changed
From Theory to Practice: How My ATTACK Perspectives Have Changed
MITRE - ATT&CKcon
 
Data Science at Intersection of Security and Privacy
Data Science at Intersection of Security and PrivacyData Science at Intersection of Security and Privacy
Data Science at Intersection of Security and Privacy
Tarun Chopra
 
Janitor vs cleaner
Janitor vs cleanerJanitor vs cleaner
Janitor vs cleaner
John Stauffacher
 
Scout: A Contactless Active Vulnerability Tool - Dissertation, a year long pr...
Scout: A Contactless Active Vulnerability Tool - Dissertation, a year long pr...Scout: A Contactless Active Vulnerability Tool - Dissertation, a year long pr...
Scout: A Contactless Active Vulnerability Tool - Dissertation, a year long pr...
Jamie O'Hare
 
IT Security Myths
IT Security MythsIT Security Myths
IT Security Myths
n|u - The Open Security Community
 
The power of regular reviews
The power of regular reviewsThe power of regular reviews
The power of regular reviews
Ebere Ikerionwu
 
How to Spend Your Cloud Security Dollar
How to Spend Your Cloud Security DollarHow to Spend Your Cloud Security Dollar
How to Spend Your Cloud Security Dollar
Hostway|HOSTING
 
Password best practices and the last pass hack
Password best practices and the last pass hackPassword best practices and the last pass hack
Password best practices and the last pass hack
Kevin OBrien
 
Applied Detection and Analysis with Flow Data - SO Con 2014
Applied Detection and Analysis with Flow Data - SO Con 2014Applied Detection and Analysis with Flow Data - SO Con 2014
Applied Detection and Analysis with Flow Data - SO Con 2014
chrissanders88
 
CISSA Lightning Talk - Building a Malware Analysis Lab on a Budget
CISSA Lightning Talk - Building a Malware Analysis Lab on a BudgetCISSA Lightning Talk - Building a Malware Analysis Lab on a Budget
CISSA Lightning Talk - Building a Malware Analysis Lab on a Budget
chrissanders88
 
BSA2016 - Honeypots for Network Security Monitoring
BSA2016 - Honeypots for Network Security MonitoringBSA2016 - Honeypots for Network Security Monitoring
BSA2016 - Honeypots for Network Security Monitoring
chrissanders88
 

More Related Content

What's hot

Estimations in Project Management
Estimations in Project ManagementEstimations in Project Management
Estimations in Project Management
Intaver Insititute
 
Crowdsourced Probability Estimates: A Field Guide
Crowdsourced Probability Estimates: A Field GuideCrowdsourced Probability Estimates: A Field Guide
Crowdsourced Probability Estimates: A Field Guide
Tony Martin-Vegue
 
Resistance Isn't Futile: A Practical Approach to Threat Modeling
Resistance Isn't Futile: A Practical Approach to Threat ModelingResistance Isn't Futile: A Practical Approach to Threat Modeling
Resistance Isn't Futile: A Practical Approach to Threat Modeling
Katie Nickels
 
Threat Hunting with Splunk
Threat Hunting with SplunkThreat Hunting with Splunk
Threat Hunting with Splunk
Splunk
 
NTXISSACSC4 - Hacking Performance Management, the Blue Green Game
NTXISSACSC4 - Hacking Performance Management, the Blue Green GameNTXISSACSC4 - Hacking Performance Management, the Blue Green Game
NTXISSACSC4 - Hacking Performance Management, the Blue Green Game
North Texas Chapter of the ISSA
 
From Theory to Practice: How My ATTACK Perspectives Have Changed
From Theory to Practice: How My ATTACK Perspectives Have ChangedFrom Theory to Practice: How My ATTACK Perspectives Have Changed
From Theory to Practice: How My ATTACK Perspectives Have Changed
MITRE - ATT&CKcon
 
Data Science at Intersection of Security and Privacy
Data Science at Intersection of Security and PrivacyData Science at Intersection of Security and Privacy
Data Science at Intersection of Security and Privacy
Tarun Chopra
 

What's hot (19)

Estimations in Project Management
Estimations in Project ManagementEstimations in Project Management
Estimations in Project Management
 
Crowdsourced Probability Estimates: A Field Guide
Crowdsourced Probability Estimates: A Field GuideCrowdsourced Probability Estimates: A Field Guide
Crowdsourced Probability Estimates: A Field Guide
 
Estimates in Project Management
Estimates in Project ManagementEstimates in Project Management
Estimates in Project Management
 
Threat Hunting with Splunk Hands-on
Threat Hunting with Splunk Hands-onThreat Hunting with Splunk Hands-on
Threat Hunting with Splunk Hands-on
 
Threat Hunting with Splunk
Threat Hunting with SplunkThreat Hunting with Splunk
Threat Hunting with Splunk
 
I am my worst enemy — A first person look at Insider Threat
I am my worst enemy — A first person look at Insider ThreatI am my worst enemy — A first person look at Insider Threat
I am my worst enemy — A first person look at Insider Threat
 
Resistance Isn't Futile: A Practical Approach to Threat Modeling
Resistance Isn't Futile: A Practical Approach to Threat ModelingResistance Isn't Futile: A Practical Approach to Threat Modeling
Resistance Isn't Futile: A Practical Approach to Threat Modeling
 
Threat Hunting with Splunk
Threat Hunting with SplunkThreat Hunting with Splunk
Threat Hunting with Splunk
 
What is the story with agile data keynote agile 2018 (Magennis)
What is the story with agile data keynote agile 2018 (Magennis)What is the story with agile data keynote agile 2018 (Magennis)
What is the story with agile data keynote agile 2018 (Magennis)
 
Threat hunting workshop
Threat hunting workshopThreat hunting workshop
Threat hunting workshop
 
NTXISSACSC4 - Hacking Performance Management, the Blue Green Game
NTXISSACSC4 - Hacking Performance Management, the Blue Green GameNTXISSACSC4 - Hacking Performance Management, the Blue Green Game
NTXISSACSC4 - Hacking Performance Management, the Blue Green Game
 
From Theory to Practice: How My ATTACK Perspectives Have Changed
From Theory to Practice: How My ATTACK Perspectives Have ChangedFrom Theory to Practice: How My ATTACK Perspectives Have Changed
From Theory to Practice: How My ATTACK Perspectives Have Changed
 
Data Science at Intersection of Security and Privacy
Data Science at Intersection of Security and PrivacyData Science at Intersection of Security and Privacy
Data Science at Intersection of Security and Privacy
 
Janitor vs cleaner
Janitor vs cleanerJanitor vs cleaner
Janitor vs cleaner
 
Scout: A Contactless Active Vulnerability Tool - Dissertation, a year long pr...
Scout: A Contactless Active Vulnerability Tool - Dissertation, a year long pr...Scout: A Contactless Active Vulnerability Tool - Dissertation, a year long pr...
Scout: A Contactless Active Vulnerability Tool - Dissertation, a year long pr...
 
IT Security Myths
IT Security MythsIT Security Myths
IT Security Myths
 
The power of regular reviews
The power of regular reviewsThe power of regular reviews
The power of regular reviews
 
How to Spend Your Cloud Security Dollar
How to Spend Your Cloud Security DollarHow to Spend Your Cloud Security Dollar
How to Spend Your Cloud Security Dollar
 
Password best practices and the last pass hack
Password best practices and the last pass hackPassword best practices and the last pass hack
Password best practices and the last pass hack
 

Viewers also liked

Viewers also liked (15)

Applied Detection and Analysis with Flow Data - SO Con 2014
Applied Detection and Analysis with Flow Data - SO Con 2014Applied Detection and Analysis with Flow Data - SO Con 2014
Applied Detection and Analysis with Flow Data - SO Con 2014
 
CISSA Lightning Talk - Building a Malware Analysis Lab on a Budget
CISSA Lightning Talk - Building a Malware Analysis Lab on a BudgetCISSA Lightning Talk - Building a Malware Analysis Lab on a Budget
CISSA Lightning Talk - Building a Malware Analysis Lab on a Budget
 
BSA2016 - Honeypots for Network Security Monitoring
BSA2016 - Honeypots for Network Security MonitoringBSA2016 - Honeypots for Network Security Monitoring
BSA2016 - Honeypots for Network Security Monitoring
 
Proper logging can catch breaches like retail PoS
Proper logging can catch breaches like retail PoSProper logging can catch breaches like retail PoS
Proper logging can catch breaches like retail PoS
 
Presentation
PresentationPresentation
Presentation
 
Физика и техника
Физика и техникаФизика и техника
Физика и техника
 
8decologicalrelationships 120607150803-phpapp01
8decologicalrelationships 120607150803-phpapp018decologicalrelationships 120607150803-phpapp01
8decologicalrelationships 120607150803-phpapp01
 
Propo viacboj 2015
Propo viacboj 2015Propo viacboj 2015
Propo viacboj 2015
 
Explaining No-Fuss Advice In Legal Background Search
Explaining No-Fuss Advice In Legal Background SearchExplaining No-Fuss Advice In Legal Background Search
Explaining No-Fuss Advice In Legal Background Search
 
Computer viruses. - Free Online Library
Computer viruses. - Free Online LibraryComputer viruses. - Free Online Library
Computer viruses. - Free Online Library
 
新生兒準備物品5大類(食衣住行育樂)
新生兒準備物品5大類(食衣住行育樂)新生兒準備物品5大類(食衣住行育樂)
新生兒準備物品5大類(食衣住行育樂)
 
Presentation1
Presentation1Presentation1
Presentation1
 
Project 1
Project 1Project 1
Project 1
 
Review jurnal kualitatif
Review jurnal kualitatifReview jurnal kualitatif
Review jurnal kualitatif
 
Character costumes
Character costumesCharacter costumes
Character costumes
 

Similar to Minding the Metacognitive Gap - BSides NOLA

Research Methods - an overview
Research Methods - an overviewResearch Methods - an overview
Research Methods - an overview
Phil302
 
Stephanie Cooper - Genuine Curiosity - Conversations for Change
Stephanie Cooper - Genuine Curiosity - Conversations for ChangeStephanie Cooper - Genuine Curiosity - Conversations for Change
Stephanie Cooper - Genuine Curiosity - Conversations for Change
AgileNZ Conference
 
deLaplante, K. [Kevin deLaplante]. (2013, January 29). What is an .docx
deLaplante, K. [Kevin deLaplante]. (2013, January 29). What is an .docxdeLaplante, K. [Kevin deLaplante]. (2013, January 29). What is an .docx
deLaplante, K. [Kevin deLaplante]. (2013, January 29). What is an .docx
simonithomas47935
 

Similar to Minding the Metacognitive Gap - BSides NOLA (20)

Campus Session 2
Campus Session 2Campus Session 2
Campus Session 2
 
Analytical skills
Analytical skillsAnalytical skills
Analytical skills
 
Lecture 5
Lecture 5Lecture 5
Lecture 5
 
Lecture 5
Lecture 5Lecture 5
Lecture 5
 
Research Proposal Writing: Methodology in Research Proposal
Research Proposal Writing: Methodology in Research ProposalResearch Proposal Writing: Methodology in Research Proposal
Research Proposal Writing: Methodology in Research Proposal
 
Managing Creativity
Managing CreativityManaging Creativity
Managing Creativity
 
Abi Bennetts - Brighton SEO 2021 - Psychology principles to power digital PR ...
Abi Bennetts - Brighton SEO 2021 - Psychology principles to power digital PR ...Abi Bennetts - Brighton SEO 2021 - Psychology principles to power digital PR ...
Abi Bennetts - Brighton SEO 2021 - Psychology principles to power digital PR ...
 
UX Process Improved: Integrating User Insight
UX Process Improved: Integrating User InsightUX Process Improved: Integrating User Insight
UX Process Improved: Integrating User Insight
 
Research Methods - an overview
Research Methods - an overviewResearch Methods - an overview
Research Methods - an overview
 
Stephanie Cooper - Genuine Curiosity - Conversations for Change
Stephanie Cooper - Genuine Curiosity - Conversations for ChangeStephanie Cooper - Genuine Curiosity - Conversations for Change
Stephanie Cooper - Genuine Curiosity - Conversations for Change
 
L1 1.1 10 things you need to know before doing your own qualitative research
L1 1.1 10 things you need to know before doing your own qualitative researchL1 1.1 10 things you need to know before doing your own qualitative research
L1 1.1 10 things you need to know before doing your own qualitative research
 
A Template for Research Presentation (Google Presentation Format)
A Template for Research Presentation (Google Presentation Format)A Template for Research Presentation (Google Presentation Format)
A Template for Research Presentation (Google Presentation Format)
 
deLaplante, K. [Kevin deLaplante]. (2013, January 29). What is an .docx
deLaplante, K. [Kevin deLaplante]. (2013, January 29). What is an .docxdeLaplante, K. [Kevin deLaplante]. (2013, January 29). What is an .docx
deLaplante, K. [Kevin deLaplante]. (2013, January 29). What is an .docx
 
Bath University taster event: Evidence-Based Decision Making
Bath University taster event: Evidence-Based Decision MakingBath University taster event: Evidence-Based Decision Making
Bath University taster event: Evidence-Based Decision Making
 
Somatic Experiencing Research Funding Online Discussion - 7/1/13
Somatic Experiencing Research Funding Online Discussion - 7/1/13 Somatic Experiencing Research Funding Online Discussion - 7/1/13
Somatic Experiencing Research Funding Online Discussion - 7/1/13
 
100 Funny Argumentative Essay Topic Ideas - Owlca
100 Funny Argumentative Essay Topic Ideas - Owlca100 Funny Argumentative Essay Topic Ideas - Owlca
100 Funny Argumentative Essay Topic Ideas - Owlca
 
Learning to Learn by Erika Andersen
Learning to Learn by Erika AndersenLearning to Learn by Erika Andersen
Learning to Learn by Erika Andersen
 
Using Problem Solving Skills To Get A Job
Using Problem Solving Skills To Get A JobUsing Problem Solving Skills To Get A Job
Using Problem Solving Skills To Get A Job
 
Communities of Trust
Communities of TrustCommunities of Trust
Communities of Trust
 
Exploring the UX Research Toolbox
Exploring the UX Research ToolboxExploring the UX Research Toolbox
Exploring the UX Research Toolbox
 

Recently uploaded

A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
Igalia
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
Overkill Security
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Drew Madelung
 

Recently uploaded (20)

A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu SubbuApidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
A Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source MilvusA Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source Milvus
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 

Minding the Metacognitive Gap - BSides NOLA

  • 1. COPYRIGHT © 2016, FIREEYE, INC. ALL RIGHTS RESERVED.
  • 2. Chris Sanders (@chrissanders88)  BBQ Pit Master  FireEye/Mandiant  Former DoD & InGuardian  Founder, Rural Tech Fund  Author  PhD Researcher Copyright © 2016 Chris Sanders
  • 3. Disclaimer Copyright © 2016 Chris Sanders I’m going to talk about matters of the brain, not just the normal tech stuff. My research for this presentation involved consultation with psychologists. I, however, am not one, ….yet.
  • 4. Learning Objectives  Increase awareness of:  Metacognitive gap  Investigation process  So you can:  Become a better analyst  Approach investigations in a more systematic way  Get better at training new analysts  Accelerate the effects of experience  Appreciate the value of teaching and learning Copyright © 2016 Chris Sanders
  • 5. The Metacognitive Gap Copyright © 2016 Chris Sanders
  • 6. Perception vs. Reality Copyright © 2016 Chris Sanders  Perception  A way of regarding, understanding, or interpreting something.  Reality  The state of things as they actually exist. Perception RealityLearning
  • 7. How do we do it? Copyright © 2016 Chris Sanders  How did you learn to catch bad guys?  Experimentation  Observation / OJT  Mentorship  KSU SOC Anthropological Study:  “SOC analysts often perform sophisticated investigations where the process required to connect the dots is unclear even to themselves.”
  • 8. Metacognition  Thinking about thinking  “Why did I do this?”  Understanding your own thought process  Relationship between metacognitive awareness and performance.  Two Components:  Knowledge of Cognition (Understand It)  Regulation of Cognition (Apply It) Copyright © 2016 Chris Sanders
  • 10. Experiment Design  Research Questions:  Are experts more metacognitively aware?  What separates novice and expert analysts?  Sample:  Novice and expert analysts  Methodology:  30 case studies  Stimulated recall interviews  Focus on individual investigations of varying types  Perform key phrase analysis Copyright © 2016 Chris Sanders
  • 11. Key Phrase Mapping Copyright © 2016 Chris Sanders Intuition Experimentation Restructuring Imagination Incubation Metacognition Evaluation Goal Setting Making Plans Reflection Analytically Viewing Data Rule-Based Reasoning Considering Alternatives  Dual Process Theory  Intuition: Implicit, unconscious, fast  Reflection: Explicit, controlled, slow
  • 12. Results Copyright © 2016 Chris Sanders Novices Experts Intuition Metacognition Reflection
  • 13. Findings Copyright © 2016 Chris Sanders 1. Experienced analysts rely on rule-based reasoning to a much larger extent. 2. Experienced analysts are more metacognitively aware than novice analysts.
  • 14. Closing the Gap Copyright © 2016 Chris Sanders  Novice: “How do I do this job?”  Expert: “Here, watch me.”  Expert: “Study this way of thinking. Then, come try it for yourself.” Goal Setting Making Plans Evaluation How can we train analysts to be more metacognitively aware, and provide them with the tools to apply that knowledge?
  • 15. Rule-Based Reasoning Copyright © 2016 Chris Sanders
  • 16. Rule-Based Reasoning Copyright © 2016 Chris Sanders  Humans think in if- then-else statements  Rules are heuristics  Shortcuts for solving problems  Derived from experience
  • 17. Investigation Heuristics  If the process name is made to look like a legitimate system process but isn’t  Then it’s probably malware  If the domain has a bunch of random characters  Then it might have been created by a DGA  Else it’s just a coincedence  If the host is beaconing externally  Then it might be command and control  Else it’s a normal service I should remember for next time Copyright © 2016 Chris Sanders
  • 18. Documenting Heuristics Copyright © 2016 Chris Sanders  We need an industry wide effort to document these…  If - Then - Else Format  Store in narrative and structured format  Use estimative language  Bonus: You can use these in IR playbooks
  • 19. Metacognition and the Investigation Process Copyright © 2016 Chris Sanders
  • 20. The Investigation Process Copyright © 2016 Chris Sanders “An investigation is the systematic inquiry and examination of evidence and observations in an effort to gain an accurate perception of whether an incident has occurred, and to what extent.” Question Hypothesi s Answer Observatio n Conclusion
  • 21. Goal-Driven Questioning  You should be able to articulate what question you’re trying to answer at any given time.  Focus questioning around uncovering relationships  Questioning is driven by rule-based reasoning  Experience really shines here due to a larger library of Question HypothesisAnswer Copyright © 2016 Chris Sanders
  • 22. Hypothesis Generation  You already do this, but it’s a passive process.  Expose and Attack Bias  Form an educated guess about the answer to your questions  Consider your “Because” statement  I believe X because Y Copyright © 2016 Chris Sanders Question HypothesisAnswer
  • 23. Seeking Answers  Key processes:  Finding and Filtering Data  Performing open source intel research  Reviewing evidence  Uncovering additional questions  Hypothesis validation/invalidation Copyright © 2016 Chris Sanders Question HypothesisAnswer
  • 24. Investigation Scenario 1 Question • Was this done maliciously? Copyright © 2016 Chris Sanders Discovery • SIEM Alert User account added to domain admin group Hypothesis • No – Normal admin activity Answer • Yes Question • What did the user account do afterwards? Hypothesis • Normal admin activities Answer • Accessed mail server and mounted exec staff mailboxes
  • 25. Investigation Scenario 2 Question • Did the host get infected? Copyright © 2016 Chris Sanders Discovery • IDS Alert Angler EK Landing Page Hypothesis • Yes Answer • No – exploitation failed Question • What type of payload was downloaded? Hypothesis • Flash exploit due to SWF file alert evidence Answer • Hypothesis Confirmed Question • Is a vulnerable version of flash installed? Hypothesis • It’s Flash, so probably Answer • No – Flash is not installed
  • 26. Further Research Copyright © 2016 Chris Sanders  More case studies  Supporting whitepaper + dissertation  Further experimentation in identified areas  Practical applications  Teaching case studies
  • 27. Action Items Copyright © 2016 Chris Sanders  Identify and document your rules/heuristics  Start framing through the investigative process  Use the process as a teaching tool  Think about thinking – applied thought has power  Try to teach this stuff to someone

Editor's Notes

  1. To understand the gap you have to understand perception
  2. START: Coin in the pocket example. END: Getting from peception to reality is cognition/learning.
  3. START: Who here does some type of investigation? THIS IS THE BIGGEST PROBLEM OUR INDUSTRY FACES – THIS IS THE GAP END: KSU is telling us we aren’t good at thinking about thinking.
  4. .
  5. START: As I went further down the rabbit hole, I started thinking about how we map the investigation process.
  6. Greater number of rule-based reasoning phrases. Also fairly obvious. Greater number of metacognition related phrases
  7. We don’t have a way to express what we do because it’s so loosley defined.
  8. Scientific Process Reflects how we actually think Way to think, teach, and learn Hunting applies