Automate or die! Rootedcon 2017

Automate or Die!
How to Survive to an Attack in the Cloud
March 3rd 2017
Toni de la Fuente (@ToniBlyx – blyx.com)
Lead Security Operations / Security Architect

Once upon a time…
September 2015

First Things First
*NIST Definition

Ubiquity
• Regions
• Availability Zones
• AWS:
• 16 regions (+2)
• 42 AZ (+4)
• Azure:
• 32 regions
• GCP:
• 6 regions (+8)
• 18 zones (+16)
*CDN locations not included

AWS Region
Amazon RDS MySQL
Master
Internet
gateway
Availability Zone 1 Availability Zone 2
Public subnet Public subnet
NAT gateway
EC2
Bastion
10.0.128.5
NAT gateway
EC2
Bastion
10.0.144.5
Alfresco One
Auto Scaling Group
Elastic Load
Balancing
Amazon RDS MySQL
Slave
S3 for
Shared Content
Store
10.0.0.0/16
10.0.128.0/20 10.0.144.0/20
10.0.0.0/19 10.0.32.0/19
Alfresco Index
Auto Scaling Group
Private SubnetPrivate Subnet
Alfresco Server Alfresco Server Alfresco ServerAlfresco Server
Index Server Index Server Index Server Index Server
* Immutable infrastructure

Shared Security Model / Responsibility Zones
IaaS
• Data
• Application
• Operating System
• Virtualization
• Infrastructure
• Physical
PaaS
• Data
• Application
• Virtualization
• Infrastructure
• Physical
SaaS
• Data
• Application
• Virtualization
• Infrastructure
• Physical

AWS manages the
security OF the Cloud
You
AWS foundation services
Compute Storage Database Networking
AWS global infrastructure
Regions
Availability zones
Edge locations
Client-side data encryption Server-side data Encryption Network traffic protection
Platform, applications, identity & access management
Operating system, network & firewall configuration
Customer applications & content
You define your controls
IN the Cloud
* Similar on other providers / subject to changes depending on the service or product
*

Challenges in Case of an Incident

Disadvantages and Challenges
Cloud Forensics and Operations
Ubiquity Enumeration
Legal jurisdiction
Elasticity Preservation of evidence
Data integrity
Data persistence (replication) Chain of custody
Evidence integrity
Multi-tenancy Data attribution
Chain of custody
Abstract Determine the best evidence
Preservation and visualization of evidence
Quantity of data and Big Data Systems that cannot be investigated or managed in a traditional manner
Knowledge Trained staff
Continuous evolution and new features almost daily
Providers Service level agreement / service level objectives
Relationship client-provider / transparency

Traditional vs Cloud Forensics
Processes Traditional Forensics Cloud Forensics
Identification Identification of an event or incident Multiple tools Few tools
Preservation Securitization and assessment of the scene Yes No
Documentation of the scene Yes No
Evidence collection: origin of the evidence Physical hardware Virtual hardware
Evidence collection: location of the evidence Crime scene Provider’s data center
Marking, packaging and transport Physical Digital through the Internet or physical media
Acquisition /
Extraction
Acquisition time Slow Fast
RAM acquisition Yes Dependant
Hash Slow Fast
Erased data recovery Possible Difficult
Metadata acquisition Yes Yes
Time stamp Precise Complex
Installation (action) of forensic software Expensive Cheap
Configuration and availability of forensic software Expensive Cheap
Transport Yes No
Analysis Analysis Slow Fast (potentially)
Presentation Documentation of evidence Acquired evidence Data from many sources
Declaration Common Difficult to explain to a judge

Storage Options
Type AWS Azure GCP
Objects S3 Object Storage
• Buckets
• 5TB max per object
• Encryption In-flight and at-rest
Azure Storage
• Blob storage
• 500TB limit per storage account
Google Cloud Storage
• Buckets
• 5TB max per object
SAN EBS (Volumes)
• Volume size: 1GB to 16TB (in 1GB
increments)
• Magnetic, SSD
• Encryption available
• Snapshots
Azure Block Storage
• Page blobs
• Volume size: 1GB to 1TB
• Standard (Magnetic), SSD premium
• Snapshots
• Encryption available
Google Block Storage
• Volume size: 1GB to 10TB
• Magnetic, SSD
• Snapshots
• Encryption by default
NAS Shared Storage (NFS)
• EFS
File Storage (CIFS) Single Node File Server + Others
Archive Glacier Azure Backup Google Cloud Storage Nearline
Migration Import Export / Snowball Import Export Third Party Solution (Iron
Mountain, etc.)
CDN AWS CloudFront (CDN) Azure CDN Google Cloud CDN
* Ephemeral, DBs, Queues, Caching and Storage GW not included

Account and Keys in AWS
• Root account: account owner, full access to all resources in the account, very specific tasks
(transfer domain, billing details, support plan)
– Email and password + MFA code (if enabled)
• IAM (Identity and Access Management)
– User name and password + MFA code (if enabled) to access AWS Management Console,
AWS discussion forums, or AWS support center
– SAML
– Users, Groups, Roles, Policies. Instance profiles (role)
• Access Keys: AWS SDKs, REST, or Query APIs (AWS CLI)
– Access Key i.e: AKIAIOSFODNN7EXAMPLE
– Secret Access Key i.e: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
• STS (Security Token Service):
– temporary, limited-privilege credentials for IAM users or for users that you authenticate
(also for federated users), and for instances (instance profile)
• Key Pairs: Key pairs are used only for:
– Amazon EC2 (SSH) and Amazon CloudFront (sign URLs or sign cookies)
*Become an IAM Ninja: https://youtu.be/Du478i9O_mc

AWS Metadata Server
“Instance metadata is data about
your instance that you can use to
configure or manage the running
instance”
“Anyone who can access the instance
can view its metadata. Therefore,
you should take suitable precautions
to protect sensitive data”
# curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
Hostname
iam/
instance-action
instance-id
instance-type
local-hostname
local-ipv4
Mac
metrics/
network/
placement/
product-codes
Profile
public-keys/
reservation-id
security-groups
services/

Common incidents
• Access Keys compromise, abuses of unconsciously published keys:
e.g. a developer publishing their keys on SourceCode repo (Github,
Bitbucket, etc) when commit, keys stolen from workstation, keys
hardcoded in application files (bin or config), resources created for
criminal purposes, mining, etc.
• Phishing attacks: your instance is going to be retired (targeted to
admins). Hard to detect phising bc comes with HTTPS, S3, etc.
• Compromised resources: e.g. an unpatched EC2 instance may be
infected with malware and act as a botnet. Poisioned AMI.
• Unintentional abuses: e.g. a crawler-type own process being
classified as a DDoS attack by a third party.
• Abuses committed by users: e.g. malware or other type of illegal
content being published by the end user of an AWS service on a
public S3 bucket.

Common incidents
• Application running in a role: can lead to an access to the
application role and stole Access Key or access metadata service
• Abuses related to configuration failures: e.g. a web-based proxy
service incorrectly configured being used as an open proxy. SMTP
relay, etc.
• Infection through 3rd party services: you give them keys to
perform actions (DataDog incident 2016)
• Hybrid attacks: attacks partly carried out from a Cloud-based
system or data storing on S3, even when mobile devices or
personal computers are used.
• Organized crime of all sort
• False positives
• Did I say CONFIGURATION FAILURES*?

Create a new one:
$ aws sts get-session-token --duration-seconds 129600
Persistence
• Instance compromised might become Access Keys Compromised
– Metadata service (inside an Instance)
• curl http://169.254.169.254/latest/meta-data/iam/
• curl http://169.254.169.254/latest/meta-data/iam/security-
credentials/<*instanceRole>
{
"Code" : "Success",
"LastUpdated" : "2017-02-02T03:07:42Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIAIWCR2OKMVILEXAMPLE",
"SecretAccessKey" : "DVvxzikewoVBrZN30fFBdEQdTczm1WuGOLXC",
"Token" : "FQoDYXdzELT//////////wEaDH7/lKtowqytymR0bSK3A0VAup4Atle7I3P6N6aRKCNpPIqt===SHORTENED",
"Expiration" : "2017-02-02T09:22:37Z”
}
*If it has been attached to the instance

Create a new one:
$ aws sts get-session-token --duration-seconds 129600
Persistence
• Instance compromised might become a Key Compromised
– Metadata service (inside an instance)
• curl http://169.254.169.254/latest/meta-data/iam/
• curl http://169.254.169.254/latest/meta-data/iam/security-
credentials/<instanceRole>
{
"Code" : "Success",
"LastUpdated" : "2017-02-02T03:07:42Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIAIWCR2OKMVILEXAMPLE",
"SecretAccessKey" : "DVvxzikewoVBrZN30fFBdEQdTczm1WuGOLXC",
"Token" : "FQoDYXdzELT//////////wEaDH7/lKtowqytymR0bSK3A0VAup4Atle7I3P6N6aRKCNpPIqt===SHORTENED",
"Expiration" : "2017-02-02T09:22:37Z”
}

Serverless!!!
• Who is auditing
serverless?
• Amazon Lambda
• CloudWatch
• Azure Cloud Functions
• WebJobs
• Google Cloud
Functions

MadKing Attack
• https://github.com/ThreatResponse/mad-king
• Using stolen access keys. Uses Zappa.io. Creates an API Gateway and
Lambda function
• Features
• Disable CloudTrails
• Encrypt CloudTrails
• Generate New Developer Access Keys
• Stop Instances
• Terminate Instances
• Burn them all (Destroy all instances) – really Mad King m/
https://danielgrzelak.com/backdooring-an-aws-account-da007d36f8f9#.ut0x2bjv5

AWS Lambda Infection Toolkit
• https://github.com/Miserlou/mackenzie (zappa.io guy)
• Persistent Lambda Malware PoC
• Features
• Encrypt with pubkey
• Exfil via POST, S3, Email, SMS, Network Resource Tags
• Install Flask backdoor
• Infect old package sources
• Infect all available functions
• Create re-infection handlers
Gone in 60 Millisecons (33c3): https://www.youtube.com/watch?v=YZ058hmLuv0

Other Attacks Tools
• Metasploit AWS module
• IAM privilege enumeration module
• Lambda module
• S3 bucket and access enumeration
• AWS pwn
• Reconnaissance, exploitation and exfiltration
• https://github.com/dagrz/aws_pwn
• Comulus Cloud Attack Module (not an attack just vulnerable sample
code)
• Presented at RSA 2017 (Serverless Security)
• https://github.com/devsecops/lambhack

Now what?
• Control
• Impact
• Recover
• Investigate
• Improve
• Notifications from AWS
• Access activity (IAM)
• Billing activity (Budget alerts)
• Logs
• Other
• Third parties (dedicated tools)
• NIDS (Snort, Suricata, etc.)
• HIDS (OSSEC, Osquery, rkhunter,
Auditd)
• SIEM
Incident Indicators

Sample Task List / Workflow
Compromised
Instance
Start
Forensic
Workstation
Live
or
Dead
Attach the
Tools Volume
Apply
Isolation
SG
Stolen
API Keys
Check new
resources created
Disable Keys
Make API log report if
enabled
If found
Isolate them
Create a report
Attach the
Evidence
Collection
Volume
Isolate
it?
Log in to the
instance
Perform
Evidence
Acquisition
Take
snapshot to
all volumes
Stop it
Make Volumes to
Snapshots
Attach Volumes
to Forensic
Workstation
Attach the
Evidence
Collection
Volume
Log in to the
Forensic
Workstation
Analyze /
Further
Investigation
Incident
Live
Dead
Create
Support Case
with Provider
Revoke Access
Revoke Sessions
Outside Info
Acquisition
(instance
profile,endpo
ints,metadat
a, etc)_
Perform
Timeline
Pre-built
Volatiliy
frofile
Pre-built
LiME krnl
mod
RAM
Acquisition
Yes
NIC Network
Scan
Open an
Internal Case
Separate Network
with Internet
Access to Scan
CAINE /
SIFT / DEFT
/ FCCU /
HELIX3 /
FIRE
Windows_Life_Response
Sysinternals
Nirsoft
FTK Imager
Autopsy
Sleuthkit
Trigger a
Network
Capture
TAG
Resources
under
investigation
*

Outside Info Acquisition
Perform
Evidence
Acquisition
AWS Infrastructure Logs:
CloudTrail and VPCFlowLogs
AWS Service Logs:
S3 Logs, RDS Logs, Lambda, etc.
Host Based Logs
Messages/System, security, audit, applications, etc.
More Inside:
instance profile, endpoints, syslogs, screen, metadata, etc
More Outside:
Limits, check resources creation from date (all regions)

Tools
• March 2016:
• https://blyx.com/2016/03/11/forensics-in-aws-an-introduction/
• June 2016:
• https://blyx.com/2016/06/16/cloud-forensics-caine7-on-aws/
• August 2016:
• Threat Response (Presented in BlackHat 2016)
• https://s3-us-west-2.amazonaws.com/threatresponse-static/us-16-Krug-
Hardening-AWS-Environments-and-Automating-Incident-Response-for-AWS-
Compromises-wp.pdf

Pre-Automation POC – AWS CLI (Scripts)
# DISABLE STOLEN KEYS
aws iam update-access-key --access-key-id
AKIAIOSFODNN7EXAMPLE --status Inactive
--user-name Bob
aws iam delete-access-key --access-key
AKIDPMS9RO4H3FEXAMPLE --user-name Bob
# LOOK FOR NEW RESOURCES CREATED
aws ec2 describe-instances --region us-east-1
--query
'Reservations[].Instances[?LaunchTime>=`2017-02-
3`][].{id: InstanceId, type: InstanceType,
launched: LaunchTime}'
# TAG INSTANCE
aws ec2 create-tags --resources i-INSTANCE-ID
--tags “Key=Environment,
Value=Quarantine:REFERENCE-ID”
# ISOLATE AN INSTANCE IN A VPC
aws ec2 create-security-group --group-name
isolation-sg --description “Security group to isolate
a EC2-VPC instance” --vpc-id vpc-1a2b3c4d
aws ec2 authorize-security-group-ingress
--group-id sg-BLOCK-ID --protocol tcp --port 22
--cidr YOUR.IP.ADDRESS.HERE/32
aws ec2 authorize-security-group-egress
--group-id sg-BLOCK-ID --protocol ‘tcp’
--port 80 --cidr ‘0.0.0.0/0’
aws ec2 modify-instance-attribute --instance-id
i-INSTANCE-ID --groups sg-BLOCK-ID
# CREATE VOLUME SNAPSHOT
aws ec2 create-snapshot –-volume-id vol-xxxx
–-description “IR-ResponderName- Date-REFERENCE-ID”

Threat Response Tool
• Incident Response Tool for AWS
• http://threatresponse.cloud/
• Compromised AWS API
credentials (Access Keys)
• Mitigate compromise: Lock
• Compromised EC2 instance
• Mitigate compromise
• Isolation
• Collect evidence
• Memory acquisition
• Create an Incident Response
Workstation in AWS
• Start an EC2 instance
• Analysis of collected evidences
• WebApp (ThreatResponseWeb)
• RAM (volatility)
• Disk (Log2time + TimeSketch)
• AWS hardening
• Threatprep
• API
• Modules

<DEMO>
ThreatResponse: aws_ir, margaritashotgun

Persistence Prevention (AWS)
• Instance compromised might become a Key Compromised
– UserData in CloudFormation: watch out!
• Prevent it is not very difficult:
• STS tokens can’t be revoked (you only can disable permissions)
• CloudTrail may help to detect it (if enabled!)
– watch the watcher
• It can shutdown your company! (you won’t be the first one: CodeSpaces)
• Use multiple AWS accounts!
iptables -A OUTPUT -m owner ! --uid-owner root -d 169.254.169.254 -j DROP

Instance / Network / Provider
• Put all what you need in your well known AMI:
• Hardening applied / Tested (Packer/Vagrant)
• CIS Benchmark!
• No config / access needed
• Local tools
• Osquery / OSSEC / rkhunter
• Update rules / serverless
• local configuration (SELinux/AppArmour)
• AuditD
• Collect telemetry host network data (snort/suricata)
• Collect everything your provider allows you
• Networking
• APIs / Accesses
• Red Team / Third party pentesting*

API calls
• Who
• When
• What call
• What resources
• Where (from)

Auditing, Assessment and Hardening Tools
• AWS Trusted Advisor
• AWS CloudTrail / Azure Operational
Insights
• AWS CloudFormation
• AWS Config Rules
• Alfresco: Prowler / Automate
Hardening CIS Section 3* /
OpenSCAP fix (AWS)
• Nccgroup: Scout2 (AWS)
• Netflix: SecurityMonkey, EDDA,
FIDO (AWS)
• Capital One: CloudCustodian (AWS)
• AWS CIS Benchmark Python code
and Lambda functions (AWS)
• CloudSploit (AWS)
• Widdix Hardening Templates (AWS)
• Awslimitchecker (AWS)
• OMS Security & Compliance (Azure)
• Spotify: gcp-audit (GCP)
• *Analytics (ELK, Splunk, Nuix etc)
• Git Secrets (AWS)

<DEMO>
Hardening Automation with templates, Prowler, Security Monkey

Takeaways
Samples, templates, code, links and this presentation is already
available at:
https://github.com/toniblyx/rootedcon2017

TL;DR
• Automation for everything (deployment multi AZ,
hardening, response, recovery/recreation, centralized
logging, log everything!)
• Encryption Everywhere (any layer, any content, on-prem,
on-transit)
• Account Separation and MFA (prod, test, devel, etc.)
• Least Privilege
• Go to Immutability / Ephemeral
• Expect to be Hacked: Buy Bitcoins…

Questions?
toni@blyx.com - @ToniBlyx

References
• Cloud Security Alliance, Mapping the Forensic Standard ISO/IEC 27037 to Cloud Computing, June 2013
• Dr. Keyun Ruan University College Dublin, Designing a Forensic-enabling Cloud Ecosystem, 2013
• International Standard ISO/IEC 27037, Information technology — Security techniques — Guidelines for identification, collection, acquisition, and preservation of digital evidence, October
2012
• Josiah Dykstra, Digital Forensics for IaaS Cloud Computing, June 2012
• Keyun Ruan, Ibrahim Baggili (PhD), Prof Joe Carthy, Prof Tahar Kechadi University College Dublin, Zayed University, Survey on Cloud forensics and critical criteria for Cloud forensic capability:
A preliminary analysis
• Keyun Ruan, Joe Carthy, Tahar Kechadi and Mark Crosbie, Cloud Forensics
• Keyun Ruan, University College Dublin, Cloud Forensics: challenges & opportunitiess, 2010
• NIST Cloud Computing Forensic Science Working Group Information Technology Laboratory, NIST Cloud Computing Forensic Science Challenges, June 2014
• Peter Mell Timothy Grance, NIST Special Publication 800-145, The NIST Definition of Cloud Computing, September 2011
• Report From the First Digital Forensic Research Workshop (DFRWS), A Road Map for Digital Forensic Research, August 2001
• Forensics-as-a-Service (FaaS): Computer Forensic Workflow Management and Processing Using Cloud. Yuanfeng Wen, Xiaoxi Man, Khoa Le and Weidong Shi
• http://static1.squarespace.com/static/5417f7f9e4b0b77770545590/t/56f3c598906340a7f6e78dbd/1458816415654/AWS_Cloud_and_Security.pdf
• https://www.blackhat.com/docs/us-16/materials/us-16-Amiga-Account-Jumping-Post-Infection-Persistency-And-Lateral-Movement-In-AWS-wp.pdf
• https://alestic.com/2015/10/aws-iam-readonly-too-permissive/
• Backdooring an AWS account
• Exploring an AWS account post-compromise
• Disrupting AWS logging
• AWS IAM "ReadOnlyAccess" Managed Policy is Too Permissive (For Us)
• Access Keys will kill you before you kill the password
• Account Jumping Post Infection Persistency and Lateral Movement in AWS
• Disrupt CloudTrail and pwning automation tools
• RSA 2017 talk: Cloud Security Automate or Die, same tittle as mine but a bit different approach
• RSA 2017 talk: Securing Serverless applications in the Cloud
• RSA 2017 talk: DevSecOps on the Offense: Automating Amazon Web Services Account Takeover

Thanks!
Special Thanks to:
Alfresco DevOps Team
Andrew K. @andrewkrug & Joel F., ThreatResponse.cloud Team
Daniel Grzelak @dagrz
Lorenzo Martinez @lawwait

Automate or die! Rootedcon 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Automate or die! Rootedcon 2017

Similar to Automate or die! Rootedcon 2017 (20)

More from Toni de la Fuente

More from Toni de la Fuente (20)

Recently uploaded

Recently uploaded (20)

Automate or die! Rootedcon 2017

Editor's Notes