In this talk, Nicki Watt will initially look to introduce and highlight some of the typical security challenges which engineers may encounter, and need to be aware of, when trying to develop and deploy a microservices-based architecture. The 2nd half of the talk tries to get a bit more practical, and through some examples, looks to demonstrate how a tool like Vault from HashiCorp can be used as part of your overall security toolkit to address some of these challenges.
This talk will not be delving into the depths of cryptography and algorithms, rather it is aimed at highlighting some typical problem areas, and giving practical insight into some of the options which can be used to address them.
About Nicki Watt
Nicki Watt is a Lead Consultant for OpenCredo having joined the company in 2011. Nicki is responsible for both hands on and overall leadership of engagements for OpenCredo. She has experience leading both development and architectural teams across a wide range of industries including enterprise organisations and start ups.
6. 6
from the monolith …
image credit: http://lovealwaysbear.blogspot.co.uk/2011_01_01_archive.html
Applications
7. 7
to microservices
image credit: http://www.guinnessworldrecords.com/world-records/most-tennis-
balls-held-in-the-mouth-dog
Applications
8. 8
to microservices
image credit: http://www.guinnessworldrecords.com/world-records/most-tennis-
balls-held-in-the-mouth-dog
Not every
problem
needs
m
icroservices!
Applications
9. 9
from Silo’d teams with manual
release processes
image credit: http://kittypluscoco.blogspot.co.uk/
2011/04/day-at-dog-park.html
Teams
32. 33
IDENTIFY
PROTECT
DETECT
RESPOND
RECOVER
What stuff needs protecting?
What can I do to protect it?
How will I know if bad stuff happens?
What should I do when bad stuff happens?
How can I get my system back up and
running after bad stuff has happened?
37. store api
store front
sensitive
data
passwords,
keys
user
service
product service
external system XXX
39
IDENTIFY
gain access
to internal
network
steal
sensitive user
data
attack
store front
/ API
sniff non
encrypted
traffic
SQL
Injection
Alter
query to
get data
modify
data in DB
social
engineering
sniff non
encrypted
traffic
39. external system XXX
41
IDENTIFY
store api
store front
sensitive
data
passwords,
keys
user
service
product service
gain access
to internal
network
steal
sensitive user
data
social
engineering
sniff non
encrypted
traffic
40. store api
store front
sensitive
data
passwords,
keys
user
service
product service
external system XXX
42
IDENTIFY
gain access
to internal
network
steal
sensitive user
data
attack
store front
/ API
sniff non
encrypted
traffic
SQL
Injection
Alter
query to
get data
modify
data in DB
social
engineering
sniff non
encrypted
traffic
42. store api
store front
sensitive
data
passwords,
keys
user
service
product service
external system XXX
44
PROTECT
attack
store front
/ API
sniff non
encrypted
traffic
SQL
Injection
Alter
query to
get data
modify
data in DB
HTTPS Use prepared statements
build web app vuln
verification into CI/CD
gain access
to internal
network
social
engineering
sniff non
encrypted
traffic
steal
sensitive user
data
HTTPS
Firewall
43. store api
store front
sensitive
data
passwords,
keys
user
service
product service
external system XXX
45
PROTECT
attack
store front
/ API
sniff non
encrypted
traffic
SQL
Injection
Alter
query to
get data
modify
data in DB
HTTPS Use prepared statements
build web app vuln
verification into CI/CD
gain access
to internal
network
social
engineering
sniff non
encrypted
traffic
steal
sensitive user
data
HTTPS
Firewall
44. store api
store front
sensitive
data
passwords,
keys
user
service
product service
external system XXX
46
PROTECT
attack
store front
/ API
sniff non
encrypted
traffic
SQL
Injection
Alter
query to
get data
modify
data in DB
HTTPS Use prepared statements
build web app vuln
verification into CI/CD
gain access
to internal
network
social
engineering
sniff non
encrypted
traffic
steal
sensitive user
data
HTTPS
Firewall
cfssl
45. store api
store front
sensitive
data
passwords,
keys
user
service
product service
external system XXX
47
PROTECT
attack
store front
/ API
sniff non
encrypted
traffic
SQL
Injection
Alter
query to
get data
modify
data in DB
HTTPS Use prepared statements
build web app vuln
verification into CI/CD
gain access
to internal
network
social
engineering
sniff non
encrypted
traffic
steal
sensitive user
data
HTTPS
Firewall
46. store api
store front
sensitive
data
passwords,
keys
user
service
product service
external system XXX
48
DETECT
Log suspicious queries
Log HTTP requests
Log HTTP
requests
attack
store front
/ API
sniff non
encrypted
traffic
SQL
Injection
Alter
query to
get data
modify
data in DB
HTTPS Use prepared statements
build web app vuln
verification into CI/CD
gain access
to internal
network
social
engineering
sniff non
encrypted
traffic
steal
sensitive user
data
HTTPS
Firewall
IDS
47. store api
store front
sensitive
data
passwords,
keys
user
service
product service
external system XXX
49
gain access
to internal
network
infect
employee
computer
install
malware via
email
sniff non
encrypted
traffic
compromise
user data
attack
store front
/ API
sniff non
encrypted
traffic
SQL
Injection
Alter
query to
get data
modify
data in DB
HTTPS
HTTPS
Firewall
antivirus
Use prepared statements
IDS
Log suspicious queries
Log HTTP requests
Log HTTP
requests
build web app vuln
verification into CI/CD
DETECT
Distributed
logging
capability
Container
level
logging
Alerting
capability
Infrastructure
level
logging
Serverless
logging
???
48. store api
store front
sensitive
data
passwords,
keys
user
service
product service
external system XXX
50
gain access
to internal
network
infect
employee
computer
install
malware via
email
sniff non
encrypted
traffic
compromise
user data
attack
store front
/ API
sniff non
encrypted
traffic
SQL
Injection
Alter
query to
get data
modify
data in DB
HTTPS
HTTPS
Firewall
antivirus
Use prepared statements
IDS
Log suspicious queries
Log HTTP requests
Log HTTP
requests
build web app vuln
verification into CI/CD
DETECT
Distributed
logging
capability
Container
level
logging
Alerting
capability
Infrastructure
level
logging
Serverless
logging
???
49. store api
store front
sensitive
data
passwords,
keys
user
service
product service
external system XXX
52
RESPOND
Redirect to
HTTPS
Block consistent offenders
Adjust firewall rules
Block attackers
Log suspicious queries
Log HTTP requests
Log HTTP
requests
attack
store front
/ API
sniff non
encrypted
traffic
SQL
Injection
Alter
query to
get data
modify
data in DB
HTTPS Use prepared statements
build web app vuln
verification into CI/CD
gain access
to internal
network
social
engineering
sniff non
encrypted
traffic
steal
sensitive user
data
HTTPS
Firewall
IDS
Change DB Password
Reset users passwords
Inform users
Redirect to HTTPS
50. store api
store front
sensitive
data
passwords,
keys
user
service
product service
external system XXX
53
Log suspicious queries
Block consistent offenders
RECOVER
Redirect to
HTTPS
Block consistent offenders
Adjust firewall rules
Block attackers
Log suspicious queries
Log HTTP requests
Log HTTP
requests
attack
store front
/ API
sniff non
encrypted
traffic
SQL
Injection
Alter
query to
get data
modify
data in DB
HTTPS Use prepared statements
build web app vuln
verification into CI/CD
gain access
to internal
network
social
engineering
sniff non
encrypted
traffic
steal
sensitive user
data
HTTPS
Firewall
IDS
Change DB Password
Reset users passwords
Inform users
Redirect to HTTPS
Restore from backup
Fix Code,
Blue/Green deploys:
redeploy microservice(s)
redeploy infrastructure
52. • Due diligence: know thy playground
• Think holistically: identify, protect, detect,
respond, recover
Summary
55
Make security a
1st class citizen
in your thinking process!
53. • Multiple, diverse, interconnected services
• More varied attack surfaces
• Harder to track what’s going on
(distributed, multi facetted logging capabilities)
• Transient components
• Dynamic transport level encryption (HTTPS)
• Authentication & Authorisation (see David’s talk :)
• Trash & burn recovery strategies
Microservice security challenges
56
57. 61
• Unified API to access multiple backends
• ACL policies - who can access what
• Audit Logs
58. 62
Unseal
Init
service 1
service 2
Allow token
to be used by tools
to access secrets
Acquire policy
constrained
token
Create microservice
mount or area, add
secrets
System X
59. 63
$ vault init -key-shares=3 -key-threshold=2
Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba
Key 2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c
Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91
Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b
Vault initialized with 3 keys and a key threshold of 2. Please
securely distribute the above keys. When the Vault is re-sealed,
restarted, or stopped, you must provide at least 2 of these keys
to unseal it again.
Vault does not store the master key. Without at least 2 keys,
your Vault will remain permanently sealed.
$ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba
Sealed: true
Key Shares: 3
Key Threshold: 2
Unseal Progress: 1
Vault init & unseal
$ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91
Sealed: true
Key Shares: 3
Key Threshold: 2
Unseal Progress: 0
60. 64
$ vault init -key-shares=3 -key-threshold=2
Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba
Key 2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c
Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91
Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b
Vault initialized with 3 keys and a key threshold of 2. Please
securely distribute the above keys. When the Vault is re-sealed,
restarted, or stopped, you must provide at least 2 of these keys
to unseal it again.
Vault does not store the master key. Without at least 2 keys,
your Vault will remain permanently sealed.
$ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba
Sealed: true
Key Shares: 3
Key Threshold: 2
Unseal Progress: 1
Vault init & unseal
$ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91
Sealed: true
Key Shares: 3
Key Threshold: 2
Unseal Progress: 0
61. 65
$ vault init -key-shares=3 -key-threshold=2
Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba
Key 2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c
Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91
Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b
Vault initialized with 3 keys and a key threshold of 2. Please
securely distribute the above keys. When the Vault is re-sealed,
restarted, or stopped, you must provide at least 2 of these keys
to unseal it again.
Vault does not store the master key. Without at least 2 keys,
your Vault will remain permanently sealed.
$ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba
Sealed: true
Key Shares: 3
Key Threshold: 2
Unseal Progress: 1
Vault init & unseal
$ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91
Sealed: true
Key Shares: 3
Key Threshold: 2
Unseal Progress: 0
64. 68
$ vault mount -path=usersvc generic
Successfully mounted 'generic' at ‘usersvc'!
$ vault mounts
Path Type Default TTL Max TTL Description
cubbyhole/ cubbyhole n/a n/a per-token private secr ...
secret/ generic system system generic secret storage
sys/ system n/a n/a system endpoints used f...
usersvc/ generic system system
Vault create new mount
65. 69
$ vault write usersvc/db-password value=ASDKJ234SF*2
Success! Data written to: usersvc/db-password
$ vault read usersvc/db-password
Key Value
lease_duration 2592000
value ASDKJ234SF*2
Vault write, then read back secret
67. 71
Unseal
Allow token
to be used by tools
to access secrets
Init
Acquire policy
constrained
token
service 1
service 2
Create segregated
area, add secrets
System X
74. 78
Push secrets in …
user
service
db1
1authenticate
2
orchestration /
deployment platform
3
provide value as
environment
variables
read secret/db-password
75. 79
user
service
db1
1authenticate
2
read secret/db-password
orchestration /
deployment platform
3
provide value as
environment
variables
$ vault auth e2d0a065-xxxx-yyyy-zzzz
Successfully authenticated! You are…
token_policies: [default, usersvc]
$ vault read usersvc/db-password
Key Value
--- -----
refresh_interval 2592000
value MyClearTextPassword
1
2
77. 81
Steal Sensitive User DataIDENTIFY
steal
sensitive user
data
steal
sensitive user
data
gain access to
internal network
gain access to
user DB
gain access to
running user
microservice(s)
dump startup
config
steal plaintext
password
social
engineering
78. the-machine$ docker ps
CONTAINER ID IMAGE ... CREATED STATUS NAMES
9950ea8e3c59 product-service:v1 ... 4 days ago Up 4 days prodsvc
29b9ebca6dab user-service:v2 ... 5 days ago Up 5 days usersvc
82
gain access to
running user
microservice(s)
81. 85
gain access to
internal network
gain access to
user DB
gain access to
running user
microservice(s)
dump startup
config
social
engineering
PROTECT
don’t expose as
plain text
steal
sensitive user
data
steal plaintext
password
limit user access
83. 87
Push secrets in …
user
service
db1
1authenticate
2
orchestration /
deployment platform
3
provide value as
environment
variables
read secret/db-password
89. 92
gain access to
internal network
gain access to
running user
microservice(s)
dump startup
config
find a
disgruntled
employee
PROTECT
steal
sensitive user
data
steal plaintext
password
don’t expose as
plain text
gain access to
user DB
limit user access
90. 93
gain access to
internal network
gain access to
running user
microservice(s)
dump startup
config
find a
disgruntled
employee
PROTECT
steal
sensitive user
data
steal plaintext
password
don’t expose as
plain text
gain access to
user DB
limit user access
91. 94
gain access to
internal network
gain access to
running user
microservice(s)
dump startup
config
find a
disgruntled
employee
PROTECT
steal
sensitive user
data
don’t expose as
plain text
gain access to
user DB
steal wrapped
password
get real
password
limit user access
92. 95
user
service
db1
1authenticate
2
read wrapped secret
orchestration /
deployment platform
3
provide wrapped
value as environment
variables
4
unwrap
$ vault unwrap 57ccef32-471d-869
error reading cubbyhole/response: Error making
API request.
URL: GET https://vault:8200/v1/cubbyhole/response
Code: 400. Errors:
* permission denied
4
93. 96
gain access to
internal network
gain access to
running user
microservice(s)
dump startup
config
find a
disgruntled
employee
PROTECT
steal
sensitive user
data
DETECT
don’t expose as
plain text
gain access to
user DB
steal wrapped
password
get real
password
Raise TOFU
alarm
Audit access
limit user access
94. 97
gain access to
internal network
gain access to
running user
microservice(s)
dump startup
config
find a
disgruntled
employee
PROTECT
steal
sensitive user
data
DETECT
don’t expose as
plain text
RESPOND
gain access to
user DB
steal wrapped
password
get real
password
Raise TOFU
alarm
Audit access
change
DB password
limit user access
95. 98
gain access to
internal network
gain access to
running user
microservice(s)
dump startup
config
find a
disgruntled
employee
PROTECT
steal
sensitive user
data
DETECT
don’t expose as
plain text
RESPOND
gain access to
user DB
steal wrapped
password
get real
password
Raise TOFU
alarm
Audit access
change
DB password
Expect secrets to change.
Make a habit of changing them regularly.
It will naturally force you to put
measures in place. limit user access
96. • Dynamic Secrets: Auto generate
credentials on the fly
Other handy options
99
98. 101
user
service
db1
1authenticate
2
orchestration /
deployment platform
3
provide value as
environment
variables
$ vault mount postgresql
Successfully mounted 'postgresql' at
'postgresql'!
$ vault write postgresql/config/connection
connection_url="postgresql://
vault:somepassword@yourhost:5432/postgres"
$ vault write postgresql/roles/usersvc-ro
sql="CREATE ROLE "{{name}}"
WITH LOGIN PASSWORD ‘{{password}}'
VALID UNTIL '{{expiration}}';
GRANT SELECT ON ALL TABLES
IN SCHEMA users TO "{{name}}";"
Success! Data written to: postgresql/roles/
read dynamic password
Human /
Other System
Users
0
0
99. 102
user
service
db1
1authenticate
2
orchestration /
deployment platform
3
provide value as
environment
variables
$ vault mount postgresql
Successfully mounted 'postgresql' at
'postgresql'!
$ vault write postgresql/config/connection
connection_url="postgresql://
vault:somepassword@yourhost:5432/postgres"
$ vault write postgresql/roles/usersvc-ro
sql="CREATE ROLE "{{name}}"
WITH LOGIN PASSWORD ‘{{password}}'
VALID UNTIL '{{expiration}}';
GRANT SELECT ON ALL TABLES
IN SCHEMA users TO "{{name}}";"
Success! Data written to: postgresql/roles/
read dynamic password
Human /
Other System
Users
0
0
100. 103
user
service
db1
1authenticate
2
orchestration /
deployment platform
3
provide value as
environment
variables
$ vault mount postgresql
Successfully mounted 'postgresql' at
'postgresql'!
$ vault write postgresql/config/connection
connection_url="postgresql://
vault:somepassword@yourhost:5432/postgres"
$ vault write postgresql/roles/usersvc-ro
sql="CREATE ROLE "{{name}}"
WITH LOGIN PASSWORD ‘{{password}}'
VALID UNTIL '{{expiration}}';
GRANT SELECT ON ALL TABLES
IN SCHEMA users TO "{{name}}";"
Success! Data written to: postgresql/roles/
read dynamic password
Human /
Other System
Users
0
0
101. 104
user
service
db1
1authenticate
2
orchestration /
deployment platform
3
provide value as
environment
variables
$ vault read postgresql/creds/usersvc-ro
Key Value
lease_id postgresql/creds/usersvc-ro/
c888a097-b0e2-26a8-b306-fc7c84b98f07
lease_duration 3600
password 34205e88-0de1-68b7…
username vault-14301-usersvc-ro
read dynamic password
Human /
Other System
Users
0
2
103. • Dynamic Secrets: Auto generate
creds on the fly
• Ability to combine security primitives
dynamic secrets + resource wrapping
Other handy options
106
104. 107
gain access to
internal network
gain access to
user DB
gain access to
running user
microservice(s)
dump startup
config
find a
disgruntled
employee
PROTECT
steal
sensitive user
data
DETECT
steal wrapped
password
don’t expose as
plain text
get real
password
Raise TOFU
alarm
Audit access
RESPOND
change
DB password
105. 108
gain access to
internal network
gain access to
user DB
gain access to
running user
microservice(s)
dump startup
config
find a
disgruntled
employee
PROTECT
steal
sensitive user
data
DETECT
steal wrapped
password
don’t expose as
plain text
get real
password
Raise TOFU
alarm
Audit access
RESPOND
change
DB password
use time limited
dynamic creds
106. 109
gain access to
internal network
gain access to
user DB
gain access to
running user
microservice(s)
dump startup
config
find a
disgruntled
employee
PROTECT
steal
sensitive user
data
DETECT
steal wrapped
password
don’t expose as
plain text
get real
password
Raise TOFU
alarm
Audit access
RESPOND
change
DB password
use time limited
dynamic creds
compromise
orchestration
platform
108. 111
gain access to
internal network
gain access to
user DB
gain access to
running user
microservice(s)
dump startup
config
compromise
orchestration
platform
find a
disgruntled
employee
steal
sensitive user
data
steal vault
token
get db
password
1
2
34
Defense in Depth
109. Put enough hurdles in the way
of attackers for you to stop
when you can, but if not, to be
able to …
- realise what’s going on
- react before too much damage
is done
112
110. • Centralised Secrets Management
• API - helps with automation
• Tries to address concerns across full
security lifecycle
• But still very new & maturing
Vault Summary
113
111. • Encryption as a service: offload
responsibility to Vault
• PKI: Generates X.509 certificates
dynamically based on configured
roles
• SSH: Dynamically generates SSH
credentials for remote hosts
Other Handy Features
114