Presented to the Philly DevOps Meetup November 29, 2016. Managing secrets is hard. It’s even harder in the cloud. At Jornaya (formerly LeadiD), we chose Hashicorp Vault to manage our secrets in AWS, and I’d like to share our experience with everyone.

1. Chickens & Eggs
Managing secrets in AWS with Hashicorp Vault
Jeff Horwitz
Jornaya
jhorwitz@jornaya.com

2. Applications need secrets
• Where do we store the secrets?
• How do we manage the secrets?
• How do servers/applications obtain the secrets?

3. Jornaya Environment
• Applications run in AWS in Autoscaling groups
• Configuration management via Chef server
• No PCI, HIPAA or other compliance requirements
• Secrets include passwords, private keys, API keys

4. Vault
“Vault secures, stores, and tightly controls access to tokens,
passwords, certificates, API keys, and other secrets in modern
computing. Vault handles leasing, key revocation, key rolling, and
auditing. Through a unified API, users can access an encrypted
Key/Value store and network encryption-as-a-service, or generate
AWS IAM/STS credentials, SQL/NoSQL databases, X.509
certificates, SSH credentials, and more.”
https://www.vaultproject.io/

5. Vault
✓ Platform independent
✓ High availability
✓ User & server authentication options
✓ Fine-grained access control
✓ Good language and tooling support
✓ Under active development

6. Other Options
• Instance userdata
• Chef encrypted databags
• Chef vault
• Citadel
• KMS

7. Vault Server
• Responds to client requests
• Interacts with backends
• storage, authentication, secret, audit
• Encrypts/Decrypts secrets with master key
• Master key is never stored on disk

8. Playing with Vault
~$ vault server --dev
==> WARNING: Dev mode is enabled!
In this mode, Vault is completely in-memory and unsealed.
Vault is configured to only have a single unseal key. The root
token has already been authenticated with the CLI, so you can
immediately begin using the Vault CLI.
The only step you need to take is to set the following
environment variables:
export VAULT_ADDR='http://127.0.0.1:8200'
The unseal key and root token are reproduced below in case you
want to seal/unseal the Vault or play with authentication.
Unseal Key: e95cf9d02c044c67dfd5d379d3a56cee7a0209f8cc681fca1435b6c022fcf028
Root Token: 611f6670-f871-ef2d-2110-87a54261d407

9. Talking to Vault
• Vault provides an HTTP(S) RESTful API
• JSON responses
• The vault command is a user-friendly wrapper
• Modules available for various languages
• Config management (e.g. Chef via Ruby gem)

10. Reading & Writing Secrets
~$ export VAULT_ADDR=http://127.0.0.1:8200
~$ vault write secret/phillydevops/luggage-combo value=12345
Success! Data written to: secret/phillydevops/luggage-combo
~$ vault read secret/phillydevops/luggage-combo
Key Value
--- -----
refresh_interval 720h0m0s
value 12345

11. HTTP API
~$ curl -X POST -d '{"value": "12345"}'
-H 'X-Vault-Token: 611f6670-f871-ef2d-2110-87a54261d407'
http://127.0.0.1:8200/v1/secret/phillydevops/luggage-combo

12. HTTP API
~$ curl -k -H 'X-Vault-Token: 611f6670-f871-ef2d-2110-87a54261d407'
http://127.0.0.1:8200/v1/secret/phillydevops/luggage-combo | jq
{
"request_id": "ac71da58-cc11-431b-598e-7eb3fe3e37a3",
"lease_id": "",
"renewable": false,
"lease_duration": 2592000,
"data": {
"value": "12345"
},
"wrap_info": null,
"warnings": null,
"auth": null
}

13. Shamir’s Secret Sharing
• https://en.wikipedia.org/wiki/Shamir's_Secret_Sharing
• Splits a key into n shards (unseal keys)
• k shards needed to derive original key (k < n)
• No one person can easily obtain the original key
• No need to revoke shards if people leave

14. Unsealing Process

15. Unsealing Process
• Vault server does not store the master key on disk
• Given the master key via the unsealing process
• Need minimum number of shards to unseal
• Key is only stored in memory
• Restarting will lose the key and “seal” the vault.

16. Unsealing Demo
~$ vault read secret/phillydevops/luggage-combo
Error reading secret/phillydevops/luggage-combo: Error making API request.
URL: GET http://127.0.0.1:8200/v1/secret/phillydevops/luggage-combo
Code: 503. Errors:
* Vault is sealed
~$ vault unseal
Key (will be hidden): ********
Sealed: false
Key Shares: 1
Key Threshold: 1
Unseal Progress: 0

17. Backends
Storage backends
Secret backends
Authentication backends
Audit backends

18. Storage Backends
• Physical storage for secrets
• Only see encrypted secrets
• High availability features

19. Storage Backends
Backend Support HA?
Consul Official Yes
Zookeeper Community Yes
etcd Community Yes
DynamoDB Community Maybe
S3 Community No
Swift Community No
Azure Community No
mysql Community No
postgresql Community No
inmem Official No
file Official No

20. Storage Backends
Backend Support HA?
Consul Official Yes
Zookeeper Community Yes
etcd Community Yes
DynamoDB Community Maybe
S3 Community No
Swift Community No
Azure Community No
mysql Community No
postgresql Community No
inmem Official No
file Official No

21. Storage Backends
Backend Support HA?
Consul Official Yes
Zookeeper Community Yes
etcd Community Yes
DynamoDB Community Maybe
S3 Community No
Swift Community No
Azure Community No
mysql Community No
postgresql Community No
inmem Official No
file Official No

22. Clustering
• Storage backend must support high availability
• Active and standby servers
• State maintained by Consul (storage backend)
• Use Consul DNS or API to discover active server
• Standby will redirect to active by default

23. Cluster Status
$ vault status
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Version: Vault v0.6.1
Cluster Name: vault-cluster-abcd0123
Cluster ID: deadbeef-beef-beef-beef-deadbeefcafe
High-Availability Enabled: true
Mode: active
Leader: https://vault-1.example.com:8200

24. Secret Backends
• Static or dynamic secrets
• Backend configuration
• Multiple backend mountpoints

25. Generic Backend
• Default backend
• Mounted at secret
• Stores and retrieves static secrets as k/v pairs
• Great for:
• username/password
• RSA private keys
• API keys

26. Tokens
• Authentication in vault is performed via tokens
• UUIDs returned by authentication backends
• Tokens can expire and must be renewed
• Tokens can be revoked
• A single “root” token never expires

27. Authentication Backends
• Returns a token based on credentials
• Tokens may have a TTL (backend-dependent)
• Options to fit different workflows and platforms

28. Authentication Backends
• AppRole (App ID)
• AWS EC2
• GitHub
• LDAP
• TLS Certificates
• Tokens
• Username & Password

29. LDAP Backend
~$ vault auth -method ldap username=jhorwitz
Password (will be hidden):
Successfully authenticated! You are now logged in.
The token below is already saved in the session. You do not
need to "vault auth" again with the token.
token: deadbeef-cafe-beef-beef-deadbeafcafe
token_duration: 28799
token_policies: [admins, default]

30. Lookup a Token
~$ vault token-lookup deadbeef-cafe-beef-beef-deadbeafcafe
Key Value
--- -----
accessor 5776eb4b-05b1-3bab-98d3-08d34040a806
creation_time 1480366497
creation_ttl 28800
display_name ldap-jhorwitz
explicit_max_ttl 0
id deadbeef-cafe-beef-beef-deadbeafcafe
meta map[policies:admins,default username:jhorwitz]
num_uses 0
orphan true
path auth/ldap/login/jhorwitz
policies [admins default]
renewable true
ttl 27379

31. aws-ec2 Backend
• Good for server-level authentication
• Fixes the chicken & egg problem on EC2 instances
• Let AWS do the dirty work for you

32. Instance Identity
$ curl http://169.254.169.254/latest/dynamic/instance-identity/document
{
"privateIp" : “10.0.0.1”,
"devpayProductCodes" : null,
"availabilityZone" : "us-east-1a",
"accountId" : "1234567890",
"version" : "2010-08-31",
"instanceId" : "i-deadbeef",
"billingProducts" : null,
"instanceType" : "t2.nano",
"pendingTime" : "2016-09-09T17:30:35Z",
"architecture" : "x86_64",
"imageId" : "ami-cafebeef",
"kernelId" : null,
"ramdiskId" : null,
"region" : "us-east-1"
}

33. Identity Signature
$ curl http://169.254.169.254/latest/dynamic/instance-identity/pkcs7
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAaCAJIAEggGjewog
ICJwcml2YXRlSXAiIDogIjEwLjUwLjAuNjgiLAogICJkZXZwYXlQcm9kdWN0Q29kZXMiIDogbnVs
bCwKICAiYXZhaWxhYmlsaXR5Wm9uZSIgOiAidXMtZWFzdC0xYSIsCiAgImFjY291bnRJZCIgOiAi
Mjk4Nzg1NDUzNTkwIiwKICAidmVyc2lvbiIgOiAiMjAxMC0wOC0zMSIsCiAgImluc3RhbmNlSWQi
IDogImktYzRkMDQwZjUiLAogICJiaWxsaW5nUHJvZHVjdHMiIDogbnVsbCwKICAiaW5zdGFuY2VU
…

34. Whitelisting
• Vault maintains a whitelist of instance IDs
• Cannot reauthenticate using an ID in the whitelist
• You can turn this off or remove ID from whitelist
• Use a nonce to prevent replay attacks

35. Roles
• Clients specify a role with authentication requests
• Roles are “bound” to instance properties
• IAM role or instance profile
• AMI ID
• Vault cross-checks with EC2 before returning token

36. Creating a Role
$ vault write auth/aws-ec2/role/deployer
bound_iam_role_arn=arn:aws:iam::1234567890:instance-profile/deployer
policies=deployer
Success! Data written to: auth/aws-ec2/role/deployer

37. Policies
• ACLs that are applied to roles, users and groups
• Applied at token creation
• Tokens can have multiple policies

38. Sample User Policy
path "secret/test/*" {
policy = "write"
}
path "secret/*" {
policy = "read"
}
path "secret/classified/*" {
capabilities = ["deny"]
}

39. Sample Server Policy
path "secret/env/prod/*" {
policy = "read"
}
path "secret/shared/*" {
policy = "read"
}
path "secret/ssl/private-key" {
policy = "read"
}

40. Integration
• Chef authenticates instance to aws-ec2 backend
• Chef stores token
• local filesystem (for scripts and apps)
• node run state (for Chef recipes)

41. Get PKCS7 Signature
def instance_identity_pkcs7
uri = URI.parse('http://169.254.169.254/latest/dynamic/instance-identity/pkcs7')
resp = Net::HTTP.get_response(uri)
resp.body.delete("n", '')
end

42. Generate Nonce
nonce = SecureRandom.uuid

43. Send Login Request
def aws_ec2_login(address, role, pkcs7, nonce)
# find the leader, since Net::HTTP doesn't handle redirects
uri = URI.parse("#{address}/v1/sys/leader")
resp = Net::HTTP.get_response(uri)
leader = JSON.parse(resp.body)['leader_address']
uri = URI.parse("#{leader}/v1/auth/aws-ec2/login")
data = {
'role' => role, 'pkcs7' => pkcs7, 'nonce' => nonce
}
req = Net::HTTP::Post.new(uri, 'Content-Type' => 'application/json')
req.body = data.to_json
resp = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http|
http.request(req)
end
…

44. Write Token to File
file node['leadid_vault']['client']['token_file'] do
content lazy { node.run_state['token'] ||
node['leadid_vault']['client']['token'] }
owner node['leadid_vault']['client']['uid']
group node['leadid_vault']['client']['gid']
mode '0440'
sensitive true
action :create_if_missing
only_if { node['leadid_vault']['client']['enabled'] }
end.run_action(:create)

45. Using the Token
• Chef recipes use the vault gem
• Other apps use consul-template

46. Configure Vault Chef gem
def configure(opts = {})
require 'vault'
Vault.configure do |config|
config.address = opts[:address]
config.ssl_ca_cert = opts[:ssl_ca_cert]
config.ssl_verify = opts[:ssl_verify]
if opts[:token]
config.token = opts[:token]
elsif opts[:token_file]
config.token = File.read(opts[:token_file])
else
raise 'must specify either token or token_file'
end
end
end

47. Helper to read secrets
def read(path)
require 'vault'
Vault.logical.read(path)
end

48. Reading secrets in Chef
Chef::Recipe.send(:include, VaultHelpers)
if node['leadid_vault']['client']['enabled']
# Grab the Datadog API and Application Key from vault
api_key = read(
"#{node['leadid_base']['datadog']['secret_path']}" +
"/datadog/api-key").data
node.default['datadog']['api_key'] = api_key[:value]
end
…
include_recipe 'datadog::dd-agent'
include_recipe 'datadog::dd-handler'

49. consul-template
$ consul-template
-consul consul.example.com
-template "/opt/jornaya/app.yml.ctmpl:/opt/jornaya/app.yml:service nginx restart"

50. consul-template
database:
host: db.example.com
dbname: myapp
{{with secret "secret/leadid/env/prod/db-user"}}
user: {{.Data.username}}
password: {{.Data.password}}
{{end}}

51. consul-template
database:
host: db.example.com
dbname: myapp
user: donald
password: WeHaveTheBestPasswords

52. Learnings
• Don't use a self-signed certificate. There is pain.
• For SSL, name your servers or use static IP addresses.
• Clustering was the easiest part. Do it.
• Set up your path hierarchy ahead of time.
• Be prepared to RTFM.
• Different backends behave differently.
• Tokens do not live forever.

53. Questions?