Turbo talk 1: "AWS secret manager for protecting and rotating credentials" - Mike Allen, CIO at Morningstar // @mikeoninfosec OWASP + AWS user groups: Using the OWASP Top 10 in AWS

1. ©2018 Morningstar, Inc. All rights reserved.
Michael Allen, CIO – 10-24-2018
AWS Secrets Manager

2. Where lots of bad decisions are made….

3. When a security person sees a password dialog box

4. When everyone else sees a password dialog box

5. Proof! - Top Passwords of 2017

6. The Problem
▪ OWASP TOP 10 – A3 – Sensitive Data Exposure
▪ Many web applications and APIs do not properly protect sensitive data, including passwords,
encryption keys, etc.
▪ Developers check in passwords, encryption keys, AWS secrets, etc. into source code repositories
(sometimes public) and bad things happen
▪ Passwords are often poorly generated and not rotated frequently
▪ When a password is compromised, it is difficult to change due to it being used in many, disparate
locations
▪ Bad password managements lead to data breaches

7. Overview of AWS Secrets Manager
▪ Makes it easier to rotate, manage and retrieve database credentials, API keys, and
other secrets throughout their lifecycle
▪ Particularly good when using built in AWS services like RDS databases as it
enables auto rotation
▪ Is region specific (e.g. Ohio and Virginia are completely distinct)
▪ Common Users of Secrets Manager:
▪IT Admins: store and manage access to secrets securely and at scale
▪Security admins: audit and monitor the use of secrets, and rotate secrets
without a risk of breaking applications
▪Developers: avoid having to deal with storing secrets in their applications and
prevent the security team from cutting you a finding. ☺

8. Typical Use Case
▪ Connect to a database from application code
▪ Step 1: DBA loads application specific password into
secrets manager
▪ Step 2: DevOps engineer deploys application with an
attached IAM role
▪ Application bootstrapping calls Secrets Manager using
permissions provided by the IAM role, retrieves credential
and connects to the database
▪ DevOps engineer never has access to password!
▪ Password set to automatically rotate

9. Secrets Manager High Level Overview

10. Structure of a Secret

11. Desired Password Lifecycle
▪ Secure and manage secrets centrally
▪ Can store, view, and manage all your secrets in one
secure password vault
▪ Secrets manager encrypts these secrets with encryption
keys you own and control
▪ Integrated with CloudTrail and CloudWatch for logging
and monitoring to meet compliance needs
▪ Example: can be configured to send notification when
an admin deletes a secret as an example

12. Example Cloud Trail Log

13. Desired Password Lifecycle
▪ IAM Policies:
▪ Tag-based access control and hierarchical names
▪ Resource-based policies for cross-account access
▪ Can restrict access to passwords using MFA or even
corporate IP address range
▪ Example: only allow folks to access production
passwords from trusted network

14. Access Control – Using IAM Policies

15. Desired Password Lifecycle
▪ Rotate Secrets Safely
▪ One-click password rotation for supported AWS services
▪ Built-in integrations for rotating MySQL, PostgreSQL and
Amazon Aurora on RDS
▪ Can create custom integrations using Lambda
▪ Uses versioning so that applications won’t break when
secrets are rotated

16. How Much Does It Cost?
▪ Pay only for what you use (no minimum)
▪ Storage: $0.40/month per secret
▪ Access: $0.05 per 10,000 API calls (reads and writes)

18. ©2018 Morningstar, Inc. All rights reserved.
Demo #1 - Interface demo and example
password creation / storage

19. Secret Manager Interface

20. ©2018 Morningstar, Inc. All rights reserved.
Demo #2 – Retrieving a password in code

21. Before
dbconfig.py
gConfiguration file with passwords stored in plain text
gConfig file checked into source code repository
gNot secure

22. After
retrieve_demo.py
▪ No passwords stored in plain text
▪ IAM role configured on EC2 instance giving permission to retrieve secret from secret manager
▪ OK to check into source code repository
▪ Configurable based on environment / application
Name of Secret & Region

23. ©2018 Morningstar, Inc. All rights reserved.
Demo #3 – Retrieving a password via AWS CLI

24. Example of CLI
▪ Step 1: Install AWS CLI and use “aws configure” to setup authentication and region
▪ Step 2: Use “aws secretsmanager help” to show a list of available commands
▪ Step 3: List all secrets in vault
▪ Use “aws secretsmanager list-secrets”
▪ Step 4: Retrieve value of a specific secret
▪ Use “aws secretsmanager get-secret-value –secret-id “dev/app1/database”
▪ Can also create, delete, rotate secrets using CLI
▪ Versioning is used to make this happen (CURRENT vs. PREVIOUS)

25. ©2018 Morningstar, Inc. All rights reserved.
Demo #3 – Rotating an RDS Database Password

26. Password Rotation Workflow
▪ Step 1: Secrets manager creates a new password with equivalent permissions
▪ Step 2: The new password is promoted and returned via subsequent secrets manager API calls
▪ Step 3: Secrets manager safely disables the original credential
▪ Versioning is used to make this happen (CURRENT vs. PREVIOUS)

27. Example of Password Rotation
▪ Rotation interval is configurable (30,60, 90 days)
▪ Creates a Lambda function to handle rotation for RDS

28. Password Rotation Lambda
▪ Built in Lambda functions for RDS
▪ Can create your own custom Lambda’s for other applications that you wish to rotate (extensible)

29. ©2018 Morningstar, Inc. All rights reserved.
Demo #3 – Notifying on delete

30. Notification Workflow
▪ #1 - Enable cloud trail
▪ #2 – Configure cloud trail log file delivery to cloud watch logs
▪ #3 – Create a CloudWatch alarm for a delete event

31. How keys are protected?
▪ How are the secrets protected?
▪ Secrets manager integrates with Amazon KMS to protect the secrets
▪ Every secret is protected with a unique data encryption key which is in turn protected via a customer master
key. Secrets are protected using envelope encryption
▪ You can use the same or different customer master keys for each secret you protect
▪ You can change the customer master key (CMK) at any time but secrets manager does not re-encrypt existing
secrets using the new CMK until the value changes.

32. Ways to access secrets
▪ Programmatically via supplied SDKs and boto3
▪ Via API call
▪ Must use TLS and Perfect Forward Secrecy
▪ Must authenticate with a signed request using AWS secret key
▪ Via AWS CLI

33. What’s missing?
▪ No “break glass” functionality (yet).
▪ No support for “dynamic” secrets (e.g. secrets that are generated on the fly and live for a specific duration of
time)

34. Reference Material
▪ When building this presentation, I leveraged slide material and documentation from both Amazon
Web Services and OWASP (thanks to both of these awesome organizations).

35. Questions?