A presentation given at PuppetCamp NYC 2014 about why Puppet users should stop storing secrets in Git/Hg and encrypt them instead. TLDR: It enables collaboration.

1. The BlackBox project
Safely storing secrets and credentials in
Git/Mg (mostly for use by Puppet)
Tom Limoncelli, SRE, StackExchange.com
Blog: EverythingSysadmin.com
My new book! the-cloud-book.com

2. StackExchange.com
125+ Q&A Communities
ServerFault.com
StackOverflow.com
(We <3 Puppet!)

3. What are secrets?
Anything you don’t want exposed externally.
● SSL Certificates (the private bits)
● Passwords
● API keys

4. Puppet manages secrets

6. If you store
secrets in git,
you’re gonna
have a bad
time.

7. ● Laptops get stolen.
● Workstations have guest accounts
● Git server “Circle of Trust” includes:
○ Everyone with admin access to workstations.
■ Your desktop support people?
○ Everyone with admin access to your git server:
■ Server team, storage team, backup team
○ Everyone you collaborate with that wants read-only
access to Puppet manifests.

8. You have 3 bad options:
1. Deny git access. (Hurts collaboration)
2. Permit git access. (Hurts security)
3. Email individual files. (Hurts… just hurts)

9. Option 4: Encrypt secret parts
● If a file contains secrets, encrypt before
checking into Git.
● Need to edit a secret?
○ Decrypt - Edit - Encrypt

10. What about Puppet master?
● After “git pull”, decrypt all files.
○ Automate this as part of CI.
● Files are unencrypted “at rest”.
● This does not decrease security:
○ No worse than what we were doing before.
○ If you can break into root or puppet on the master,
you’ve already won.

11. Easy, right?
Decrypt:
Encrypt:

12. Easy, right?
Decrypt:
Encrypt:
● ...and don’t make any typos when entering the command
● ...and don't accidentally check in the unencrypted version

13. Security is 1% technology plus 99% following
the procedures correctly.
Any process with more than 1 step probably
won't be followed consistently most of the time.
Related reading: "Why Johnny Can't Encrypt: A Usability Evaluation of PGP
5.0”, Alma Whitten", Usenix Security 1999

14. Therefore…. we automate
Introducing: Blackbox
Scripts for keeping Puppet secrets in git/hg.

15. User commands:
Decrypt for editing:
Encrypt when done:

16. First time a file is encrypted:
Enroll a file into the system:

17. Commands that act on all GPG files:
Decrypt all files: (for use on puppet master)
Re-encrypt all files: (after new users added)

18. Everyone has their own key
This doesn’t use “symmetric encryption” where
there is one passphrase to decrypt/encrypt all
files.
We maintain a keyring of:
● Each person that should have access.
● A key for the Puppet master.

19. Indoctrinate a new user:
1. New user does this:
●●
●
(Currently a doc, not a script. Patches gladly accepted.)

20. Indoctrinate a new user:
2. Existing admin does this:

21. Demo: Edit a file

22. Demo: Edit a file

23. Demo: Edit a file

24. Demo: Edit a file

25. Demo: Edit a file

26. Demo: Edit a file

27. Demo: Edit a file

30. Code is open source as of 7/2014
● Entirely written in bash.
● MIT License.
● Download it now:
○ https://github.com/StackExchange/blackbox

31. In the project’s first 9 months:
StackExchange/ServerFault has eliminated
plaintext secrets in our Puppet git repo.
● 7 SREs+Devs sharing the repo securely.
● 50+ files now stored encrypted.
○ Mostly SSL certs and SSH private keys.
● 40+ individual passwords/API keys:
○ Everything from SNMP communities, SaaS API
keys, and many many passwords.

32. Future plans
❏ Open source scripts.
❏ More usability enhancements.
❏ Better setup documentation.

33. Join the open source project
http://github.com/StackExchange/blackbox

34. Q&A
URLs from this talk:
https://github.com/StackExchange/blackbox
EverythingSysadmin.com

35. Shameless plug
Pre-order now! Save 35%
Ships in September.
informit.com/TPOSA
Discount code TPOSA35
Read “rough cuts” today:
safaribooksonline.com

36. Q&A
URLs from this talk:
https://github.com/StackExchange/blackbox
EverythingSysadmin.com
the-cloud-book.com
informit.com/TPOSA (code TPOSA35)

37. Why didn’t we use eyaml?
● Easier transition. No Puppet code changes
for big files like SSL certs.
● Faster. Zero run-time performance impact
on master.
● eyaml didn’t exist when we started.