Slides from presentation by Alex Honor and Damon Edwards at DevOps Connect at RSA 2015 in San Francisco on April 20, 2015. Abstract: IT organizations are feeling the squeeze from seemingly conflicting business mandates. At one moment the message is “Go Go Go. DevOps, Lean Startup, Continuous Delivery… move faster and give more people access”. The next moment the message is “Be more secure. Compliance above all. Keep us out of the press!”. Damon Edwards and Alex Honor work with many enterprises who are facing these challenges. This talk is an in the trenches view of how these companies are responding and learning to go faster and be more secure.

1. Go Fast AND Be Secure?
DevOps and Security from an Enterprise
Toolsmith’s Perspective
Alex Honor
Damon Edwards

2. @damonedwards
Damon Edwards Alex Honor
@alexhonor

3. DevOps Consulting
Automation Design
Operations
Tools

4. Business Demands
Our #1 priority is moving faster
than our competitors!

5. IT Responds

6. IT Responds

7. IT Responds

8. … but what about security and
compliance?

9. Business Demands
Our #1 priority is moving
faster than our competitors!
Our #1 priority is security and
compliance!and

10. IT Under Pressure

11. Can we go faster and be more secure?

12. Can we go faster and be more secure?

13. What gets in the way?

14. Everything is different

15. Everything is different
● Many servers hand built

16. Everything is different
● Many servers hand built
● Custom is the rule

17. Everything is different
● Many servers hand built
● Custom is the rule
● Inconsistent access control
policy and rules

18. Everything is different
● Many servers hand built
● Custom is the rule
● Inconsistent access control
policy and rules
● Network spaghetti topology
reflects snowflakes

19. Everything is different
● Many servers hand built
● Custom is the rule
● Inconsistent access control
policy and rules
● Network spaghetti topology
reflects snowflakes
● … it’s always a network
problem ;-)

20. Multiplied by Datacenter
● Geographically spread
● Generations of
hardware & software
● WAN latencies and
bandwidths
● Sometimes outsourced

21. Culture clashes between silos

22. Culture clashes between silos
● “Too much change breaks
stuff” - Ops

23. Culture clashes between silos
● “Too much change breaks
stuff” - Ops
● “Let me do it myself” - Dev

24. Culture clashes between silos
● “Too much change breaks
stuff” - Ops
● “Let me do it myself” - Dev
● “This is dangerous!” - Sec

25. Culture clashes between silos
● “Too much change breaks
stuff” - Ops
● “Let me do it myself” - Dev
● “This is dangerous!” - Sec
● “It’s not ready” - QA

26. Culture clashes between silos
● “Too much change breaks
stuff” - Ops
● “Let me do it myself” - Dev
● “This is dangerous!” - Sec
● “It’s not ready” - QA
● Finger pointing - everyone

27. Bureaucracy to get anything delivered
“Have you got 27B-6?”
- said a guy, in a downstream
silo
“I’m a bit of a stickler
for paperwork”
“All I need is a
ACL/VIP/etc”

28. It always ends up an escalation
● Who yells loudest
● Cube driveby and
who you know
● Crisis at deadline
or outage
● Sometimes still a
rubber stamp

29. Hard to see how delivery work gets done
across the organization

30. Process Islands Multiple
Development
teams out here
somewhere

31. Process Islands
“I know there are
problems delivering, not
sure where, but I know
they are outside my
island of control”
“We all have the
best intentions
from our
perspective

32. Process Islands

33. Process Islands

34. Process Islands

35. Process Islands
I really wish to
deploy multiple
times daily
Friday evening

36. Process Islands
Monday morning

37. Process Islands

38. Process Islands
Everybody on
bridge call with
the boss

39. Complicated and self inflicted
● Left hand doesnt know
what the right hand
doing
● “Bandaids” and
“exception is the rule”
● Telephone and Tribal
knowledge
● Low MTTD/MTTR

40. How do we know when things are getting
any better?

41. You’ll know you are better when...

42. You’ll know you are better when...
● Security policy is applied reliably and consistently

43. You’ll know you are better when...
● Security policy is applied reliably and consistently
● Security isn’t the bottleneck

44. You’ll know you are better when...
● Security policy is applied reliably and consistently
● Security isn’t the bottleneck
● An audit trail is easy to pull together

45. You’ll know you are better when...
● Security policy is applied reliably and consistently
● Security isn’t the bottleneck
● An audit trail is easy to pull together
● Security engineers aren’t left out until the end of the party (or
never consulted)

46. You’ll know you are better when...
● Security policy is applied reliably and consistently
● Security isn’t the bottleneck
● An audit trail is easy to pull together
● Security engineers aren’t left out until the end of the party (or
never consulted)
● Everyone has the control they need (without root)

47. You’ll know you are better when...
● Security policy is applied reliably and consistently
● Security isn’t the bottleneck
● An audit trail is easy to pull together
● Security engineers aren’t left out until the end of the party (or
never consulted)
● Everyone has the control they need (without root)
● Nobody feels like they are having the rug pulled out from
underneath them

48. Shift left: Host OS SDLC
Collaborate with
source code
Artifacts move through
the “supply chain”

49. Bastion host

50. Bastion host
● centralized access
point for authorized
access

51. Bastion host
● centralized access
point for authorized
access
● disallow home run
connections

52. Bastion host
● centralized access
point for authorized
access
● disallow home run
connections
● dispatcher interfaces
remote execution layer

53. Bastion host
● centralized access
point for authorized
access
● disallow home run
connections
● dispatcher interfaces
remote execution layer
● hides network
complexity like jump
boxes per DC

54. Bastion host
● centralized access
point for authorized
access
● disallow home run
connections
● dispatcher interfaces
remote execution layer
● hides network
complexity like jump
boxes per DC

55. User traceability: Delegate account
● User logs in as himself to bastion host
● Remote commands and processes run
under a service account
● Eg, SSH keys used for delegate account
identity

56. User traceability: End to end
● User logs in as himself to bastion host
● Remote commands executed using
same user account
● Eg., User may raise privilege via sudo

57. White List and Wrapper
● No ad-hoc interactive logins.
● Use wrapper script and a white list
● Escalate privilege with sudo
● Not foolproof!
SELinux still considered too hard for most
eg.: ssh forced command (~/.ssh/
authorized_keys: command=wrapper.sh
and $SSH_ORIGINAL_COMMAND)

58. Leverage the toolchain to enforce policy

59. Leverage the toolchain to enforce policy
Design
and code
reviews

60. Leverage the toolchain to enforce policy
Design
and code
reviews
Code and
binary scans

61. Leverage the toolchain to enforce policy
Design
and code
reviews
Code and
binary scans
“Bake” security tests into
your “immune system”

62. Leverage the toolchain to enforce policy
Design
and code
reviews
Code and
binary scans
“Bake” security tests into
your “immune system”
Component vulnerability and
governance

63. Leverage the toolchain to enforce policy
Design
and code
reviews
Code and
binary scans
“Bake” security tests into
your “immune system”
Component vulnerability and
governance
Access policy and
operational security checks

64. Automate Evidence Collection for Audits

65. Automate Evidence Collection for Audits
What’s the
change?

66. Automate Evidence Collection for Audits
What’s the
change?
How did you
validate the
change?

67. Automate Evidence Collection for Audits
What’s the
change?
How did you
validate the
change?
How was the
change
distributed?

68. Automate Evidence Collection for Audits
What’s the
change?
How did you
validate the
change?
How was the
change
distributed?
Who did
what when
and where?

69. Automate Evidence Collection for Audits
What’s the
change?
How did you
validate the
change?
How was the
change
distributed?
Who did
what when
and where?
What
executed on
the node?

70. Summary
● Shift left
● Bastion host
● User traceability
● White lists and wrappers
● Leverage the toolchain to enforce policy
● Automate evidence collection for audits

71. ● Shift left
● Bastion host
● User traceability
● White lists and wrappers
● Leverage the toolchain to enforce policy
● Automate evidence collection for audits
● ?
Summary