DevOps brings the potential of faster time to market and higher quality software applications. But to accelerate adoption or realize its full benefit, organizations need to adapt to the policy, staff, and technology changes that inherently accompany it.
Join us for an educational webinar that examines DevOps from an implementation and risk perspective and how to minimize organizational impact.
Topics include:
* What’s the difference? DevOps vs. other development approaches
* Third-party risk: COTS, open source, and cloud
* Implications of accelerated development and automation
* Making room for DevOps: organizational changes
2. 2
• Securing software in all the challenging places….
• ….while helping clients get smarter
Assessment: show me the gaps
Standards: set goals and make it easy
Education: help me make good decisions
Over
3 Million
Users
Authored
18
Books
Named
6x
Gartner MQ
About Security Innovation
3. 3
• CEO by day; engineer by trade (and heart)
• Mechanical Engineer, Software Engineer
• Ponemon Institute Fellow
• Privacy by Design Ambassador, Canada
• In younger days, built non-lethal weapons
systems for Federal Government
About Me
4. 4
Agenda
DevOps vs. Traditional Approaches
DevSecOps in Practice
Managing / Making Room for DevOps
5. 5
DevOps vs. DevSecOps
• DevOps
Set of practices that automates the processes between development and IT operations
teams to build and release software quickly and consistently
• DevSecOps
Improved collaboration, streamlined feedback, and faster cycles to address security issues
6. 6
• DevOps includes the same ol’ security engineering activities
• Harden design, reduce attack surface, code defensively
• Perform Code Reviews, Pen Tests, & Deployment Reviews
• However…
• Done more iteratively with an emphasis on early stage
• Introduce more automation and different toolsets
• Security is everyone’s job, roles more blurred
• Which means you need
• Clearly defined policies and easy-to-audit configurations
• Ability to scale or rollback configurations when required
• Cross functional skills and normalized communication language
Familiar Friend
7. 7
Treating Everything as Code
• With “Infrastructure as Code” think of everything in a software-defined way
• Servers (usually VMs and/or cloud instance)
• Containers
• Application stacks
• “Networks”
• Roles/Privileges/Access models
• Security needs to be defined in this way, too
• More visible, easier to audit, simpler to reproduce, quicker to find issues
• Hardened Dockerfiles, Chef cookbooks, digital certificates, hardening scripts, firewall
configurations, Cloud-Config files, user permissions
9. 9
Planning
• Similar to any security planning
• Threat Modeling, policy development, security assurance models
• Sets the stage for integrating security into deployment pipeline
• Determine tooling
• Test acceptance criteria (note: you need security defect classification)
• Give guidance to development and operations teams
• Challenge is to inject enough security but don’t obstruct
• Risk to overwhelm with unnecessary false positives
• Integrate key (and well-defined) checkpoints specific and sufficient to each phase
Tip: lack of proper planning in DevOps inability to execute; slower throughput
10. 10
Coding Phase
• Ensure developers are working in secure environments
• Virtual Machines, containers, VPNs, etc expand your attack surface
• Leverage tools to find and prevent common security mistakes
• Know how to handle passwords, API keys, private certificates, and other secrets
• At code commit, focus checks on most severe security issues
• e.g. hard-coded secrets or API info developers may have left in the code
• Determine defect saturation that “breaks the build” (need plan/policy for this)
11. 11
Build
• Check for vulnerabilities in 3rd party components and dependencies
• Software composition analysis useful here
• 70-90% of enterprise applications are 3rd party and/or open-source software
• Break the build for high risk publicly disclosed 3rd party vulnerabilities?
• Good time to gather metrics and digitally sign/store build-time artifacts
• Tip: policies around what’s a “high risk” vulnerability are important
• Did I mention defect classification 2 slides ago?
12. 12
Test
• At this point obvious bugs were found with automated tests … right?
• Manual checks for critical vulnerabilities ID’s in threat model(s)
• Tools:
• Static Analysis Security Testing (SAST) – scans for bad coding errors
• Dynamic application security testing (DAST) - tests app in “deployed” environment
• Interactive application security testing (IAST) - interacts with the app UI controls and APIs
• Fuzzing: feed irregular characters and data lengths to identify error handling flaws
• Also test OS/cloud configurations, databases, hardening scripts, certificates, etc.
• Over time, most testing will be verifying and documenting policy compliance
13. 13
Deploy
• Once application is approved for deployment, release team green-lights
• Security team performs deployment-related security steps:
• Create or update “secrets” – check for leaks and clues left
• Document what was updated and by whom
• Produce compliance documentation and final security scans
• Perform final configuration and hardening
• Determine authenticity (e.g. code signing) and versions of 3rd party apps/components
• Create communication ports matrix so DevOps can configure firewall and spot unusual traffic
• Tip: while there are generally accepted deployment checks, they need to be
contextual to your environment and data
14. 14
Operate & Monitor
• Continuous feedback loops are what makes DevOps effective
• In addition to traditional bug reports, error logs, and telemetry, include:
• Pen testing reports (these may look different than traditional pen tests)
• Monitoring metrics
• User-reported issues
• Vulnerability reports, path and update notifications
• Intrusion detection, antivirus, and SIEM alerts
• Tip: this is why policy is good
16. 16
Make Way for DevOps
Need to consider changes to:
• 3rd-party Software
• Staff Skills
• Governance, Policies, and Standards
• Risk Assessment
• Change Management/Control
• Shifting Security Left
• Security as Code
17. 17
COTS and OSS: Co-Owning the Software
Still need documented steps and standards to follow along properly
SLA SLA
Configure (cloud
services/infrastructure)
Your data
OSS/GitHub
Buy
18. 18
Assessing 3rd party/COTS software
• Can’t just rely on patching. Anticipate deployed scenario weaknesses
• You don’t have access to source code to fix vulnerabilities
• This makes risk mitigation tedious … but achievable
• Cloud
• Wonderful scale and “turn-key” operations
• Loads of (new) deployment configuration issues to consider
• Info gathered from assessments…
• Improve the supply chain procurement processes
• Inform the supply chain risk management process
• Serve as foundation for additional testing or SLA conversations
19. 19
DevSecOps Requires New Skills Focus
Coding skills
Orchestration/automation tools
Secrets management
Hardening systems
Privilege control
Web/micro services
APIs (often RESTful and lightweight)
Scalable Cloud Infrastructure (software-
defined environments)
Virtualization/containers
Hypervisor lockdown
Virtual and software-defined networking
Container lockdown and scanning
Ruby, Python, React, Shell Scripting, etc
20. 20
Sample Skills
While each role needs to understand fundamentals of secure development, privacy
protection, and cloud infrastructure, additional job specific training is needed
Product Owners • Mitigate risk in supporting the application
• Safeguard data based on application, infrastructure and operating environment
Cloud Architect • Securely design application deployed on cloud infrastructure
• Analyze implementation options for core features
• Analyze configuration options for architectural frameworks, API gateways, microservices
Code Release
Manager
• Detect and mitigate insecure interfaces and APIs
• Prevent common vulnerabilities
• Recognize insufficient identity controls and insecure or out-of-date dependencies
Site Reliability
Engineers (SREs)
• Understand how adverse events like Denial of Service (DoS) attacks affect availability
• Implement application whitelisting and apply least privileged access to data
Automation
Engineers
• Leverage automation without compromising security
• Conduct vulnerability assessments
21. 21
Key to Staff Competency
• As roles bleed into one another through collaboration, individuals
need wider breadth of knowledge
• Risk watering down deep technical competence in the trenches
Prevent Jack of All
Trades
• Create a core set of common knowledge
• Security principles, regulatory concerns, infrastructure issues,
commonly used tools
Don’t make everyone
security experts
• Developers today are using open-source libraries, spinning up
servers, containers, and otherwise self-provisioning infrastructure
• Training often focused just on “writing” secure code
Consider evolving
Developer role
22. 22
Accelerating Success with DevOps Security Champions
• InfoSec team can help, but not scalable
• Practitioners that serve as mentors
• Determine when to engage the security team
• Single point of contact w/in their group
• Clearly define goals and responsibilities
• Conduct and/or verify security reviews
• Guard and promote “best practices”
• Raise issues for risks in existing/new software
• Build threat models for new features
• Should be knowledgeable (and passionate)
about software engineering
Gartner Report: 3 Steps to Integrate Security Into DevOps:
https://web.securityinnovation.com/gartner-report-devops
23. 23
Good Security Hygiene Begins with Good Policy
• There are many aspects of governance to consider
• Security policies
• Procurement and risk assessment practices
• Change management and control
• What goes into a policy?
• Acceptable cloud deployment models/practices
• Data types that can and cannot go to the cloud
• Key/credential/data protection requirements
• APIs documentation
• Logging and monitoring
• Compliance requirements
24. 24
Ensure Policies are Tailored for….
DevOps
Network, Infrastructure, and Software security
Especially secure build/dev standards
Cloud platforms
How Amazon IAM users/groups should be configured
Proper use, distribution and storage of cloud API keys
COTS supply chain
Acceptance policy and culpability
Eliminate need for inspection on a mass basis by ensuring
quality in each component (built or supplied)
25. 25
Risk Assessment
• At each checkpoint perform checks applicable to that stage
• As deployment progresses, checks evolve from simple automated pattern
matching to more complex human-driven
• e.g., DAST scan Penetration Testing / Red Teaming
• Can be run in parallel as you cycle through the DevOps “figure 8”
• Procurement / Risk Assessment
• Early: static and dynamic code analysis
• Early: defined libraries and configs
• Later: monitoring and control in instances
• Later: SLA-based acceptance testing
26. 26
Change Control
• Too heavy-handed sign-off process slows speed to release
• Micro change management; policy driven; importance of
planning
Traditional change
control often falls apart
in a DevOps model
• Which changes need traditional tracking
• Which changes can be “log and review”
• Review cycles and review board groups
• Daily defect triage teams with filtered/streamlined reviews
Security professionals
need to work with
DevOps to determine:
27. 27
Hardening Design
• Define policies for components, networks, and more
• Configurations (Puppet, Chef)
• App deployment and automation (Ansible, Jenkins)
• Orchestration, automation and cloud provider tools
• These will be specific use cases and requirements:
• Input validation for app X
• Use of TLS for all communications
• Hardening to CIS Benchmark standards
• Implement in code or via policy files
• Threat models developed early useful here
28. 28
Security as Code
• Testing tools can simplify policy validation
• Coordinate penetration tests and routine checks
to validate policies’ effectiveness
• Are only required ports open?
• Are credentials secured?
• Are encryption keys secured?
• Are privileges assigned properly?
• Re-usable test scripts to validate no security regressions
• Also aids with consistency; quickly ID’s undocumented change
29. 29
Summary
• Secure DevOps solves some plaguing issues:
• Lack of cross-functional knowledge between build and deployed state
• Security slowing down ability to get features out the door
• Streamlines communications from the customer to the product owner
• Success of DevOps relies on 3 key principles
• Automate what can be done faster than humans
• Understand where manual inspection is needed
• Balance tools, knowledge, and process – can’t rely too heavily on one
30. 30
How Can We Help?
DevOps/SDLC Risk Review
• Fill compliance gaps with tools,
activities and skills
• Roadmap with optimal sequencing
Computer Based Training
• Specific to DevOps roles
• Covers all major technologies,
roles, frameworks
Cyber Range
• Turn-key, fun
• Automated scoring
• Real-world applications,
platforms, systems
31. 31
Cyber Range: Build Cross-Functional Security Skills
• Experts predict a shortage of 3.5 million cybersecurity professionals for 2021*
• Hard to build a strong DevOps outfit without expertise to support it
• Quickly and accurately identify security champions
*https://www.csoonline.com/article/3200024/security/cybersecurity-labor-crunch-to-hit-35-million-unfilled-jobs-by-2021.html
Commercial Ranges Free Range: InstaFriends
Community.securityinnovation.com
DevOps has long proven to be an effective approach to speeding up development cycles and improving reliability, along with many other technical, business, and cultural benefits. Those same qualities of DevOps, that improve software development and deployment, can also improve your application’s security. Integrating security into your DevOps pipeline can provide a united culture between the developer, operations, and security teams. Integrating security into your DevOps pipeline can also provide improved feedback loops and faster cycles for addressing security issues, better collaboration to make all players involved with security, and repeatable, automated, and easy-to-audit configurations.
The planning stage for DevSecOps is similar to any security planning; threat modeling, policy development, and security assurance models. But in a DevSecOps model, planning is where you set the stage for integrating security into the deployment pipeline. This includes determining tooling, test acceptance criteria, and providing guidance and education to developers and operations teams. Planning how you integrate DevSecOps into the deployment pipeline will set the stage for the entire project. This can be the most difficult part because you need to inject enough security to be effective at the same time it shouldn't be an obstruction to deployment schedules, nor should you overwhelm other teams with unnecessary false positives. The safest way to integrate security is to spread it out across the entire pipeline so each stage has a natural and incremental security checkpoint..
The coding stage provides many opportunities for introducing and automating security. First is the environment the developers use for writing code. As you move security left on the DevOps pipeline you should make sure that developers are working in secure environments. Virtual machines, Docker containers, VPNs, and user desktop machines all have the potential for expanding the attack surface and increasing risk and should be secured accordingly. Next is the tooling you provide to developers to identify common security mistakes. This tooling might include IDE add-ons, linters, and similar tools. These tools typically do not catch complex logic errors but are well-suited for spotting common vulnerability patterns. The final consideration is how developers handle secrets such as passwords, API keys, or private certificates.
. At this stage you still want to focus on the most serious issues, but at build time you may perform additional test such as checking for vulnerabilities in third-party components and dependencies or performing more in-depth risk-based checks. You don’t need to break the build unless you encounter serious vulnerabilities, a large number of vulnerabilities, or publicly disclosed issues with third-party components. For all other issues you can simply trigger alerts for the security team.
The testing phase is your last chance to identify security vulnerabilities before the code goes into production. Unlike waterfall or other development methodologies, DevOps is meant to allow for rapid and continuous deployments. Some organizations may push dozens of builds for deployment every day. With these fast deployment cycles, you don’t want security checks to delay deployment, nor do you want to overwhelm your security team with a never-ending stream of builds to test. This is why all testing up to this point should be automated and only break a build for the most serious issues. For all other issues, you can address them through your normal feedback loops, such as filing a bug report and alerting the security team. On the other hand, you also don’t want to neglect more in-depth testing so you should perform manual testing periodically or when there are concerning automated test results. At this point in deployment, many obvious bugs should have been identified with earlier automated tests, over time existing code will become more secure, and vulnerabilities reported from other sources will already have been documented. Therefore, much of the testing in this phase will be to continuously verify and document policy compliance. Nevertheless, there is always room for advanced testing and to address emerging threats. For this stage of testing, SAST tools should check for a broader range of vulnerabilities and any unit tests should address any additional items identified and your threat modeling. Here you would also want to make use of additional classes of tools such as: Dynamic application security testing (DAST), which are tools that test applications in a running state and covers the full execution environment. Interactive application security testing (IAST), automated tools that interact with the application’s user interface controls and API models to identify vulnerabilities. Fuzzing, tools that feed irregular characters and data lengths to identify common flaws with bounds checking and error handling. This testing phase should not only include application code, but also include: OS Configurations, databases and other back-end software, hardening scripts, other automation files, and certificates and certificate chains.
Once the application has passed testing and is approved for deployment, the security team will need triggers to perform any deployment-related security steps. These might include: Creating or updating any secrets necessary for deployment or operations, documenting what was updated and by whom, producing any compliance documentation and final security scan output as artifacts, performing any final configuration or hardening, and determining authenticity of any third-party applications or components. At this stage you should assume the deployment is approved for release. Any significant security issues should have been addressed in earlier
Continuous feedback loops are a big part of what makes DevOps so effective. In addition to the traditional bug reports, error logs, and telemetry that provide feedback to developers, integrated security provides additional sources of feedback include pen-testing reports, monitoring metrics, user-reported issues, and bug bounties. Also there will be vulnerability reports and update notifications for third-party components. Lastly there will be intrusion detection, antivirus, and security intelligence event management (SIEM) alerts. These feedback loops will take you back to the beginning of the pipeline where you will repeat the process again
Security and IT leaders need to lay out a detailed plan for how they’re
going to develop security skills relevant to every function
DevOps depends on an increased collaboration between IT roles and self- service. The challenge is as roles start to bleed into one another through deeper collaboration, individuals start to need a wider breadth of knowl- edge than ever before about how their actions impact the organization’s threat posture. Take developers, for example: so much of application security (APPSEC) training today is focused solely on secure coding techniques, with- out accounting for the reality that developers today are using open-source libraries, spinning up servers, containers, and otherwise self-provisioning the infrastructure their software is running on.
. For example, consider setting policies around how to the company should be using the Amazon’s IAM feature. Make specific policies around how IAM users and groups should be configured and used. Use the doctrine of least privilege and design a sane access control policy. Consider a policy where only a few trusted users have full admin access and other users are in a power users group where they can launch instances, but cannot alter AWS console configuration or access. You might want to give your accounting team limited access to the console as well to review billing and usage. While you are at it, develop specific policies around the use, distribution and storage of cloud API keys. Furthermore, consider enforcing two factor authentication. These kinds of specific use case policies will go a long way for security teams in gaining confidence with the DevOps teams.
DevOps might be the cool new hotness that is making waves in every kind of organization, but that doesn’t mean we can forget our information security requirements in a DevOps environment. One of the starting points in ensuring DevOps organizations follow good security practices is to start with a minimum set of security policies. Make sure you champion these policies and tailor them to your specific organization and use case. With a little luck and determination even your DevOps teams can follow security best practices.