Book Club: The DevOps Handbook (Chapter 22. Information Security as Everyone’s Job, Every Day)

The following is a chapter summary for “The DevOps Handbook” by Gene Kim, Jez Humble, John Willis, and Patrick DeBois for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Background on The DevOps Handbook

More than ever, the effective management of technology is critical for business competitiveness. For decades, technology leaders have struggled to balance agility, reliability, and security. The consequences of failure have never been greater―whether it’s the healthcare.gov debacle, cardholder data breaches, or missing the boat with Big Data in the cloud.

And yet, high performers using DevOps principles, such as Google, Amazon, Facebook, Etsy, and Netflix, are routinely and reliably deploying code into production hundreds, or even thousands, of times per day.

Following in the footsteps of The Phoenix Project, The DevOps Handbook shows leaders how to replicate these incredible outcomes, by showing how to integrate Product Management, Development, QA, IT Operations, and Information Security to elevate your company and win in the marketplace.

The DevOps Handbook

Chapter 22

The goal is to create & integrate security controls into the daily work of Development and Operations, so that security is part of everyone’s job, every day. Ideally the work will be automated and put into a deployment pipeline. Manual processes, acceptances, and approvals should be replaced with automated controls, relying less on separation of duties and change approval.

To integrate security, compliance, and change management:

• Make security a part of everyone’s job
• Integrate preventative controls into the shared source code repository
• Integrate security with the deployment pipeline
• Integrate security with telemetry to better enable detection and recovery
• Protect the deployment pipeline
• Integrate the deployment activities with the change approval processes
• Reduce reliance on separation of duty

Integrate Security Into Development Iteration Demonstrations

One of the goals is to have feature teams engaged with Infosec as early as possible, as opposed to primarily engaging at the end of the project. Invite Infosec to the product demonstrations at the end of each development interval so that they can better understand the team goals in the context of organizational goals, observe their implementations as they are being built, and provide guidance and feedback at the earliest stages of the project, when there is the most amount of time and freedom to make corrections.

“By having Infosec involved throughout the creation of any new capability, we were able to reduce our use of static checklists dramatically and rely more on using their expertise throughout the entire software development process.”

Justin Arbuckle, Chief Architect at GE Capital

Integrate Security Into Defect Tracking and Postmortems

Track all open security issues in the same work tracking system that Development and Operations are using, ensuring the work is visible and can be prioritized against all other work. InfoSec traditionally has the security vulnerabilities stored in a GRC (governance, risk, and compliance) tool.

Integrate Preventive Security Controls Into Shared Source Code Repositories and Shared Services

Add to the shared source code repository any mechanisms or tools that help ensure applications and environments are secure. What should be added? authentication and encryption libraries and services. Version control also serves as a omni-directional communication mechanism to keep all parties aware of changes being made.

Items to include in version control related to Security:

• Code libraries and their recommended configurations
• Secret management using tools such as Vault, credstash, Trousseau, Red October, etc.
• OS packages and builds

Integrate Security Into Deployment Pipelines

Prior state: security reviews were started after development was completed. The documentation would be given to Development and Operations, which would be completely un-addressed due to project due date pressure or problems being found too late in the SDLC.

Goal state: automate as many information security tests as possible so they run as part of the deployment pipeline. Security should provide both Dev and Ops with fast feedback on their work.

Ensure Security of the Application

Development testing focuses on the correctness of functionality or happy path, which validates user journeys where everything goes as expected, with no exceptions or error conditions. QA, Infosec, and Fraud practitioners will often focus on the sad paths, which happen when things go wrong, especially in relation to security-related error conditions.

Static analysis: this is testing performed in a non-runtime environment, ideally in the deployment pipeline. Typically, a static analysis tool will inspect program code for all possible run-time behaviors and seek out coding flaws, back doors, and potentially malicious code. Examples of tools include Brakeman, Code Climate, and searching for banned code functions.

Dynamic analysis: dynamic analysis consists of tests executed while a program is in operation. Dynamic tests monitor items such as system memory, functional behavior, response time, and overall performance of the system. Ideally, automated dynamic testing is executed during the automated functional testing phase of a deployment pipeline.

Dependency scanning: Another type of static testing that would normally perform at build time inside of a deployment pipeline involves inventorying dependencies for binaries and executables, and ensuring that these dependencies are free of vulnerabilities or malicious binaries.

Source code integrity and code signing: All developers should have their own PGP key, perhaps created and managed in a system such as keybase.io. All commits to version control should be signed —that is straightforward to configure using the open source tools git. All packages created by the CI process should be signed, and their hash recorded in the centralized logging service for audit purposes.

Ensure Security of Software Supply Chain

“We are no longer writing customized software—instead, we assemble what we need from open source parts, which has become the software supply chain that we are very much reliant upon.”

Josh Corman

Using commercial or open source libraries brings in vulnerabilities along with their functionality.

The 2015 Sonatype State of the Software Supply Chain Report on vulnerability had some noteworthy findings. For one, the typical organization relied upon 7,601 build artifacts and used 18,614 different versions. 7.5% of those components had known vulnerabilities, with over 66% of those vulnerabilities being over two years old without having been resolved. For open source projects with known vulnerabilities registered in the National Vulnerability Database, only 41% were ever fixed and required. Additionally, on average they took 390 days to publish a fix. For those vulnerabilities that were labeled at the highest severity, fixes required 224 days.

Ensure Security of the Environment

Environments should be in a hardened, risk-reduced state.

One approach is to generate automated tests to ensure that all appropriate settings have been correctly applied for configuration hardening, database security settings, key lengths, etc.

Integrate Information Security Into Production Telemetry

Internal security controls are often ineffective in successfully detecting breaches in a timely manner, either because of blind spots in monitoring or because no one in the organization is examining the relevant telemetry in their daily work.

Deploy the monitoring, logging, and alerting required to fulfill information security objectives throughout applications and environments, as well as ensure that it’s adequately centralized to facilitate easy and meaningful analysis and response.

Creating Security Telemetry in Applications

In order to detect problematic user behavior that could be an indicator or enabler of fraud and unauthorized access, create the relevant telemetry in applications:

• Successful and unsuccessful user logins
• User password resets
• User email address resets
• User credit card changes

Creating Security Telemetry in Environments

Create sufficient telemetry in test environments to detect early indicators of unauthorized access. Monitoring opportunities include:

• OS changes (in production or in build infrastructure)
• Security group changes
• Changes to configurations
• Cloud infrastructure changes
• Cross-site scripting attempts
• SQL Injection attempts
• Web server errors

“Nothing helps developers understand how hostile the operating environment is than seeing their code being attacked in real-time.”

Nick Galbreath, Director of Engineering at Etsy

Security Telemetry at Etsy:

• Abnormal production program terminations
• Database syntax error
• Indications of SQL injection attacks

Protect The Deployment Pipeline

If someone compromises the servers running deployment pipeline that has the credentials for the version control system, it could enable someone to steal source code. If the deployment pipeline has write access, an attacker could also inject malicious changes into version control repository and inject malicious changes into application and services.

Risks to CI/CD pipelines include:

• Developers introducing code that enables unauthorized access (mitigate through controls such as code testing, code reviews, and penetration testing)
• Unauthorized users gaining access to the code or environment (mitigated via controls such as ensuring configurations match known, good states, and effective patching)

In order to protect our continuous build, integration, or deployment pipeline, a mitigation strategy may include:

• Hardening continuous build and integration servers and ensuring they can be reproduced in an automated manner
• Reviewing all changes introduced into version control, either through pair programming at commit time or by a code review process between commit and merge into trunk, to prevent continuous integration servers from running uncontrolled code
• Instrumenting the repository to detect when test code contains suspicious API calls is checked in to the repository, perhaps quarantining it and triggering an immediate code review
• Ensuring every CI process runs on its own isolated container or VM
• Ensuring the version control credentials used by the CI system are read-only

Series Navigation