Book Club: The DevOps Handbook (Chapter 3. The Second Way: The Principles of Feedback)

The following is a chapter summary for “The DevOps Handbook” by Gene Kim, Jez Humble, John Willis, and Patrick DeBois for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Background on The DevOps Handbook

More than ever, the effective management of technology is critical for business competitiveness. For decades, technology leaders have struggled to balance agility, reliability, and security. The consequences of failure have never been greater―whether it’s the healthcare.gov debacle, cardholder data breaches, or missing the boat with Big Data in the cloud.

And yet, high performers using DevOps principles, such as Google, Amazon, Facebook, Etsy, and Netflix, are routinely and reliably deploying code into production hundreds, or even thousands, of times per day.

Following in the footsteps of The Phoenix Project, The DevOps Handbook shows leaders how to replicate these incredible outcomes, by showing how to integrate Product Management, Development, QA, IT Operations, and Information Security to elevate your company and win in the marketplace.

The DevOps Handbook

Chapter 3

The Second Way describes the principles that enable the fast and constant feedback at all stages of the value stream. In technology, work happens almost entirely within complex systems with a high risk of catastrophic consequences. As in manufacturing, problems are often discovered only when large failures are underway, such as a massive production outage or a security breach resulting in the theft of customer data.

Working Safely Within Complex Systems

The goal is to make the system of work safer by creating fast, frequent, high-quality information flow throughout the value stream and organization, which includes feedback and feedforward loops. One of the defining characteristics of a complex system is that it defies any single person’s ability to see the system as a whole and understand how all the pieces fit together.

Four conditions to help make complex systems safe:

• Complex work is managed so that problems in design and operations are revealed.
• Problems are swarmed and solved, resulting in quick construction of new knowledge.
• New local knowledge is exploited globally throughout the organization.
• Leaders create other leaders who continually grow these types of capabilities.

See Problems As They Occur

Our goal is to increase information flow in the system from as many areas as possible with as much clarity between cause and effect as possible. In the technology value stream, poor outcomes arise because of the absence of fast feedback. When feedback is delayed and infrequent, it’s too slow to prevent undesirable outcomes. Fast feedback can be enabled with the creation of automated build, integration, and test process.

Feedback loops enable quick detection and recovery of problems, as well as how to prevent these problems from occurring again in the future. Doing this increases the quality and safety of the system of work and creates organizational learning.

Swarm and Solve Problems To Build New Knowledge

Swarming is necessary for the following reasons:

• It prevents the problem from progressing downstream (cost and effort to repair it increases downstream and technical debt accumulates).
• It prevents the work center from starting new work, which will likely introduce new errors into the system.
• If the problem is not addressed, the team could have the same problem in the next cycle or iteration, which requires more fixes and work.

Keep Pushing Quality Closer To The Source

In complex systems, adding more inspection steps and approval processes actually increases the likelihood of future failures. The effectiveness of approval processes decreases as decision-making is pushed further away from where the work is performed.

Examples of Ineffective Quality Controls:

• Requiring another team to complete tedious, error-prone, and manual tasks that could be easily automated and run as needed by the team who needs the work performed.
• Requiring approvals from busy people who are distant from the work, forcing them to make decisions without an adequate knowledge of the work or the potential implications, or to merely rubber stamp their approvals.
• Creating large volumes of documentation of questionable detail which become obsolete shortly after they are written.
• Pushing large batches of work to teams and special committees for approval and processing and then waiting for responses.

Enable Optimizing For Downstream Work Centers

Lean defines two types of customers that to design for: (1) the external customer (who most likely pays for the service being delivered) and (2) the internal customer (who receives and processes the work immediately after the development team).

According to Lean, the most important customer is the one next step downstream.

Creating fast feedback is critical to achieving quality, reliability, and safety in the technology value stream. Fast feedback is achieved by seeing problems as they occur, swarming and solving problems to build new knowledge, pushing quality closer to the source, and continually optimizing for downstream work centers.

Series Navigation