Book Club: The DevOps Handbook (Chapter 7. How to Design Our Organization and Architecture with Conway’s Law in Mind)

The following is a chapter summary for “The DevOps Handbook” by Gene Kim, Jez Humble, John Willis, and Patrick DeBois for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Background on The DevOps Handbook

More than ever, the effective management of technology is critical for business competitiveness. For decades, technology leaders have struggled to balance agility, reliability, and security. The consequences of failure have never been greater―whether it’s the healthcare.gov debacle, cardholder data breaches, or missing the boat with Big Data in the cloud.

And yet, high performers using DevOps principles, such as Google, Amazon, Facebook, Etsy, and Netflix, are routinely and reliably deploying code into production hundreds, or even thousands, of times per day.

Following in the footsteps of The Phoenix Project, The DevOps Handbook shows leaders how to replicate these incredible outcomes, by showing how to integrate Product Management, Development, QA, IT Operations, and Information Security to elevate your company and win in the marketplace.

The DevOps Handbook

Chapter 7

Conway’s Law: “organizations which design systems… are constrained to produce designs which are copies of the communication structures of these organizations…The larger an organization is, the less flexibility it has and the more pronounced the phenomenon.”

How teams are organized has a powerful effect on the software produced as well as resulting architectural and production outcomes.

Conway’s Law Example: Sprouter at Etsy. Sprouter is shorthand for stored procedure router. Sprouter resided between their front-end PHP application and the Postgres database, centralizing access to the database and hiding the database implementation from the application layer.

“Sprouter was designed to allow the Dev teams to write PHP code in the application, the DBAs to write SQL inside Postgres, with Sprouter helping them meet in the middle.”

Etsy case study, Chapter 7

The problem with Sprouter? Any changes to business logic meant the DBAs had to write a new stored procedure. Developers creating new functionality had a dependency on the DBA team; required prioritization, communication, and coordination.

Etsy changed by investing into site stability, having developers perform their own deployments into production, and the retirement of Sprouter.

They wrote a PHP Object Relation Mapping (ORM) layer that allowed front-end developers to make calls directly to the database and reduced the number of teams required to change business logic (from 3 teams to 1 team).

Organizational Archetypes

Three primary types of organization structures for DevOps value streams: functional, matrix, and market.

Functional-oriented Organizations optimize for expertise, division of labor, or reducing cost. Characterized by centralized expertise, which helps enable career growth and skill development, as well as tall hierarchical organizational structures.

Matrix-oriented Organizations attempt to combine functional and market orientation. Matrix organizations can result in complicated organizational structures, such as individual contributors reporting to two managers or more.

Market-oriented Organizations optimize for responding quickly to customer needs.
These organizations tend to be flat and composed of multiple, cross-functional disciplines.

Problems Often Caused By Overly Functional Orientation (“Optimizing For Cost”)

Typical problems include:

• Teams are organized by specialties, which leads to long lead times as groups coordinate work handoffs.
• Teams are in a creativity and motivation vacuum because they can’t see the bigger picture.
• When each Operations area must serve multiple value streams who compete for their cycles, the troubles are exacerbated.
• Characterized by long queues & long lead times with poor handoffs & lots of rework.
• Issues scaling with more frequent releases.

Enable Market-Oriented Teams (“Optimizing For Speed”)

Typical aspects of market-oriented teams include:

• Many small teams working safely and independently, quickly delivering value to the customer.
• Market-oriented teams are responsible not only for feature development, but also for testing, securing, deploying, and supporting their service in production, from idea conception to retirement.
• Cross-functional and independent; design, build & deliver new features, deploy & run their service in production, and fix defects with minimal dependencies.
• Embedded functional engineers and skills (Ops, QA, Infosec) into each service team.

Making Functional Orientation Work

Achievement of DevOps goals is possible with functional orientation so long as everyone in the value stream views customer and organizational outcomes as the shared goal.

Many of the most admired DevOps organizations retain functional orientation of Operations, including Etsy, Google, and GitHub.

“As tempting as it seems, one cannot reorganize your way to continuous improvement and adaptiveness. What is decisive is not the form of the organization, but how people act and react.”

Mike Rother, Toyota Kata

Testing, Operations, and Security As Everyone’s Job, Every Day

In high-performing organizations, everyone within the team shares a common goal—quality, availability, and security aren’t the responsibility of individual departments but are a part of everyone’s job, every day.

Shared pain can reinforce shared goals. Facebook Example: In 2009 they had problems related to code deployments. In a meeting full of Ops engineers where someone asked that all people not working on an incident close their laptops, and no one could.

The solution: all Facebook engineers, engineering managers, and architects rotate through on-call duty for the services they built.

Enable Every Team Member To Be A Generalist

When departments over-specialize, it causes siloization.

Siloization: when departments operate more like sovereign states. Complex operational activity then requires multiple handoffs and queues between the different areas of the infrastructure, leading to longer lead times.

Don’t create specialists who are “frozen in time,” only understanding and able to contribute to that one area of the value stream. Instead, provide opportunities to learn all skills necessary to build and run the systems teams are responsible for; this includes regularly rotating roles.

When people are valued merely for their existing skills or performance in their current role rather than for their ability to acquire and deploy new skills, the organization has what Dr. Carol Dweck describes as the fixed mindset, where people view their intelligence and abilities as static “givens” that can’t be changed in meaningful ways.

Encourage learning and ensure people have relevant skills & a defined career road map. This fosters a “growth mindset”. The organization must provide training and support to create a sustainable team.

Fund Not Projects, But Services and Products

Another way to enable high-performing outcomes is to create stable service teams with ongoing funding to execute their own strategy and road map of initiatives.

The more traditional model has Development and Test teams assigned to a “project” and then reassigned to another project as soon as the project is completed, and funding runs out.

The goal of a product-based funding model is to value the achievement of organizational and customer outcomes, such as revenue, customer lifetime value, or customer adoption rate, ideally with the minimum of output.

Design Team Boundaries In Accordance With Conway’s Law

As organizations grow, one of the largest challenges is maintaining effective communication and coordination between people and teams. Collaboration is also impeded when the primary communication mechanisms are work tickets and change requests.

Software architecture should enable small teams to be independently productive, sufficiently decoupled from each other so that work can be done without excessive or unnecessary communication and coordination.

Create Loosely-Coupled Architectures To Enable Developer Productivity and Safety

With tightly-coupled architecture, small changes can result in large scale failures. The result is not only long lead times for changes (typically measured in weeks or months) but also low developer productivity and poor deployment outcomes.

Architecture that enables small teams of developers to independently implement, test, and deploy code into production safely and quickly, can increase and maintain developer productivity and improve deployment outcomes. Having architecture that is loosely-coupled means that services can update in production independently, without having to update other services.

Keep Team Sizes Small (The “Two-Pizza Team” Rule)

As part of its transformation initiative away from a monolithic code base in 2002, Amazon used the two-pizza rule to keep team sizes small—a team only as large as can be fed with two pizzas—usually about five to ten people.

Small Team Size Benefits:

1. Teams have a clear, shared understanding of the system they’re working on.
2. Limits the growth rate of the product or service being worked on.
3. Decentralizes power and enables autonomy.
4. Employees gain leadership experience in an environment where failure does not have catastrophic consequences.

Series Navigation