Book Club: The Phoenix Project (Chapters 30-35)

The following is a chapter summary for “The Phoenix Project” by Gene Kim for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Chapters 26-29 HERE

Background on the Phoenix Project

“Bill, an IT manager at Parts Unlimited, has been tasked with taking on a project critical to the future of the business, code named Phoenix Project. But the project is massively over budget and behind schedule. The CEO demands Bill must fix the mess in ninety days or else Bill’s entire department will be outsourced.

With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with a manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.

In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again.”

The Phoenix Project

Chapter 30

Bill joins Erik at MRP-8.

Erik reveals he was a Special Forces officer in the US Army.

“A manufacturing plant is a system. The raw materials start on one side, and a million things need to go just right in order for it to leave as finished goods as scheduled out the other side. Everything works together. If any work center is warring with the other work centers, especially if Manufacturing is at war with Engineering, every inch of progress will be a struggle.”

Erik

“takt time” is the cycle time needed in order to keep up with customer demand. If any operation in the flow of work takes longer than the takt time, you will not be able to keep up with customer demand. The feedback loop must go back to product definition, design, and development.

Erik points out the most profitable items made in the plant have the longest setup and process times. To solve the problem, they developed a machine that combined four work centers into one, eliminating over thirty manual, error-prone steps, completely automating the work cycle.

Bill is challenged to decrease IT’s changeover time and enable faster deployment cycle time. Erik expects 10 deployments a day.

“Dev and Ops working together, along with QA and the business, are a super-tribe that can achieve amazing things. . . Until code is in production, no value is actually being generated, because it’s merely WIP stuck in the system. He kept reducing the batch size, enabling fast feature flow. . . He automated the build and deployment process, recognizing that infrastructure could be treated as code, just like the application that Development ships.”

Erik

Jez Humble and Dave Farley codified the practices and principles that enable multiple deployments per day in the book “Continuous Delivery”.

Deployment Pipeline: entire value stream from code check-in to production. Everything in version control — not just code but everything required to build the environment. Automate the entire environment creation process.

Erik tasks Bill with moving Brent to a team that will automate the build process.

“Stop focusing on the deployment target rate. Business agility is not just about raw speed. It’s about how good you are at detecting and responding to changes in the market and being able to take larger and more calculated risks.”

Erik

Features are always a gamble. So the faster features are pushed to market and tested, the better off a company is at adapting.

Chapter 31

SWAT Team kick-off meeting led by Chris. Steve has authorized a small team to deliver the promotion functionality and do whatever it takes to make a positive impact on the holiday shopping season.

The chief concern is the deployment process and the way Parts Unlimited is building the environments.

Chris is shocked at the “ten deploys per day” request, but Patty thinks the idea is solid; they can deploy bug fixes and performance enhancements. The company could enable Marketing to make their own changes to content or business rules or enabling faster experimentation and A/B split testing.

Development and Operations start to generate all the steps needed to deploy a feature

Development examples: automated tests in Dev, creating QA environment that matches dev, deploying code into it, executing tests, deploying & migrating to a stage environment, load testing.

Operations examples: preparing new server instances, loading and configuring the operating system, databases, and applications’ making all the changes to the networks, firewalls, and load balancers.

Bill marks each step where IT had problems with deployments in the past; almost all steps are marked.

Patty creates a “Value Stream Map” on the whiteboard. She writes the time-commitment for each step and whether the step requires wait time.

Patty suggests focusing on environments and code packaging process.

Brent proposes creating a common automated script for building the dev, qa, and prod environment.

New requirement for IT: at the end of each Sprint, the deployable code AND the environment must be checked into version control.

New process for IT: someone is responsible for package creation and Dev handoff — generate and commit the packaged code to trigger an automated deployment into the QA environment.

“Brent, if it’s okay with you and everyone else, I’d like to invite you to our team sprints, so that we can get environment creation integrated into the development process as early as possible. At each three-week sprint interval, we not only need to have deployable code but also the exact environment that the code deploys into, and have that checked into version control, too.”

Chris

Chapter 32

Bill reflects on how different software engineers are today than when he was coming up through the ranks. They are more loose and less likely to follow process.

The SWAT Team calls themselves “Unicorn”. The objective is doing whatever it takes to deliver effective customer recommendations and promotions.

Project Unicorn had a code base completely decoupled from Phoenix.

The first challenge was to start analyzing customer purchase data. The team created a completely new database, using open source tools with data copied from Phoenix and order/inventory management systems.

Decoupling from other projects made changes easier and did not put other projects at risk.

Unicorn team developers were using the same OS, library versions, databases, database settings, etc.

Brent goes missing from the team for two days and can’t be contacted. Brent has been taken to a secret off-site to discuss breaking up the company. He believes a split will be a complete nightmare.

“Dick and the finance team rushed me out the door yesterday morning to be a part of a task force to create a plan to split up the company. Apparently, this is a top priority project, and they need to figure out what the implications to all the IT systems are.”

Brent

Bill schedules a meeting with Steve Masters to discuss the off-site.

Bill believes Brent is someone who has the respect of developers, has enough deep experience with IT infrastructure, and can describe what the developers need to build. Steve agrees to “bring Brent home”.

Chapter 33

Sarah demands to have Brent returned in an angry email to the Chairman of the Board.

Meanwhile, the Unicorn promotion report takes much longer to execute. One of the developers recommends using cloud compute instances. Maggie looks into cloud providers and Security works to identify risks with sending customer data to the cloud. During the Demo, the team reports they can deploy easily to the cloud with the automation in place.

Maggie demos the promotions offering system and proposes to do an e-mail campaign to one percent of the customers (as a trial before Thanksgiving).

The marketing campaign is a success, with over 20 percent of the respondents going to the website and six percent making a purchase. The conversion rates are 5x higher than any prior campaign.

Steve Masters publicly congratulates the entire team on the marketing success.

John reports the security fixes for Unicorn are much easier than Phoenix.

“After being forced to automate our security testing, and integrating it into the same process that William uses for his automated QA testing, we’re testing every time a developer commits code. In many ways, we now have better visibility and code coverage than any of the other applications in the company!”

John

Developers only have read-only access in production and security tests are integrated into the build procedure. The automated controls help resolve the SOX-404 audit findings.

Only a single issue — promotions for out of stock items — temporarily halted deployment, but was fixed within a day.

Chapter 34

The high traffic to the Parts Unlimited website on Thanksgiving led to a Sev-1 emergency because the e-commerce systems were going down.

The team puts more servers into rotation and turned off computationally-intensive features.

Taking down the real-time recommendations can be disabled with a configuration setting in minutes.

Bill calls in the entire team to the office on Black Friday. Store Managers are having trouble handling the volume of requests from Unicorn promotions. To fix the problem, the team will deploy a web page for store personnel to type in the coupon promotion code to automate the cross-shipment from our warehouses. They will also create a new form on the customer account web page to get items delivered directly to them.

At a Monday meeting with Steve Masters, he reports the in-store and web sales are breaking records.

“I want to congratulate you for all your hard work. It has paid off beyond my wildest expectations. Thanks to Unicorn, both in-store and web sales are breaking records, resulting in record weekly revenue. At the current run rate, Marketing estimates that we’ll hit profitability this quarter. It will be our first profitable quarter since the middle of last year.”

Steve

Chris reports the team can do daily deployments and they can also do A/B testing all the time — so the company is faster responding to the market.

Bill proposes setting up all new teams just like the Unicorn team for fast deployments. Steve commends them all for their work.

“If we’re done congratulating ourselves, I’ve got a business wake-up call for you. Earlier this month, our largest retail competitor started partnering with their manufacturers to allow custom build-to-order kits. Sales of some our top selling items are already down twenty percent since they launched this offering.”

Sarah

The team is next charged with creating “build-to-order kits” with their manufacturing partners.

The next day, Wes says the change Sarah requested would be difficult because their original system is an outsourced mainframe application. To make the change, their outsourced will need six months to gather requirements and nine months to develop & test.

In a meeting with Steve, the team propose to break contract with the outsourcing company early at a cost of $1M and regain control of the MRP application & underlying infrastructure.

With MRP in-house, development could build an interface to Unicorn. The manufacturing capability could be moved from “build to inventory” to “build to order”.

One risk is the outsourcer may have changed the codebase. The outsourcer could also make the transition difficult. John also needs to remove access from the staff who are not being retained.

Sarah does not like the proposal, and wants approval from the board & Bob Strauss. Steve reminds Sarah who she works for, which quiets her.

“I think we need to check with Bob Strauss and get full board approval before we undertake a project this big and risky. Given the previous performance of IT, this could jeopardize all of our manufacturing operations, which is more risk than I think we should take on. In short, I personally do not support this proposal.”

Sarah

“Remember that you work for me, not Bob. If you can’t work within that arrangement, I will need your immediate resignation.”

Steve

Chapter 35

There are few Sev-1 incidents and Bill spends most of his time coaching his managers through two-week improvement cycles according to the Improvement Kata.

The team is closing their monitoring gaps, refactored or replaced the top ten fragile artifacts for stability, and the flow of planned work is fast.

Bill also deployed Project Narwhal (Simian Army Chaos Monkey) that routinely creates large-scale faults, thus randomly killing processes or entire servers.

Development and IT Operations worked together to make their code & infrastructure more resilient to failures.

John similarly started a project called “Evil Chaos Monkey”  that would exploit security holes, fuzz applications with malformed packets, try to install backdoors, gain access to confidential data, and other attacks.

Steve Masters hosts a party at his home and asks Bill to arrive an hour early.

The company will have a record-breaking quarter. The average order size hit a record after Project Unicorn delivered.

Sarah has decided to look for other options elsewhere and is on a leave of absence.

The last few months, Bill has been interviewing candidates for CIO. He feels most would revert the changes the company has made the last few months. Bill puts Chris forward for the position.

Steve explains that Bill was the unanimous choice to take the position, but he won’t be getting the job.

Steve wants Bill to do rotations in sales & marketing, manage a plant, get international experience, manage the relationships with critical suppliers, and manage the supply chain. Erik will be his mentor. If successful, Bill will be moved into the COO role.

“In ten years, I’m certain every COO worth their salt will have come from IT. Any COO who doesn’t intimately understand the IT systems that actually run the business is just an empty suit, relying on someone else to do their job.”

Steve

IT should either be embedded into business operations or into the business.

The rest of the IT Management and executive staff arrive at Steve’s home for the party. They gift Bill with a bronzed “craptop”.

Erik will not become a board member, but instead will be a large investor in the company. He wants to create a hedge fund that invests in companies with great IT organizations.

Erik charges Bill to write “The DevOps Cookbook” to show how IT can regain the trust of the business.

“I want you to write a book, describing the Three Ways and how other people can replicate the transformation you’ve made here at Parts Unlimited. Call it The DevOps Cookbook and show how IT can regain the trust of the business and end decades of intertribal warfare. Can you do that for me?”

Erik

The group assembled reflects much more than Dev or Ops or Security. It’s Product Management, Development, IT Operations, and Information Security all working together and supporting one another.

Series Navigation

<< Book Club: The Phoenix Project (Chapters 26-29)