Book Club: The Phoenix Project (Chapters 17-20)

The following is a chapter summary for “The Phoenix Project” by Gene Kim for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Chapters 13-16 HERE

Background on the Phoenix Project

“Bill, an IT manager at Parts Unlimited, has been tasked with taking on a project critical to the future of the business, code named Phoenix Project. But the project is massively over budget and behind schedule. The CEO demands Bill must fix the mess in ninety days or else Bill’s entire department will be outsourced.

With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with a manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.

In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again.”

The Phoenix Project

Chapter 17

Bill takes his son to see the trains after quitting but is interrupted by multiple calls from Wes & Patty.

The inventory management systems are down. No one can get inventory levels in the plants or warehouses, and they don’t know which raw materials need to be replenished.

“Well, we’ve pretty much screwed the pooch since you’ve left,” Wes says, sounding genuinely abashed, confirming my worst fears. “Steve insisted that we bring in all the engineers, including Brent. He said he wanted a ‘sense of urgency’ and ‘hands on keyboards, not people sitting on the bench.’ Obviously, we didn’t do a good enough job coordinating everyone’s efforts, and…”

Wes

Steve Masters attempts to call Bill after calling his wife Paige. Eventually, Bill returns his call and listens to Steve’s apology.

Steve had promised to get “his hands dirty” with IT but hasn’t lived up to the promise. His delegation of IT to Sarah was a total screwup.

“I’m convinced that IT is a competency that we need to develop here. All I’m asking is that you spend ninety days with me and give it a try.”

Steve

Steve Masters convinces Bill to rejoin Parts Unlimited.

Chapter 18

Bill attends Steve’s IT Leadership Off-Site, which is actually located on the Parts Unlimited campus.

Wes, Patty, Chris, Erik, and Steve are all in attendance.

“Erik described the relationship between a CEO and a CIO as a dysfunctional marriage. That both sides feel powerless and held hostage by the other.”

Steve

“There are two things I’ve learned in the last month. One is that IT matters. IT is not just a department that I can delegate away. IT is smack in the middle of every major company effort we have and is critical to almost every aspect of daily operations.”

Steve

“The second thing I’ve learned is that my actions have made almost all our IT problems worse. I turned down Chris and Bill’s requests for more budget, Bill’s request for more time to do Phoenix right, and micromanaged things when I wasn’t getting the results I wanted.”

Steve

Steve apologizes to Bill, taking full responsibility for the failures of Phoenix and the audit.

Steve identifies trust as the primary issue.

“A great team doesn’t mean that they had the smartest people. What made those teams great is that everyone trusted one another. It can be a powerful thing when that magic dynamic exists.”

Steve

Five Dysfunctions of a Team: In order to have mutual trust, you need to be vulnerable.

Steve asks each person to share something about themselves.

Steve was the first person in his family to make it to college. He worked in a copper mine to pay for college. He eventually went on to work for a pipe manufacturing plant. Steve joined the ROTC to help pay for school and then the US Army.

Steve is an excellent officer with high ratings but none of his subordinates enjoy working with him. Steve commits to changing his ways.

“Over the next three decades, I became a constant student of building great teams that really trust one another. I did this first as a materials manager, then later as a plant manager, as head of Marketing, and later, as head of Sales Operations. Then twelve years ago, Bob Strauss, our CEO at the time, hired me to become the new COO.”

Steve

Steve asks for commitment from everyone to develop IT as a competency by starting to trust one another. Everyone in attendance nods in agreement, except for Bill. . .

Chapter 19

Bill eventually nods in agreement as well.

Patty apologizes for reacting so coldly to Bill. She credits Bill for changing the IT Department.

“The goal of this exercise is to get to know one another as people. You’ve learned a bit about me and my vulnerabilities. But that’s not enough. We need to know more about one another. And that creates the basis for trust.”

Steve

Chris volunteers to start. He was born in Beirut and speaks four languages. He describes the story of his wife’s pregnancy complications and how it taught him to not be selfish.

Wes participates next. He was engaged three times and called off each before getting married. Wes races cars and has struggled with his weight.

Patty started as an Art major but ended up switching majors five times in college. She dropped out of college to become a singer-songwriter, touring the country. She decided to work for Parts Unlimited because she couldn’t make a living as an artist.

Bill grew up in a family with an alcoholic father. He ran away from home and got into trouble. After being arrested, he chose to join the Marines.

Bill cries as he describes the lessons learned from the Marines: “What did I learn? That my main goal is to be a great father, not like the shitty father I had. I want to be the man that my sons deserve.”

“Solving any complex business problem requires teamwork, and teamwork requires trust. Lencioni teaches that showing vulnerability helps create a foundation for that.”

Steve

Steve identifies missing every commitment and schedule as a primary problem in IT. He surmises that the team is not good at making internal commitments.

Chris counters that his team hit their targets, including on Phoenix. However, Phoenix was a disaster. If success was Chris getting all the Phoenix tasks done, then they met their target. If success was putting Phoenix into production fulfilling business goals, then they failed.

Development does not factor in the work Operations needs to complete.

Part of the problem is planning and architecture. Development is also waiting for operations to deploy because there is backlog of work.

“Erik has helped me understand that there are four types of IT Operations work: business projects, IT Operations projects, changes, and unplanned work. But, we’re only talking about the first type of work, and the unplanned work that get’s created when we do it wrong. We’re only talking about half the work we do in IT Operations.”

Bill

Bill realizes while discussing the types of work (the audit project specifically) they have forgotten to invite John. Steve takes a 15-minute break to invite John.

The IT staff is unsure how they make commitment decisions for projects, unlike the manufacturing plant. No capacity or demand analysis is done.

IT takes shortcuts, which means fragile applications in production, and firefighting, which leads to technical debt.

Technical debt compounds over time.

“If an organization doesn’t pay down its technical debt, every calorie in the organization can be spent just paying interest, in the form of unplanned work.”

Erik

“Unplanned work has another side effect. When you spend all your time firefighting, there’s little time or energy left for planning. When all you do is react, there’s not enough time to do the mental work of figuring out whether you can accept new work. So projects are crammed onto the plate, with fewer cycles available to each one, which means more bad multitasking, more escalations from poor code, which mean more shortcuts.”

Erik

Identify where the constraint is and then protect it. Ensure time is never wasted on the constraint.

Bill believes Brent is the constraint for Parts Unlimited.

To fix the problems of IT, Bill proposes to stop doing all other non-Phoenix work to focus on improving their processes for two weeks.

Erik agrees, because the goal should be to increase the throughput of the entire system.

Steve promises to send out an email to the company announcing the work stoppage, to prevent managers from “strong arming” Operations into helping pet projects.

The team will identify the top areas of technical debt, which Development will tackle to decrease the unplanned work being created by problematic applications in production.

Chapter 20

The company has made great progress on Phoenix; more accomplished in 7 days than in the prior month.

The company experiences a Sev-1 incident that took out internal phones and voicemail. The incident was caused by a vendor accidentally making changes to the production phone system. The team will put together a project to monitor critical systems for unauthorized changes.

“How do we currently prioritize our work? When we commit to work on a project, a change, a service request, or anything else, how does anyone decide what to work on at any given time? What happens if there are competing priorities?”

Bill

Priorities are typically based on the most senior person making the request or most recent request.

Erik and Bill take another trip to the manufacturing plant.

Understanding the flow of work is the first key to achieving the First Way.

Bill surmises that Brent is a worker supporting way too many work centers, which is why he’s a constraint.

“Every work center is made up of four things: the machine, the man, the method, and the measures. Suppose for the machine, we select the heat treat oven. The men are the two people required to execute the predefined steps, and we obviously will need measures based on the outcomes of executing the steps in the method.”

Erik

Bill is standardizing Brent’s work so others can execute it. Documenting the steps helps with consistency and quality.

Bill comes to the conclusion that only those projects that don’t require Brent are safe to begin work on again.

The monitoring project is the most important because it elevates the constraint by removing unnecessary work from his plate by bypassing him.

Total Productive Maintenance

• Do whatever it takes to assure machine availability by elevating maintenance
• ‘Improving daily work is even more important than doing daily work.’

“The Third Way is all about ensuring that we’re continually putting tension into the system, so that we’re continually reinforcing habits and improving something. Resilience engineering tells us that we should routinely inject faults into the system, doing them frequently, to make them less painful.”

Erik

Improvement Kata: Mike Rother says it almost doesn’t matter what you improve, as long as you’re improving something. Because if you are not improving, entropy guarantees that you are getting worse, which ensures that there is no path to zero errors, zero work-related accidents, and zero loss.

Kata: repetition creates habits, and habits are what enable mastery

Just as important as throttling the release of work is managing the handoffs.

The wait time for a given resource is the percentage that resource is busy, divided by the percentage that resource is idle.

If a resource is fifty percent utilized, the wait time is 50/50, or 1 unit. If the resource is ninety percent utilized, the wait time is 90/10, or nine times longer.

“A critical part of the Second Way is making wait times visible, so you know when your work spends days sitting in someone’s queue—or worse, when work has to go backward, because it doesn’t have all the parts or requires rework.”

Erik

The Security Projects from John don’t help scalability, availability, survivability, sustainability, security, supportability, or the defensibility of the organization. At present, they are not a good use of time.

Series Navigation