Book Club: The Phoenix Project (Chapters 13-16)

The following is a chapter summary for “The Phoenix Project” by Gene Kim for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Chapters 8 – 12 HERE

Background on the Phoenix Project

“Bill, an IT manager at Parts Unlimited, has been tasked with taking on a project critical to the future of the business, code named Phoenix Project. But the project is massively over budget and behind schedule. The CEO demands Bill must fix the mess in ninety days or else Bill’s entire department will be outsourced.

With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with a manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.

In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again.”

The Phoenix Project

Chapter 13

The Phoenix crisis is still an issue on Monday, and the problems are front page news on technology sites.

Bill is at a Phoenix status meeting, and Steve says that they are massively screwing their customers and shareholders. He says that Sarah is not off the hook until all of the store managers say that they can transact normally.

Steve also wants to meet with Sarah, Chris, Bill, Kirsten, and Ann once the stores are off life support.

Once Steve leaves and slams the door, Sarah says she wants the usability issues fixed, but Bill and others tell her how impossible that is at the moment.

“We are keeping Phoenix alive by sheer heroics. Wes wasn’t joking when he said that we’re proactively rebooting all the front-end servers every hour. We can’t introduce any more instabilities. I propose code rollouts only twice a day and restricting all code changes to those affecting performance.”

Bill

The team produces a plan to tie all code commits to a defect number or they will be rejected.

Bill visits Ann and her team across the hall. They have tables covered in faxes that represent orders that need deduplicated or reversed.

On the wall Ann’s team shows that 5,000 customers have had duplicate payments or missing orders, and they estimate 25,000 more transactions that still need investigated.

John also stops by to check out the activity. When he looks at an order, he tells Bill that they have a major problem.

John tells Bill that they are storing the CVV2 codes, which is against the law. John wants Bill to destroy all that information, but Bill says that they first need to take care of the transactions.

John remembers that the auditors are actually on site that day. Bill instructs him to not allow the auditors close to Ann’s team and the CVV2 information.

Later, John tells Bill that he may have some extra engineers to spare. Bill is thrilled by this since his team is literally at full capacity and is pulling all nighters.

“I then wonder if the fatigue is getting to me. Something is really screwy in the world when I’m finding reasons to thank Development and Security in the same day.”

Bill

Chapter 14

By late Monday, they had finally stabilized the Phoenix situation. The stores had working registers (although the fix is only temporary) and the company is no longer keeping sensitive cardholder data.

The leadership team is waiting outside Steve’s office, and Sarah comes out nearly in tears. Bill and Chris then take their turn to talk to Steve.

Steve says the company has nothing to show for the $20 million they’ve spent on Phoenix. He also says they may have lost loyal customers, and marketing is giving away $100 vouchers.

Bill gets frustrated that Steve didn’t follow his initial advice to delay Phoenix: “No offense, sir, but this is supposed to be news to me? I called you, explaining what would happen, asking you to delay the launch. You not only blew me off, you told me to try to convince Sarah. Where’s your responsibility in all of this? Or have you outsourced all your thinking to her?”

Steve responds by telling Bill that he needs some actual solutions from him. He also says that he needs the business to be able to tell him that it is no longer being held hostage by IT.

Steve goes on to say that the board is considering splitting up the company.

“Second, I’m done playing Russian roulette with IT. Phoenix just shows me that IT is a competency that we may not be able to develop here. Maybe it’s not in our DNA. I’ve given Dick the green light to investigate outsourcing all of IT and asked him to select a vendor in ninety days.”

Steve

Bill and Steve are shell shocked and decide to meet for lunch. Bill mentions that Paige tells him that he shouldn’t trust Chris.

Chris says that maybe his group being outsourced wouldn’t be the worst thing in the world and wonders if it might be time for a change. He says he used to love his work but lately it is so hard to keep up with change.

“It’s harder than ever to convince the business to do the right thing. They’re like kids in a candy store. The read in an airline magazine that they can manage their whole supply chain in the cloud for $499 per year, and suddenly that’s the main company initiative. When we tell them it’s not actually that easy, and show them what it takes to do it right, they disappear. Where did they go? They’re talking to their Cousin Vinnie or some outsourcing sales guy who promises they can do it in a tenth of the time and cost.”

Chris

Chris says that it’s getting harder and harder to hit dates. He was in a meeting where they were planning out work 3 years in the future, but he says that they can’t even effectively plan for one year.

Chris apologizes for his part in the Phoenix fiasco. He says that when he told Sarah a date for when code could be complete, he didn’t know that she would use it as a go live date.

Bill and Chris agree that they are worried that Sarah might try to stick the whole situation on them. They say she is like “Teflon” because nothing sticks to her.

Bill and Chris agree that they will meet once a week for the next few months.

Once back at the office, Bill gets an email from Chris. Chris tells him that they are throwing a celebration party since the Phoenix deployment is “finished” and invites Bill and his team.

Bill forwards the email to Wes and Patty, but Wes says his team still has a lot of work to do.

Chapter 15

The chapter opens on Wednesday with Bill taking Paige out to breakfast. She says she has never seen him this stressed. Bill tells her that he has no idea when life will be normal again.

Paige says she doesn’t know why Bill decided to accept the job. He thinks to himself that the organization is better off because of his contributions and is happy he’s one of the people that can try to fend off the outsourcing.

Bill starts thinking about how the pay raise will help his family pay down their debt. Paige catches him wandering off in thought and says she wishes they picked someone else for the job.

Bill drops off Paige at home and sees that he has an email that Wes has forwarded him. The email is giving praise to the new change board and how it saved two different groups from making changes to the database and app servers at the same time.

Patti knocks on Bill’s door and tells him that she thinks they have a problem. She asks him to follow her to the Change Coordination Room.

“I groan. Every time Patty’s summons me there, it’s because of some new intractable problem. But problems, like dog poop left in the rain, rarely get better just by ignoring them.”

Bill

Bill notices that the change boards look different. He notices there are barely any changes posted that are coming up, and the cards are missing. Patti tells him that there are about 600 cards of changes that need to be rescheduled due to Phoenix.

Bill discovers that the fourth type of work that Erik had mentioned was unplanned work (the other three are: business projects, internal projects, and changes).

“That’s why Erik called it the most destructive type of work. It’s not really work at all, like the others. The others are what you planned on doing, allegedly because you needed to do it.”

Bill

“So much of what I’ve been trying to do during my short tenure as VP of IT Operations is to prevent unplanned work from happening: coordinating changes better so they don’t fail, ensuring the orderly handling of incidents and outages to prevent interrupting key resources, doing whatever it takes so that Brent won’t be escalated to. . .”

Bill

Bill goes outside and calls Erik. Erik asks Bill how he is doing “after Phoenix crashed and burned so spectacularly”, and asks him if he can tell him the four categories of work now.

“At the plant, I gave you one category, which was business projects, like Phoenix,” I say. “Later, I realized that I didn’t mention internal IT projects. A week after that, I realized that changes are another category of work. But it was only after the Phoenix fiasco that I saw the last one, because of how it prevented all other work from getting completed, and that’s the last category, isn’t it? Firefighting. Unplanned work.”

Bill

Erik asks Bill about the change board he’s been working on, and Bill describes it to him.

“You’ve put together tools to help with the visual management of work and pulling work through the system. This is a critical part of the First Way, which is creating fast flow of work through Development and IT Operations. Index cards on a kanban board is one of the best mechanisms to do this, because everyone can see WIP. Now you must continually eradicate your largest sources of unplanned work, per the Second Way.”

Erik

Bill explains all the chaotic events that he has been dealing with lately. Eric responds and mentions that Brent is Bill’s constraint, and Bill is surprised.

“Well if we’re going to talk about your next steps, you definitely need to know about constraints because you need to increase flow. Right now, nothing is more important.”

Erik

Erik tells Bill that he hopes that he read The Goal by Eli Goldratt.

“Goldratt taught us that in most plants, there are a very small number of resources, whether it’s men, machines, or materials, that dictates the output of the entire system. We call this the constraint—or bottleneck. Either term works. Whatever you call it, until you create a trusted system to manage the flow of work to the constraint, the constraint is constantly wasted, which means that the constraint is likely being drastically underutilized.”

Erik

Erik describes the first 3 steps (of 5) in The Goal:

1. Identify the constraint
2. Exploit the constraint
3. Subordinate the constraint

Erik tells Bill his homework is to figure out how to set the tempo of work according to Brent. He also tells Bill he is still missing a piece of the First Way in that he can’t distinguish what is important to the business and what isn’t.

“[Chris] is spending all his cycles on features, instead of stability, security, scalability, manageability, operability, continuity, and all those other beautiful ’itties. Remember, outcomes are what matter—not the process, not controls, or, for that matter, what work you complete.” – Erik

Chapter 16

Bill is at his desk when Ellen runs in with an email printout from Dick. It says something has gone wrong with the company invoicing systems. It was discovered that no customers were invoiced for 3 days.

Leadership gathers in the NOC room. Bill instructs everyone not to touch anything without approval from him.

The team investigates possible causes for the issue, and Patty’s team found over 20 different potential failures. Eventually they narrow it down to 8. They agree to reconvene at 10 pm.

As Bill is reading a book to his son, he checks the emails on his phone. He’s amazed at the difference in his team’s process: “During the last Sev 1 incident that hit our credit card processing systems, the conference call was full of finger-pointing, denials, and, most importantly, wasted time when our customers couldn’t give us money. Afterward, we did the first of a series of ongoing blameless postmortems to figure out what really happened and come up with ideas on how to prevent it from happening again. Better yet, Patty led a series of mock incident calls with all hands on deck, to rehearse the new procedures.”

At 9:15, Bill receives a call from Steve about the incident. Steve tells Bill that he just talked to Dick, and Dick said that Bill is dragging his feet. Steve is clearly angry.

Bill tries explaining his points again, but Steve cuts him off and asks if he’s in the office.
Steve: “We’ll probably miss almost every target that we’ve promised the board: revenue, cash, receivables—everything. In fact, every measure we’ve promised the board is going the wrong way! This screwup may confirm the board’s suspicion that we’ve completely lost control of managing this company!”

Steve tells Bill that he wants to see a sense of urgency, and that he should be getting people out of bed.

“Steve, if I thought it would help, I’d have everyone pull all-nighters in the data center tonight. For Phoenix, some people didn’t go home for nearly a week. Trust me, I know the house is on fire, but right now, more than anything, we need situational awareness. Before we send the teams crashing through the front door with fire hoses, we have to have someone at least quickly walk the perimeter of the yard — otherwise, we’ll end up burning down the houses next door!”

Bill

Steve replies to Bill that Brent disagrees with Bill’s approach. Bill responds that he hopes Brent is at home. He doesn’t want him working until they know exactly what’s wrong.

Steve tells Bill that they’re going to start doing things his way. He screams at Bill to call in Brent along with everyone else.

“You think I’m being overly cautious, and that I’m hesitating to do what needs to be done. But you are wrong. Dead wrong.”

Bill

Steve still is not convinced. Bill responds by telling Steve to do the work himself, and to expect Bill’s resignation in the morning.

Series Navigation

<< Book Club: The Phoenix Project (Chapters 8-12)Book Club: The Phoenix Project (Chapters 17-20) >>