Book Club: The Phoenix Project (Chapters 21-25)

The following is a chapter summary for “The Phoenix Project” by Gene Kim for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Chapters 17-20 HERE

Background on the Phoenix Project

“Bill, an IT manager at Parts Unlimited, has been tasked with taking on a project critical to the future of the business, code named Phoenix Project. But the project is massively over budget and behind schedule. The CEO demands Bill must fix the mess in ninety days or else Bill’s entire department will be outsourced.

With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with a manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.

In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again.”

The Phoenix Project

Chapter 21

Bill arrives at an audit meeting in building 2. He arrives to a packed conference room with attendees such as Dick, John, Wes, Erik, Ann, Nancy, and the auditors. Bill is surprised at how awful and stressed John looks.

The meeting lasts for 5 hours and ends with everyone surprised with the auditor’s conclusion that the company probably isn’t in trouble.

Bill is surprised to learn after that meeting that Erik and the lead auditor are old friends.

Bill describes the meeting and says that Dick kept marching out business SME’s to demonstrate that they have their own controls outside of IT where fraud would be caught.

John then asks if Bill has a minute to talk. He is still visibly flustered.

“John looks awful. If his shirt were just a little more wrinkled, and maybe had a stain or two in front, he could almost pass as a homeless person.”

Bill

John starts to ramble on to Bill about the systemic IT issues and information security. Erik is still talking to the auditor in the room and guides him out into the hallway upon hearing John starting his rant. John says that no one cares about IT security and the entire dev organization hides their activities from him.

“You all look down on me. You know, I used to manage servers, just like you do. But I found my calling doing information security. I wanted to help catch bad guys. I wanted to help organizations protect themselves from people who were out to get them. It came out of a sense of duty and a desire to make the world a better place.”

John

Erik comes back into the room angrily and grabs a chair.

“You know what your problem is, Jimmy? You are like the political commissar who walks onto the plant floor…sadistically poking your nose in everybody’s business and intimidating them into doing your bidding, just to increase your own puny sense of self-worth. Half the time, you break more than you fix. Worse, you screw up the work schedules of everyone who’s actually doing important work.”

Erik (to John)

Erik continues to go off on John and tells him that he doesn’t have anything else to say to him until John understands what just happened in that room.

“This should be your guiding principle: You win when you protect the organization without putting meaningless work into the IT system. And you win even more when you can take meaningless work out of the IT system.”

Erik

Erik tells John to go to the MRP-8 plant and talk to the safety officer.

Erik leaves John and Bill alone. John says goodbye and pushes his binder off the table. He says he may not be back tomorrow. Bill sees a haiku that John had written:

Here I sit, hands tied

Room angry, I could save them

If only they knew

Chapter 22

The Monday after the audit meeting, John disappeared.

Bill finds Wes and takes him to Patty’s office to talk about the monitoring project. Bill tells them both about how Erik validated that they can release the monitoring project and how important it is so that they can elevate Brent.

Patty seems to entertain the idea that IT is like manufacturing but Wes is still skeptical.

“Let’s use the example of configuring a server. It involves procurement, installing the OS and applications on it according to some specification and then getting it racked and stacked. Then we validate that it’s been built correctly. Each of these steps are typically done by different people. Maybe each step is like a work center, each with its own machines, methods, men, and measures.”

Patty

Patty concludes that she’s not sure what the machine would be in her scenario, and that they are better off trying out the process first. Otherwise, they are just stumbling around in the dark.

Wes is still skeptical, but Bill explains how challenging some of the floor work in the factory was. He says how those workers had to call upon their experience to solve problems and how they earned his respect.

Bill says that they should start the monitoring project as soon as they can.

The next Monday, Bill goes to the change room with Patty. She has put up a new Kanban board. Over the weekend she went to the MRP-8 factory and learned to split work into Ready, Doing, and Done in order to reduce WIP.

Patty plans on putting Kanban boards around key resources to manage their work. She thinks this will help predict lead time and get faster throughput.

“Imagine what this will do to user satisfaction if we could tell them when they make the request how long the queue is, tell them to the day when they’ll get it, and actually hit the date, because we’re not letting our workers multitask or get interrupted!”

Patty

Patty has also implemented an improvement kata and two-week improvement cycles.

Patty wants to implement a Kandan board around Brent in order to further isolate him from crises.

Later, Bill is sitting with Patty and Wes to figure out how to get project work started again. Bill states that they have two queues: business and internal projects.

They decide to only release the five most important business projects. The hard part is prioritizing the 73 internal projects.

Bill remembers what Erik has told him. He says that unless a project increases Brent’s capacity by reducing his workload or allowing someone else to take it over, then it isn’t important.

He asks for 3 lists: one that requires Brent, one that increases Brent’s throughput, and one that is everything else.

“We’re doing what Manufacturing Production Control Departments do. They’re the people that schedule and oversee all of production to ensure they can meet customer demand. When they accept an order, they confirm there’s enough capacity and necessary inputs at each required work center, expediting work when necessary. They work with the sales manager and plant manager to build a production schedule so they can deliver on all their commitments.”

Patty

Two days later, Bill finally gets a new laptop. This is two days earlier than planned.

Chapter 23

The following Tuesday, Bill gets a call from Kirsten. Brent is almost a week late on delivering a Phoenix task and the schedule is in jeopardy again. There are also several other late tasks.

Bill gets to work and joins Patty and Wes in a conference room. Patty explains that the late task is a test environment that was supposed to be delivered to QA. It turns out that this one “task” was more like a small project and involved multiple layers and teams.

Bill goes to the whiteboard and draws a graph:

Bill explains how wait times depend upon resource utilization: “The wait time is the ‘percentage of time busy’ divided by the ‘percentage of time idle.’ In other words, if a resource is fifty percent busy, then it’s fifty percent idle. The wait time is fifty percent divided by fifty percent, so one unit of time. Let’s call it one hour. So, on average, our task would wait in the queue for one hour before it gets worked.”

Bill recalls the Phoenix deployment when Wes was complaining about an excess amount of tickets that would take weeks to resolve. He concludes that the handoff between Dev and IT Ops is very complex.

Patty states that the group shows that everyone needs idle time. Otherwise, WIP gets stuck in the system.

The group then decides that they can create a Kanban lane for each large, recurring “task”.

“You know, deployments are like final assembly in a manufacturing plant. Every flow of work goes through it, and you can’t ship the product without it. Suddenly, I know exactly what the Kanban should look like.”

Patty

They decide that Patty will work with Wes’s team to assemble the 20 most frequently recurring tasks.

Chapter 24

The chapter starts with Bill and his family visiting a pumpkin patch on a Saturday. They spend the day together, and then Bill is watching a movie with his wife on the couch.

He gets a call on his phone from John. He answers the phone after a few ignored calls, and John asks Bill to meet him at a bar. Bill eventually agrees.

When Bill sees John, he looks awful. John is also very drunk.

John says that he has just been at home watching TV. He wants to ask Bill one last question before he leaves.

“Just tell me straight. Is it really true that I haven’t done anything of value for you? In all the three years that we’ve worked together, I’ve never, ever been helpful?”

John

Bill answers, “Look, John. You’re a good guy, and I know your heart is in the right place, but up until you helped hide us from the PCI auditors during the Phoenix meltdown, I would have said no. I know that’s not what you want to hear, but. . . I wanted to make sure that I wasn’t feeding you a line of bullshit.”

John downs a glass of scotch after hearing Bill’s response and asks for another, but Bill tells the waitress not to get it and order a cab.

Bill puts John in the cab and sends him home.

He tries calling John the next day, but John does not answer. There are still rumors circulating at the office regarding what happened to him.

Later Monday night, Bill receives a text from John: Thanks for the lift home the other day. Been thinking. I told Dick that I’ll be joining our 8am mtg tomorrow. Should be interesting.

Bill doesn’t know what meeting John is talking about.

When Bill asks John what meeting he’s talking about, John responds that he’s been arrogant and doesn’t know Dick that well. He says that him and Bill need to change that together. Bill calls John to see what’s going on.

“I kept thinking about our last conversation at the bar. I realized that if I haven’t done anything useful for you, who I should have the most in common with, then it stands to reason that I haven’t been useful to almost everyone else, who I have nothing in common with.”

John

Bill reluctantly agrees to join John in the meeting.

Chapter 25

The next day, Bill heads toward Dick’s office for the meeting with Dick and John. He sees John outside the office, and John has totally cleaned up his appearance from when Bill last saw him. Bill: “With the shaved head, his calm friendly smile and perfect posture, he looks like some sort of enlightened monk.”

Bill is shocked when John asks Dick, “. . . what exactly you do here at Parts Unlimited? What is your exact role?”

Dick plays along and answers the question seriously. He says that when he was hired, he was a traditional CFO, but now he also takes care of planning and operations for Steve. John calls him the de-facto COO, but Dick acknowledges that is now part of his job.

“With a very small smile, he adds, “Want to hear something funny? People say that I’m more approachable than Steve! Steve’s incredibly charismatic, and let’s face it, I’m an asshole. But when people have concerns, they don’t want to have their minds changed. They want someone to listen to them and help make sure Steve gets the message.”

Dick

When asked what a good day for himself looks like, Dick says that it’s when they are beating the competition and writing big commission checks to their salesmen.

“Steve would be excited to announce to Wall Street and the analysts how well the company is performing—all made possible because we had a winning strategy, and also because we had the right plan and the ability to operate and execute.”

Dick

Dick says that they haven’t had a day like that in over four years. He says that a bad day looks like the Phoenix project launch.

John asks Dick what his goals for the year are. Dick gives him a list:

• Health of Company
• Revenue
• Market Share
• Average Order Size
• Profitability
• Return of Assets
• Health of Finance
• Order to Cash Cycle
• Accounts Receivable
• Accurate Financial Reporting
• Borrowing Costs

Dick continues to the company goals, which he says are more important than the goals for just his department:

1. Are we competitive?
2. Understanding customer needs and wants: Do we know what to build?
3. Product portfolio: Do we have the right products?
4. R&D effectiveness: Can we build it effectively
5. Time to market: Can we ship it soon enough to matter?
6. Sales pipeline: Can we convert products to interested prospects?
7. Are we effective?
8. Customer on-time delivery: Are customers getting what we promised them
9. Customer retention: Are we gaining or losing customers?
10. Sales forecast accuracy: Can we factor this into our sales planning process?

Dick says all of those measurements are currently at risk. He says they are $20 million into Phoenix and still are not competitive, and the best favor they can do him is to stay focused and get it working.

After the meeting, Bill says that Dick doesn’t realize how much his measurements depend on IT.

Bill calls Erik to get some advice. He wants to convince Dick that IT is capable of screwing up less often and helping the business win.

“Your mission is twofold: You must find where you’ve under-scoped IT—where certain portions of the processes and technology you manage actively jeopardizes the achievement of business goals—as codified by Dick’s measurements. And secondly, John must find where he’s over-scoped IT, such as all those SOx-404 IT controls that weren’t necessary to detect material errors in the financial statements.”

Erik (to Bill)

He also says that John also needs to learn exactly how business was able to dodge the audit bullet, and to feel free to invite him to Bill’s next meeting with Bill.

Series Navigation