Book Club: The Phoenix Project (Chapters 4-7)

The following is a chapter summary for “The Phoenix Project” by Gene Kim for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Chapters 1-3 HERE

Background on the Phoenix Project

“Bill, an IT manager at Parts Unlimited, has been tasked with taking on a project critical to the future of the business, code named Phoenix Project. But the project is massively over budget and behind schedule. The CEO demands Bill must fix the mess in ninety days or else Bill’s entire department will be outsourced.

With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with a manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.

In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again.”

The Phoenix Project

Chapter 4

Bill is inundated with emails and voicemails just one day on the job. One high priority email comes from Sarah Moulton (SVP of Retail Operations) regarding delays in the Phoenix Project.

Development on the Phoenix Project is behind and they have not considered how to test and deploy the application. This is typical for handoffs between Development and IT Operations at Parts Unlimited.

“The majority of our marketing projects can’t be done without IT. High touch marketing requires high tech. But if there’s so many of us assigned to these Marketing projects, shouldn’t they be coming to us?”

Bill Palmer

Introductions:

• Kirsten Fingle, Project Management Office. She is organized, levelheaded, and a stickler for accountability.
• Sarah Moulton, SVP of Retail Operations.
• Chris Allers, VP of Application Development and acting CIO. Has a reputation as a capable and no-nonsense manager.

The Phoenix Project team has grown by 50 people in the last two years, many through offshore development shops.

Steve Masters attends the Phoenix Project project management meeting. The project has been red for four weeks. Sarah Moulton attacks Bill’s team for the delays.

“See, Bill, in order for us to increase market share, we must ship Phoenix. But for some reason, you and your team keep dragging your feet. Maybe you’re not prioritizing correctly? Or maybe you’re just not used to supporting a project this important?”

Sarah Moulton

Parts Unlimited has spent over $20 million on Phoenix and are two years late.

Chris says Phoenix can be delivered in a few weeks but Wes is not convinced. It would take three weeks just to order the infrastructure necessary and the performance of Phoenix is slow. Additionally, Operations does not have a specification on how the production and test systems will be configured.

“I’ve seen this movie before. The plot is simple: First, you take an urgent date-driven project, where the shipment date cannot be delayed because of external commitments made to Wall Street or customers. Then you add a bunch of developers who use up all the time in the schedule, leaving no time for testing or operations deployment. And because no one is willing to slip the deployment date, everyone after Development has to take outrageous and unacceptable shortcuts to hit the date.”

Bill Palmer

Bill tries to convince Steve to delay the release of Phoenix to no avail. Phoenix impacts thousands of point of sale systems and all of the back-office order entry systems.

After the meeting, Bill and Wes conclude that they’re going to have to get a huge team of their employees together in a room to make the release happen and will also need members of Chris’s team. They also need to free up Brent from fire fighting so that he can help solve problems at the roots.

To make things worse, Bill gets the dreaded blue screen of death on his laptop. His new secretary, Ellen, informs them that a lot of people are experiencing the issue.

Bill attends the CAB (change advisory board) meeting which Patty runs. They are the only two people in attendance. Bill sends out an email to the org stating that all relevant people must attend another mandatory CAB meeting on Friday afternoon.

Bill is given a replacement laptop that is ~10 years old since the help desk team was unable to fix is blue screen of death.

Wes talks to Bill and objects to his mandatory CAB meeting. He says last time the org tried to enforce this it bogged down all his developers in paperwork and they were unable to be productive.

Chapter 5

Bill wakes up the next day to an email from Steve. They need to meet with Nancy Mailer, the Chief Audit Executive. The auditors have uncovered some issues that need to be discussed.

The room is quiet when Bill arrives at the 8 AM meeting. Also in attendance are John, Wes, and Tim, an IT auditor.

The auditing team has found nearly a thousand issues, although only 16 of them are “significant deficiencies”.

Nancy requires a management response letter which includes a remediation plan. Normally the remediation of these issues takes months, but Bill’s team is only given a few weeks before the external auditors arrive.

John tries to grandstand and state his team is on top of things, but that doesn’t seem to be the case. Bill finds out that John’s fix that broke the payroll system may not have even been necessary since it’s out of scope for this audit.

When Bill asks what the most important issue is, he is told: “The first issue is the potential material weakness, which is outlined on page seven. This finding states that an unauthorized or untested change to an application supporting financial reporting could have been put into production. This could potentially result in an undetected material error, due to fraud or otherwise. Management does not have any control that would prevent or detect such a change.”

Nancy Mailer

Bill is also told his team was unable to produce any change meeting minutes, which he already knows but pretends that this is news to him.

After some more discussion and confrontation between Wes and John, Bill agrees to get with his team and come up with a plan, even though everyone is already buried with Phoenix project work.

Wes and Bill stick around after the meeting to talk. Bill is beginning to get the impression that it’s hard to do much of anything without Brent. Wes says they tried to hire some other people at the same level as Brent but they have either left or aren’t as good as Brent.

Bill also discovers that there is no overall backlog of work. They have no visibility into how many business projects and infrastructure projects.

“We also have all the calls going into the service desk, whether it’s requests for something new or asking to fix something. But that list will be incomplete, too, because so many people in the business just go to their favorite IT person. All that work is completely off the books.”

Patty

The team (Bill, Patty, and Wes) set out to get a list of organizational commitments from their key resources, with a one-liner on what they’re working on and how long it will take. Bill will take all of Patty and Wes’s data to Steve on Monday to frame an argument for needing more people.

Chapter 6

Bill realizes during a status meeting that the development team is even more behind than he had feared, and almost all testing is being deferred to the next release.

Patty and Wes have put together data for what all their people are working on, and they share it with Bill. They discover that they have a high number of projects compared to the number of people, and their people to projects ratio is going to be about 1:1.

Most of the Operations resources are committed to Phoenix, and the 2nd largest project is Compliance. They also mention the compliance project would take all of their resources almost an entire year.

“Most of our resources are going to Phoenix. And look at the next line: Compliance is the next largest project. And even if we only worked on compliance, it would consume most of our key resources for an entire year! And that includes Brent, by the way.”

Wes

The 3rd largest project is incident and break-fix work, which is currently taking about 75% of the staff’s time.

Patty states that the one consistent theme in the interviews was that everyone struggles to get their project work done. When they do have time, the business is constantly making requests.

The numbers show that they will need to hire seven people so that everyone can complete their work.

Later that day, everyone attends a meeting for the Change Advisory Board (CAB).

“We need to tighten up our change controls, and as managers and technical leads, we must figure out how we can create a sustainable process that will prevent friendly-fire incidents and get the auditors off our back, while still being able to get work done. We are not leaving this room until we’ve created a plan to get there. Understood?”

Bill

The group starts off by stating that the change management tool is impossible to use. Bill calls a 10-minute break since things are slowly getting away from him. When the meeting reconvenes, Bill states that they must record all the necessary changes that must take place over the next 30 days.

Everyone dives in and starts taking the change management meeting seriously, however the discussions for individual changes go on for a lot longer than anticipated. To keep it simple, they request (1) who is planning the change, (2) the system being changed, and (3) a one-sentence summary.

The team comes up with a definition of change: “a ‘change’ is any activity that is physical, logical, or virtual to applications, databases, operating systems, networks, or hardware that could impact services being delivered.”

Parts Unlimited IT Operations Team

Later, Patty calls Bill and says that they can expect about 400 changes to be submitted that need to happen the next week. Bill tells Patty that all Monday changes can go through without being authorized, but that all changes for later in the week will have to be reviewed.

Chapter 7

Bill gets a call that a potential new board member, Erik Reid, is in town and needs to talk with all the IT executives. Bill decides to meet with Erik even though it’s been a long day.

Bill mistakes Erik for a deliveryman since Erik is wearing wrinkled khakis and an untucked shirt. Erik seems to have trouble remembering names of people he’s met but has assessed the IT situation accurately.

“It looks like you’re in a world of hurt. IT Operations seems to have lodged itself in every major flow of work, including the top company project. It has all the executives hopping mad, and they’re turning the screws on your Development guy to do whatever it takes to get it into production.”

Erik Reid

Erik then takes Bill to one of the company’s manufacturing plants to learn about WIP. WIP is “work in progress”.

“In the 1980s, this plant was the beneficiary of three incredible scientifically-grounded management movements. You’ve probably heard of them: the Theory of Constraints, Lean production or the Toyota Production System, and Total Quality Management. Although each movement started in different places, they all agree on one thing: WIP is the silent killer. Therefore, one of the most critical mechanisms in the management of any plant is job and materials release. Without it, you can’t control WIP.”

Erik Reid

Erik talks to bill about prioritizing work, and why bottlenecks are important to selecting work. Bill says that running IT operations is not like running a factory, but Erik disagrees with him.

The Theory of Constraints:

“Eliyahu M. Goldratt, who created the Theory of Constraints, showed us how any improvements made anywhere besides the bottleneck are an illusion. Astonishing, but true! Any improvement made after the bottleneck is useless, because it will always remain starved, waiting for work from the bottleneck. And any improvements made before the bottleneck merely result in more inventory piling up at the bottleneck.”

“Your job as VP of IT Operations is to ensure the fast, predictable, and uninterrupted flow of planned work that delivers value to the business while minimizing the impact and disruption of unplanned work, so you can provide stable, predictable, and secure IT service.”

Erik Reid

The Three Ways:

“The First Way helps us understand how to create fast flow of work as it moves from Development into IT Operations, because that’s what’s between the business and the customer. The Second Way shows us how to shorten and amplify feedback loops, so we can fix quality at the source and avoid rework. And the Third Way shows us how to create a culture that simultaneously fosters experimentation, learning from failure, and understanding that repetition and practice are the prerequisites to mastery.”

Series Navigation