Book Club: The Phoenix Project (Chapters 26-29)

The following is a chapter summary for “The Phoenix Project” by Gene Kim for an online book club.

The book club is a weekly lunchtime meeting of technology professionals. As a group, the book club selects, reads, and discuss books related to our profession. Participants are uplifted via group discussion of foundational principles & novel innovations. Attendees do not need to read the book to participate.

Chapters 21-25 HERE

Background on the Phoenix Project

“Bill, an IT manager at Parts Unlimited, has been tasked with taking on a project critical to the future of the business, code named Phoenix Project. But the project is massively over budget and behind schedule. The CEO demands Bill must fix the mess in ninety days or else Bill’s entire department will be outsourced.

With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with a manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.

In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again.”

The Phoenix Project

Chapter 26

Bill’s team will conduct business process owner interviews for “understanding customer needs and wants,” “product portfolio,” “time to market,” and “sales pipeline”.

John researches the business SOX-404 control environment.

Meeting with Ron Johnson, VP of Sales. He is the owner of the sales pipeline and sales forecast accuracy.

The Sales Forecast Accuracy starts with a revenue target. Ron’s team always misses the target because it’s not obtainable.

Parts Unlimited does not know what its customers want. The company has too much product that will never sell and never enough of product that does sell.

“It’s a crappy way to run a company. It demoralizes my team, and my top performers are quitting in droves. Of course, we’ll replace them, but it takes at least a year for replacements to perform at full quota capacity. Even in this lousy economy, it takes too long to find qualified sales people.”

Ron Johnson

Sales Forecast Accuracy is jeopardized by poor understanding of customer needs and wants.

Sales Pipeline — challenging for salespeople to get information from the Customer Relationship Management (CRM) system.

Bill wants money for the monitoring to enforce control policies to ensure incidents like the phone outage won’t happen again.

Establishing what applications / infrastructure are fragile to Sales, they can prioritize preventive work to ensure revenue is not hurt.

Maggie owns the merchandising and pricing roadmaps for Parts Unlimited. She’s the business sponsor for half the IT projects.

“Ultimately, the way I measure our understanding of customer needs and wants is whether customers would recommend us to their friends. Any way you cut it, our metrics aren’t very good.”

Maggie

Data in the order inventory and management system are almost always wrong.

Maggie wants accurate and timely order information from the stores and online channels. That data can be used for marketing campaigns that continually do A/B testing of offers, find the ones that our customers jump at.

The information can be used to drive the production schedule to manage the supply and demand curves.

“In these competitive times, the name of the game is quick time to market and to fail fast. We just can’t have multiyear product development timelines, waiting until the end to figure out whether we have a winner or loser on our hands. We need short and quick cycle times to continually integrate feedback from the marketplace.”

Maggie

“The longer the product development cycle, the longer the company capital is locked up and not giving us a return. Dick expects that on average, our R&D investments return more than ten percent. That’s the internal hurdle rate. If we don’t beat the hurdle rate, the company capital would have been better spent being invested in the stock market or gambled on racehorses.”

Maggie

Maggie dislikes the three-year lead time on projects. They should be six to twelve months. Phoenix cost $20M/3Yrs. There is intense competition for IT in supporting projects. Given the WIP and capital locked into Phoenix, the project should have never been approved.

Chapter 27

Phone and MRP systems need predictive measures that include compliance with the change management process, supervision and review of production changes, completion of scheduled maintenance, and elimination of all known single points of failure.

John introduces the idea of CIA — confidentiality, integrity, and availability.

For Marketing Needs and Wants: Support weekly and eventually daily reporting of orders and percentage of valid SKUs created by Marketing.

“Seems pretty obvious to me. We need to come with the controls to mitigate the risks in your third column. We then show this table to Ron and Maggie, and make sure they believe that our countermeasures help them achieve their objectives. If they buy it, we work with them to integrate IT into their performance measures”

John

The company used to have the CIO attend the quarterly business reviews but stopped inviting him after negative feedback.

The company must consider IT Risks as Business Risks, otherwise the company will miss their objectives.

Bill proposes to integrate risks into leading indicators of performance. The goal is to improve business performance and get earlier indicators of whether the company will achieve them.

The company is moving too slowly with too much WIP and too many features in flight. The releases must be smaller and shorter and deliver cash back faster. Bill proposes meeting for three weeks with each business process owner to identify business risks posed by IT to integrate them into leading indicators of performance. He also proposes meeting with Dick & Chris about Phoenix to improve throughput.

From the audit-compliance side, John excitedly reports back on his findings.

Faye, a Financial Analysts who works in Finance, created SOX-404 control documents. They show the end-to-end information flow for the main business processes in each financially significant account.

“The control being relied upon to detect material errors is the manual reconciliations step, not in the upstream IT systems.” – Faye’s document

John wants to rebuild the compliance program from scratch. He proposes to: 

1. drastically reduce the scope of the SOX-404 compliance program 
2. root cause analysis of production vulnerabilities
3. flag all systems in scope for compliance audits to avoid changes that risk audit and create on-going documentation for auditors
4. reduce the size of PCI compliance program by eliminating anything that stores or processes cardholder data
5. pay down technical debt in Phoenix

“We quickly agree to pair up people in Wes’ and Chris’ group with John’s team, so that we can increase the bench of security expertise. By doing this, we will start integrating security into all of our daily work, no longer securing things after they’re deployed.”

Bill

Chapter 28

The number of Sev-1 outages this month is down by more than two-thirds. Incident recovery time is reduced by half.

Improving the production monitoring of the infrastructure and applications, IT knows about incidents before the business does.

The project backlog has been reduced by eliminating unneeded security projects from audit preparation and replacing them with preventive security projects.

Bill comes to the conclusion that IT Operations work is similar to planned work. The team is mastering the First Way: curbing the handoffs of defects to downstream work centers, managing the flow of work, setting the tempo by our constraints, and understanding what is important versus what is not.

Sarah’s group is trying to make unauthorized purchases for online or cloud services. Her team has four instances of using outside vendors and online services. 

Sarah’s vendors will cause Parts Unlimited to break their customer privacy regulations and potentially state privacy laws. One vendor uses a database technology that is not secured.

“The first problem is that both projects violate the data privacy policy that we’ve given our customers,” John says. “We repeatedly promise that we will not share data with partners. Whether we change that policy or not is, of course, a business decision. But make no mistake, if we go ahead with the customer data mining initiative, we’re out of compliance with our own privacy policy. We may even be breaking several state privacy regulations that expose us to some liability.”

John

Sarah has been able to get away with murder because she has the strategy that Steve needs, whereas Steve is execution-focused.

During the next Phoenix deployment, one of the critical database migration steps failed. Brent made a change to a production database that no one knew about. This change was one of Sarah’s side projects.

The Dev and QA environments don’t match the production environment. The team still manages to finish the deployment before the stores open. Patty sends out a list of known errors to look out for, an internal web page for the latest Phoenix status, and instructions on how to report new problems. The service desk is on standby and both dev & ops teams are on-call.

Chapter 29

Steve Masters is happy with the state of IT even though the latest Phoenix Deployment went in late. 

“I am very proud to be a part of this team that is obviously working together better than ever, trusting one another, and getting incredible results.”

Steve

Sarah is caught in the meeting having started unauthorized IT projects.

“Supporting those projects also requires an incredible amount of work. We’d need to give your vendors access to our production databases, explain how we’ve set them up, do a bunch of firewall changes, and probably over a hundred other steps. It’s not just as easy as signing an invoice.”

Bill

Sarah announces that Board Member Bob Strauss believes the company should be split up and leaves the room in a huff.

Erik challenges the group to master the Second Way: creating constant feedback loops from IT Operations back into Development, designing quality into the product at the earliest stages.

“In any system of work, the theoretical ideal is single-piece flow, which maximizes throughput and minimizes variance. You get there by continually reducing batch sizes. You’re doing the exact opposite by lengthening the Phoenix release intervals and increasing the number of features in each release. You’ve even lost the ability to control variance from one release to the next.”

Erik

Bill proposes to pause deployments until they can figure out how to keep environments synchronized.

Erik proposes that work only flows forward. With rework and long release cycles, the team will never hit the internal rate of return.

“The flow of work should ideally go in one direction only: forward. When I see work going backward, I think ‘waste.’ It might be because of defects, lack of specification, or rework. . . Regardless, it’s something we should fix.”

Steve

Bill proposes a SWAT Team to deal with the Phoenix capacity issues, with that team tasked with delivering features that hit revenue goals.

The features are focused on customer recommendations and promotions that match the customer profile from consumer data.

Series Navigation