ALL CIOS HAVE AT least one
turkey in their past-an IT project launched with the best of intentions
that simply could not fly. We convinced a group of CIOs to reveal the
details behind their own worst nightmares, offering insights on
Why these projects failed How the CIOs were able to recover from failure-or not
Ways other CIOs can avoid sponsoring similar turkeys
Survival Tip
Don't
let a doomed project run on—or die a quiet death. Admit it's failed and
announce the failure.
Survival Tip
IS must break its "code of silence" when working on challenging
projects. "If [IS managers] had spoken up sooner, when things started to
go wrong, 14 jobs might have been saved."
Survival Tip
Require change
control documentation so that managers can see what changes were made
during development, when and why.
Survival Tip
Communicate with
the business: Send a monthly letter to senior management with updates as
the project progresses.
Survival
Tip
Look before you leap into a new career "opportunity," and
don't be afraid to conduct due diligence if you're taking on a project
that's already underway.
GartnerGroup Inc
(http://www.gartner.com/)
Standish Group
International Inc
(http://www.standish
group.com/)
The
Martin Agency
(http://www.martin
agency.com/)
Washington State Department of Licensing
(http://www.wa.gov/dol/main.htm)
You may not be able to avoid them, but at least you can recognize them while they're still half-baked. To that end, a group of brave CIOs and IT staffers have granted us insight into the failures with which they've regrettably been involved. Our purpose in recounting these setbacks is not to focus on the failures, but to draw lessons learned and perhaps help other CIOs avoid the same traps.
Some of the CIOs with whom Senior Writers Tom Field, David Pearson and Polly Schneider spoke did so completely on the record; others requested anonymity either for themselves or their employers, and we've respected their wishes. Reading these case studies does not ensure that you'll never hatch another turkey, but the expert advice may help you avoid eating crow.
THE PROJECT
In 1990 the Washington State Department of Licensing launched its License Application Mitigation Project (LAMP), a five-year, $41.8 million mission to automate the state's vehicle registration and license renewal processes. But when George Lindamood stepped in as director of the state's IS department in 1993, LAMP was hardly glowing. The budget had swelled to a projected $51 million, the legislature changed its schedule in the course of the project, and even if the project were completed, it promised to be a colossal money-waster. "It was like the Lilliputians dragging down Gulliver," Lindamood says. "The project was stopped dead in its tracks." And even had the project not stalled, he says, "it would have been much too big and obsolete by the time it was finished." Lindamood tried to save the project, but to no avail. In 1997 the plug was pulled on LAMP, but not before seven years and roughly $40 million had been wasted.
THE PROBLEM
The project's failure stems in part from bad project management, Lindamood says—the undertaking was too big with too few solid deliverables along the way. But the chief downfall was administrative meddling. From authorization to purchasing to quality assurance, LAMP was overseen by a mix of elected officials and political appointees whose agendas were more personal than project-driven, Lindamood says. He adds, "In government, it's almost irresistible for new people to get in [office] and want to stick their fingers into these things." By design, the project was inexplicably split between in-house developers and a private industry contractor, which led to poor coordination and delays. After Lindamood left, legislators diverted money from LAMP to fund construction of an offramp at a racetrack and passed new licensing and registration laws that altered the project scope and caused further delays. "The real shame of the failure of a project like LAMP," Lindamood laments today, is that "the people who are ultimat
The Recovery
There was none. LAMP was turned off in 1997, after legislators calculated thatthe project ultimately would cost $4.2 million more annually to run than the state's $800,000 per year incumbent system. Before the plug was pulled, Lindamood washed his hands of LAMP and of his IS position, opting instead to be a consultant with GartnerGroup Inc., where he now is charged with (his description) "the care and feeding of CIOs." Asked what lessons he learned from the LAMP experience, Lindamood cites several:
- LAMP should have been bid entirely to a private industry contractor. "Government doesn't build [systems] well in-house," he says.
- The project should have been broken down into smaller, measurable chunks, rather than spread out over several years.
- Experienced quality-control personnel should have been hired from private industry and empowered upfront, as opposed to putting political appointees in charge during the project.
- Project managers should have expressed from the start exactly what conditions would necessitate shutting down the project rather than letting it limp along.
Looking back, Lindamood regrets that although he could see from the start that LAMP was headed for disaster, he couldn't save it. "The system doesn't want to hear [that a project is failing]," he says. "The system is the problem; it's the system that leads to the failure."
—Tom Field
THE PROJECT
Back in the '80s, independent Blue Cross licensees around the country found themselves in a major consolidation mode. Big blues were swallowing up little ones, and the newly massive companies that resulted were having a difficult time merging disparate claims-payment systems. Inside one of the biggest blue sharks in the water, not to be named here, the time was ripe for a major new system—one that could grow along with the company, which was looking to expand beyond health care.
The company brought in a major consultancy, conducted a study, made recommendations and acquired a new claims system. And the $8 billion company got comfortable with the idea of spending $5 million to $15 million over two to three years to complete a project it saw as critical to its continued steady growth.
Chuck Southworth, an applications manager at the project's inception and CIO by the time of its demise, says that, looking back, the directive from senior management to "go get a new system, put it in place and make it work" should have tipped off the company's IT leadership to the problems that were soon to follow. "Things handed down from on high by fiat tend not to ever work," he says.
THE PROBLEM
Southworth, now CIO of The Martin Agency, an advertising firm in Richmond, Va., says the fatal flaw in the doomed claims system lay in executives' blindness to the human factors involved. Of an IT staff of around 400, 75 of the best and brightest were selected—ahem, make that encouraged—to "volunteer" to implement the new system. They faithfully stepped up and faltered.
"The big issue was backfilling," says Southworth. "You can't have people on two important endeavors at once, so who's going to do these folks' other work while they're concentrating on this big project?" The answer from all those good and talented individuals was that they'd work in two worlds at once. "It's a basic law of physics that you can't be in two places at one time," says Southworth.
Nor did it help matters that the staffers had been motivated to accept the new project largely out of job insecurity. They knew their employer was weighing outsourcing against in-sourcing on the project—and that, for them, the former choice might spell joblessness down the road. After all, the new claims system represented the future of the company. Their willingness to take on "extra" work convinced the company to keep most of the work in-house and provided management with a chance to cast a vote of confidence in its people. Then, too, good and loyal "team players" will often take on more work than they can handle. The consequence, revealed over the ensuing months, was constant conflict over priorities. "It scarred people really badly," says Southworth.
The Recovery
Southworth was promoted to CIO around nine months into the project. He labored to bring it under control until the end of the budget year, then approached the vice chairman of the company with the bad news. "I think you'd better sit down," Southworth told his boss. "We need to kill this thing now, before it gets any further out of hand." Southworth was surprised and relieved at the reaction his tidings elicited. After a brief denial—What do you mean kill it? That's out of the question!—he stepped back and let Southworth make the call. "The whole thing actually helped build my personal credibility in the organization," says Southworth. "They respected that I had the [guts] to come forward with something so painful, and to do it so forthrightly." The company abandoned the project, at a cost of millions. "We had moved people out, rented the space, bought the system, gone through all the planning, and it all wound up on the scrap heap."
Eventually, the company took up revamping its claims system once again, notes Southworth, and succeeded. What was different this time? Better dedication of resources.
Here's what Southworth took away from the experience:
- No matter how far your staff goes to convince you they're master jugglers, don't assign anyone to more than one major undertaking at a time.
- Downward buy-in means convincing staff that a new priority is more important than whatever they've been working on. To secure acceptance, stress ownership. "When the system was subsequently deployed—successfully—it was because people believed and understood that it was really theirs" to bring to life, says Southworth.
- Don't let a doomed project run on—or die a quiet death. Admit it's failed and announce the failure.
Southworth says that, over the years, his few failures have taught him more than his many successes. "The key," he adds, "is never having to learn the same lesson twice."
—David Pearson
THE PROJECT
Four years ago, a state chapter of a well-known national consumer group embarked on what was to be an 18-month, $1 million project to replace its customer database. The new system, to be designed, built and installed by the company's IS staff, would distinguish among different "preferred customer" levels, affording the appropriate products and levels of service on demand. The good news was that the company's IS team did deliver a new system on time. The bad news was that the system didn't work as promised, handling routine transactions smoothly but tripping over more complex ones. Within three weeks of throwing the switch, managers found that the new system couldn't distinguish among customers and that some transactions were being canceled while others were kept on hold. Immediately, the database was shut down, transactions were processed by hand and new IS leadership was brought in to rebuild both the system and the strained relationship between business and IS.
THE PROBLEM
IS couldn't—or wouldn't—say no. So eager to please their business partners, IS executives said yes, they could build this system, yes, it would be scaleable and yes, it would be Y2K compliant, on time and within budget. But in reality, IS could deliver on none of the promises. The other big mistake by IS was making this a date-driven project. There was no flexibility for mistakes or unforeseen challenges; developers simply kept their eyes on the calendar and failed to speak up about any glitches they saw along the way. "It was kind of like one of those horror stories you hear about," says the anonymous CIO who inherited the mess a couple of years ago, after more than a dozen people (including the former CIO) lost their jobs because of their roles in this disaster.
The Recovery
The new CIO immediately empowered "SWAT" teams to diagnose the problems and analyze whether the best course was to repair or replace the new database. "There wasn't enough good code there to repair," the CIO says, so plan B went into effect. IS partnered with two vendors—one to design and build, the other to oversee the first vendor's work—and a new multimillion-dollar database is projected to be installed sometime in 1999. But replacing the database is only half the battle. IS also must rebuild its fractious relationship with the business side, and that project has its own unique requirements. "I'm trying to dig out of a hole," the new CIO says. "Anything negative that happens now just reinforces the perception [among business people] that IS doesn't know what it's doing." But IS is making headway. The new CIO is making sure new projects are on time and within budget, and he is convinced that his organization's recovery from the database debacle will secure a new reputation for the group. Key lessons learne
- IS must break its "code of silence" when working on challenging projects. "If [IS managers] had spoken up sooner, when things started to go wrong, 14 jobs might have been saved," the new CIO says.quickly by stepping in, assessing the problem and developing a plan of action that stated clearly when the recovery would be completed and at what cost.
- You can't do too much too soon. Like the project itself, recovering
IS's reputation is a slow process. "I'm trying to do things
incrementally to restore customers' confidence," the new CIO says. "You
can't do it all in a big bang—although you sure can tear it all down
real fast." How long will the recovery take? Can the new CIO really pull
it off? "Those are loaded questions," he says, "but I remind people that
IS didn't get this way overnight; it won't get turned around overnight."
—T. Field
THE PROJECT
In 1996 a major San Francisco bank was poised to roll out an application for tracking customer calls routed to its "elite" group of customer service reps who handled problem cases. Reports provided by the new system would be going directly to the president of the bank and board of directors. An initial product demo seemed sluggish, yet the consultants assured both IS and the telephone banking division managers that all was well. They were wrong. "The source code was so bad it took 20 minutes to load the program on the PC," recalls Jim Daviner, the systems analyst on the project, now a business systems manager for marketing at The Gymboree Corp. in Burlingame, Calif., a children's retail chain.
THE PROBLEM
The system crashed constantly, could not support multiple users at once and did not meet the bank's security requirements. IS hired a new consultant to help rewrite the application, but after three months the project was killed—resulting in a loss of approximately $200,000 in staff time and consulting fees, and a bad rap for IS.
According to Daviner, the first mistake was the bank's failure to check references and work samples of the consulting firm. Daviner caught a major flaw in the database design in time, but as the project progressed, the programming team became increasingly isolated and hard to work with, refusing to release the source code to the project managers. Daviner says the programmers didn't want the bank to find out that they were, in his words, "pretty inept."
But the root of the project's failure was a complicated reporting structure that left no clear line of command. Between Daviner, the lead analyst from the business side, and the four consulting programmers located in Arizona, there were two other layers of management from IS. Another layer, Daviner's boss, was the lead project manager and sponsor—yet she had no direct contact with or control over the programmers and left the company in the middle of the project. Worse, the project correspondence was lost in the process of cleaning out her PC, Daviner says. At the same time, the bank's IS department was being reorganized. One of the two IS managers on the project resigned to go to another company, and the other was restructured out of a job after the project's problems came to light. In late 1996, with no leadership or business sponsor to rescue a coding disaster, Daviner had to mop up the mess.
The Recovery
The project was never revived, and Daviner says the biggest loss for the bank is not having access to valuable customer information the system was to deliver. On a positive note, the business units are getting rid of unnecessary layers in projects. Today project managers have direct oversight of the programming consultants and approve all hiring of IS personnel. "Things happen more smoothly now," he says. Daviner (who, as an aside, says his reasons for leaving the bank were not related to this project) shares the following tips:
- Institute a formal review process for hiring consultants.
- Require change control documentation so that managers can see what changes were made during development, when and why.
- Ownership is essential. When sponsors or top players leave the company or the project, new owners should be identified immediately and supported with documentation.
- Assign a central manager for the project team who is the conduit for communication and decisions. Result: Everyone is on the same page rather than working in parallel and reporting to different managers.
In the end, a disorganized project team with unstable leadership wrought the ruin of an important customer application. Unfortunately, due to the ongoing IS reorganization at the bank, Daviner says that IS has not improved its project management practices—a clear example of chaos breeding chaos.
—Polly Schneider
THE PROJECT
The night before the launch of a new payroll system in a major Boston health-care organization, project managers were sweating. During a sample run, the off-the-shelf package began cutting checks for negative amounts, for sums larger than the top executive's annual take-home pay and checks that incorrectly matched employee numbers with names, recalls a former director of systems analysis who asked not to be identified. Called in to review the project before the launch, he wound up sitting in the payroll department for six months to help fix the problem.
Payroll was still delivered on time, and out of 7,000 employees, the fiasco affected the checks of only roughly 50 to 100 people. Still, the payroll office had to pull the bad checks and rerun them manually to create new ones every payday until the application was debugged. The incident damaged the relationship between IS and the payroll and finance departments, and the programming manager resigned in disgrace.
THE PROBLEM
"It became apparent the entire system was never tested or run in parallel" with the old system, says the director of systems analysis. IS did not check to ensure that an existing human resources database was compatible with the new payroll system software; the problem was that limit controls on hourly rates were not in place. An entry of 8.0 dollars per hour could be read as 800 dollars per hour if the user did not enter the decimal point, which revealed another problem: No one had been adequately made aware of the differences between the old system and the new one. "There was no turnover between development and production," the director says. Business managers were signing off on the system without really understanding how the people in their department would be using it, he recalls.
"A lack of clear leadership was a problem from the beginning," says the director of systems analysis. The main sponsor, the payroll director, suffered a heart attack three weeks before the launch date and was placed on disability; he was gone for months. Throughout the course of the project, the organization's top IT director was "uninvolved," the director of systems analysis reveals.
The Recovery
The organization hired a consultant to help and provide support, and eight months after the implementation date hired a new director of IS. An administrator who oversaw all IS-related projects was transferred to another area of the company altogether. With the new management, IS established a more formal reporting methodology and some ground rules. Outside consultants would be used on every major project, and business units had to identify a project manager for every project and provide adequate training for the staff. Overall, project managers began to forge better partnerships with users. "The effect was really a mandate to change," says the director of systems analysis. "It created the demand for a more professional way of managing projects." The director of systems analysis (who subsequently went on to become CIO of this organization and others) has a few lessons to share from the experience:
- Test, test and test again: "It's much harder to fix things afterward."
- Hire an outside contractor to complete an independent project review.
- Communicate with the business: Send a monthly letter to senior management with updates as the project progresses.
- When buying off-the-shelf software, hire a consultant who understands how the application will run in your environment. The former director says that this is the puzzle piece software vendors often can't deliver.
Looking back, the director-turned-CIO says that the bungled project was a blessing because it helped raise awareness of the company's growing dependence on IT. "Without a major catastrophe, there was a perception that IS didn't affect operations."
—P. Schneider
THE PROJECT
You do the math: Anticipating growth, a $100 million division of a $740 million manufacturing business earmarks $5 million for a new distribution and customer service system to replace its old, sputtering setup. The project is to take a year and a half to complete, involve 20 business and tech staffers, make up for 20 percent of the missing source code and set the division in good stead for Y2K.
Two years later, the CIO is fired. A knight in armor, an IT executive with years of experience fixing troubled projects, is brought in to save the day. He's informed that the challenge ahead is serious, but he's kept in the dark as to how goofed up things have gotten.
THE PROBLEM
"This project had the two deadly sins built in," reflects the old pro, who's since changed jobs and agreed to share memories of his nightmare under cover of anonymity. "They'd chosen the wrong software solution, thanks to a terribly naive RFP process, and they had no project plan." Worse still, no one owned the project. "IS thought the business users owned it, the business users thought IS owned it and the CEO thought the vendor owned it." There was an unwritten project plan with five or six major milestones, but not a single one of those had been met and no one on the project team could say when they might be met.
The new IT boss pulled together all hands and, in just two days, orchestrated a 2,000 line-item plan. Three months later, with the company's best IT talent and the software vendor's consultants busting their chops, the system broke down altogether. "It was supposed to be the show-stopper for us," says the source, "and we couldn't get customer orders through the system." Internal users began panicking; customers began complaining about incompetence. "We were sending people the wrong stuff and sending everything late."
The Recovery
Finally, three years and $4 million into the project, the CIO polled the 20 players anonymously. Eighteen said they thought the project was beyond saving. The two who held out were the project manager (a business-side person) and his boss, the VP of manufacturing—apart from the CIO, the two whose necks were most at risk and, in the end, the only two to lose their jobs.
The CIO approached his boss, the CEO. "It was kind of like telling him a relative had died," he recalls. "First he denied it, then he went through a grieving process, then he accepted it. It was just so much money for a division that size to wave in the wind." A settlement was negotiated with the software vendor, and corporate IT stepped in and assumed direct control of all IT operations in the division.
For the CIO, it was time to leave—and to reflect on some hard lessons:
- If you don't have extensive experience with the RFP process, get help from someone who does.
- Decide upfront how decisions will be made through the duration of the project. "An essential aspect of good project management is a consistent approach to the inevitable forks in the road. You can't decide by show of hands at one turn, edict on another and a secret ballot on the third."
- Look before you leap into a new career "opportunity," and don't be afraid to conduct due diligence if you're taking on a project that's already underway.
"Before I accepted that job, I should have talked to the project manager, asked to see the project documentation, studied the RFP, spoken with the software vendor and talked to some internal users," says the CIO. "Had I known how amiss this project was, I would not have joined that company."
—D. Pearson
All content copyright CXO Media Inc., 1994-2001. All rights are reserved. No material may be reproduced electronically or in print without written permission from CXO Media, 492 Old Connecticut Path, Framingham, MA 01701.
