Signposts on the US Government’s Trail of IT Failures

The U.S. House Science Subcommittee on Investigations and Oversightreport is referenced as “Questions regarding technical flaws, poor government oversight and potential contractor mismanagement on the National Counterterrorism Center’s RAILHEAD information technology program.”

As that language suggests, it chronicles a series of missteps in a US$500 million upgrade to a key government database: the National Counterterrorism Center’s terrorist watch list, which holds data on some 400,000 known or suspected terrorists. Indeed, it collects data from some 30 intelligence agencies, earning the moniker, “the mother of all databases.”

So why is this project, which falls under the umbrella name of “Railhead,” failing? And failing it is. “[R]ailhead may actually degrade the ability to provide intelligence data for use in the consolidated terrorist watch list,” according to the subcommittee report.

If it seems as though the U.S. has been down this road before, that’s because it has. Large IT projects in agencies from the FBI to the Census Bureau spectacularly — and, it seems, regularly — crash and burn, much to taxpayers’ disgust.

It is easy enough to make jokes about government failures and then dismiss them as more evidence of individuals’ ineptitude.

It seems there are certain aspects of government culture that make IT failure more likely. It just may be that failure is built into the system.

Private Sector Not Immune

There is no continuity of management within most government structures, Jeff Kalwerisky, chief security evangelist for Alpha Software, told the E-Commerce Times. “The chance that the people who are in charge or requiring this project today will be there in five years is slim. So, there is not a major sense of responsibility.”

Worse, when the next administration gets into power, its inclination is to change anything major that was begun in the preceding administration in order to put its own stamp on it. Plus, political appointees are well aware that career recognition is not built on small but well-managed, projects, Kalwerisky added.

Troubleshooting must start at the RFP (request for proposal) stage, “where you need to be very clear up front,” Brad Brown, managing director of Protiviti’s global public services, told the E-Commerce Times. Unfortunately, most government projects are handled by procurement or contract officers — who are naturally removed by several layers from the original project.

Still, it must be acknowledged that big bang IT failures occur in both the public and private sectors, and it’s instructive to look a little deeper into the reasons that is the case.

“There are plenty of stories of failed IT projects in the private sector as well,” Rita Gunther McGrath, associate professor at Columbia Business School and coauthor of The Entrepreneurial Mindset and MarketBusters: 40 Strategic Moves that Drive Exceptional Business Growth, told the E-Commerce Times.

“In fact, there are entire companies who have blamed poor performance — or even bankruptcy — on failed projects,” Gunther McGrath said. “The difficulties are the same — but in government there is often much less accountability, so the losses pile up and are more visible.”

Six Stumbling Blocks

Gunther McGrath distills the reasons for IT failures into six categories:

• lack of good governance structures;
• a “waterfall” method of systems design;
• all-at-once funding;
• scope creep;
• automating before redesigning; and
• failing to budget for “cleanup.”

1. Lack of good governance structures is indeed a key reason for failure. Often, large projects fall between agencies or between units of agencies, and it is hard to achieve consensus on what each wants, Gunther McGrath explained. “What often happens is that decisions then get made by committees and there is little accountability.”

Stakeholders must also be taken into account, Rich Dougherty, CEO of Expert Choice, told the E-Commerce Times. “A fundamental problem is a misalignment among the various parties — and that misalignment problem exists at both a strategic and tactical level.”

A good system “has to be structured explicitly to encourage trade-offs between the various stakeholders,” he maintained.

“Projects fail when you don’t have sound project program management practices in place,” Gil Digioia, vice president of federal project management sales for CA Clarity, told the E-Commerce Times.

“These include many checks and balances in the cycle to make sure projects don’t get off track,” Digioia explained, “which is especially easy to happen when there are multiple stakeholders and vendors and system integrators working on the program. You also need a program manager on top monitoring the program for unforeseen risk, and able to put the project back on track if it develops problems.”

2. The waterfall method of systems design, Gunther McGrath said, is an approach in which the system is scoped out in terms of desired outcomes; users or analysts prepare specifications; the specs are given to the programming team; and the programmers create the code, which is then tested and put into production.

“You can see that this is a relatively linear process,” she noted. “It creates all kinds of problems, beginning with the fact that often users and analysts really don’t know what a system should do until they get the chance to play with it, coders operate on specifications which may not lead to a desired result, and the project can be way down the road to development before people realize it should have been done differently.”

A far better approach, she said, is to iterate — that is, mock up the functionality to get a basic process flow, and then build it out little by little, working with future users so that when the system is finally delivered, it’s what they wanted.

“The problem with this for governments is that it is almost impossible to write an RFP or requirements document that allows for iteration and nonspecified deliverables,” said Gunther McGrath. “In the long run, though, it costs taxpayers far more.”

3. All-at-once funding. In any large scale project, you’re better off getting just enough funds to get to the next major milestone. Then, a mini-post mortem can reveal problems before a lot of expense has been incurred, and the project can be redirected or stopped while it is still inexpensive, Gunther McGrath explained. “With government projects, however, it is such a pain to get funding approval that the temptation to get funding all at once is irresistible — what that means, unfortunately, is that projects can get very large and very expensive before anyone has a good hard look to stop them.”

Fundamentally, says Protiviti’s Brown, “government IT shops have to learn to budget realistically. You can’t plan for gold standard but budget for bronze.”

4. “Scope creep” is a term that’s very familiar to many in the private sector, but “there is a tendency in government to want any new system to meet all possible needs with people thinking that if they don’t get their feature on board with this system, it will never happen,” Gunther McGrath said.

“So the scope of projects gets larger and larger,” she continued, “and if you don’t have good governance to put a stop to that, it can cause a project to become extremely costly and unwieldy. A better approach is to plan projects as a series of additional functionalities, so that people don’t feel they have to get their feature in the first release.”

5. Automating before redesigning is a common problem with government projects. Systems are often designed around processes as they are, rather than around processes as they should be, Gunther McGrath pointed out. “So, essentially, you end up automating a dumb way of doing things without making any real operational improvements. This frequently happens because people are unwilling to put the serious design effort in up front and want to get right to the programming.”

6. Failing to budget for “cleanup.” This usually occurs in the implementation part of the process, when people suddenly realize that the data they are working with has to be moved from the old system to the new system, and that it can’t be done easily, Gunther McGrath said.

“I estimate that a good 40 percent of an IT project’s budget could go into just making sure the information in the system is clean, accurate and representative of what it is supposed to do,” she concluded.