This article originally appeared in the Sacramento Business Journal on Aug. 16, 2013, and is posted here in its entirety and without additional edits.
Why Have So Many State IT Contracts Been Bungled?
By Christopher Arns
Take a lumbering multinational software company. Add a large helping of bureaucracy. Bake until overheated. What’s the result?
California’s favorite recipe for a government technology disaster.
“Many state projects could have had a better outcome if done by a local firm,” said M Corp partner Alex Castro, “simply because those firms live and die by their reputation here.”
The most recent example: In February, officials scuttled a massive overhaul of the state’s payroll system known as the 21st Century Project. The job was $250 million over budget and four years behind schedule. A test run in the California State Controller’s Office revealed glaring flaws in software designed by SAP Public Services, the Washington, D.C.-based firm chosen for the work.
Controller John Chiang blamed SAP for the disaster. The Senate Office of Oversight and Outcomes pointed the finger at Chiang, slamming the controller’s office for poor oversight and for misleading lawmakers about the size of the mess.
SAP spokesman Andy Kendzie said the company supports the Senate staff report. It “details a very good picture of what went wrong and where fault lies.”
The pattern is not new. Many of California’s most prominent tech projects in the past few years have suffered colossal delays and significant cost overruns — more than $2 billion alone for seven big projects since 2011.
The California Department of Technology, which last month took control of the state’s largest IT projects as part of an effort to improve efficiency and prevent boondoggles, says the state’s massive and complex technology needs are largely to blame.
“We’re building our oversight capacity based on those lessons learned,” said department director Carlos Ramos, who also is the state’s chief information officer.
But others point out what seems to be a common thread: Big projects involving big information technology consulting firms such as Deloitte, Accenture, Hewlett-Packard Co. and SAP seem to have big troubles.
“The problem on our end is you have these companies that are replete with failure,” said Chiang.
Critics of California’s IT track record also point to a Byzantine state procurement process that they say hampers competition and curbs innovation while repeatedly rewarding large companies for shoddy work.
For smaller tech firms, especially Sacramento companies with both federal and state consulting experience, the problem is especially frustrating. They have the background to do the work but feel shut out of the process.
“It’s frustrating. It’s been going on for years,” said Martin McGartland, president and CEO of Natoma Technologies, an IT consulting firm in Sacramento with roughly 50 employees. “It’s not even incrementally getting better.”
An Ongoing Problem
Let’s be clear — catastrophes in government technology are not exclusive to California. They’re not even limited to the public sector.
According to a study by the global consulting firm McKinsey & Co., 66 percent of 5,400 public and private IT software projects it studied busted their budgets and 33 percent didn’t meet completion deadlines. And 17 percent of projects fail so badly that they’re known as “black swans” in the IT world, the name for ventures with cost overruns of at least 200 percent. The numbers were similar regardless of industry.
But there just seems to be something different about California.
For one thing, its IT needs are enormous. The state Department of Technology has nine large systems-integration projects in the works right now. The combined value: $3.75 billion.
But the list of problems is equally impressive. Since 2011, at least seven contracts larger than $100 million have been plagued by spectacular delays, cost overruns and even cancelations.
Two huge projects have been killed just this year, including the controller’s 21st Century Project and a $208 million upgrade of the driver’s license and vehicle registration system at the California Department of Motor Vehicles. Last year, the Judicial Council voted to terminate a courts management system because of state budget cutbacks. The council already had spent more than $500 million on the project.
In total, the state paid out nearly $900 million on those three projects before canceling them.
And the process may be repeating with the Financial Information System for California, a $616 million overhaul of the state’s fiscal management system. It’s the largest IT modernization program listed by the Department of Technology, and state officials say a new multi-phased procurement process was supposed to weed out unqualified IT vendors. But the project has suffered several high-profile staff defections, struggled to fill vacant positions and lacked funding for years. A June status report from the California Department of Finance revealed the project recently missed 106 noncritical benchmarks.
State Bureaucracy Draws Fire
There’s disagreement over what’s causing the headaches.
Business owners in the IT industry claim California takes longer to develop and award government contracts than other states. Even the United States government — by no means a nimble bureaucracy — seems to pick vendors more quickly, according to several Sacramento firms.
State Department of Technology spokesman Anthony Lewis disagrees. “California’s projects are large and complex,” he said. “In our discussions with IT leaders with other states, our experiences were not unusual or uncommon.”
But Carol Henton, vice president of state and local government at industry group TechAmerica, said her members have called California worse than many other governments; they report an average wait of 18 months for agencies to develop and award a project. “That’s if you’re lucky,” she said.
Local IT exec McGartland said the state often takes at least three years to develop a project, request bids and award the contract. According to a 2009 report from the Department of General Services, the state averaged between 29 and 78 months to prepare an IT project and select vendors.
The report blamed extensive revisions to project requirements, inexperienced users creating the proposals, and a lengthy review process that drags more people into the process.
The practice takes roughly 90 days for similar projects at the federal level, McGartland says.
“The procurement cycle in the state is completely out of control,” he said.
When more time passes, technology evolves and becomes outdated, forcing firms to request changes to the project. Agency staff members sometimes underestimate costs and have to revamp budgets when they delve deeper into a system’s overhaul.
That’s one thing that went wrong with the controller’s payroll upgrade, according to a report from the nonpartisan Legislative Analyst’s Office.
“When they began, the project leadership and often state officials perhaps weren’t exactly aware of where they would ultimately go,” said Chas Alamo, a fiscal and policy analyst at the LAO. “As the project developed, a better understanding of the challenges” led to revisions.
At the same time, California has a reputation for micromanaging government projects and demanding excessive customization, which technology executives say stifles innovation and adds time, cost and confusion. McGartland said he’s seen state contract proposals with hundreds of specific solutions described.
According to SAP’s Senate testimony, the state requested 126 customizations after the 21st Century Project was supposed to go live. The company claimed the requests were “extraordinarily high compared to other large SAP payroll systems.”
“The amount of customization that has to go into implementing a tool is often so severe that it’s beyond the core capability of the tool,” said Alex Castro, a partner at M Corp, a Sacramento IT consulting firm with 105 employees.
State spokesman Lewis said the state believes giving detailed directions to consultants is actually a good thing.
“The more micromanaging we’re doing up front, it will actually lessen the confusion in the implementation and development process,” he said.
Question of Quality
The delays can have another side effect. Companies usually list their most experienced staff members on bids submitted to the state. But if agencies take longer than expected to award the contract, companies will reassign those workers and often replace them with less experienced staff.
That’s apparently what happened with the state’s failed 21st Century Project. According to a report from state analysts, SAP staff listed on the original agreement for the 21st Century Project never worked on the venture for the controller’s office.
“They were concerned about the experience of the people that the vendor was putting forward on the project,” said Lourdes Morales, another LAO fiscal and policy analyst who studied the work.
State officials suspect those workers didn’t have the skills to grapple with a project of that magnitude, and their lack of experience may have contributed to the venture’s disaster.
Chiang said the process needs to be reformed so the state clearly can assess staff experience after picking a vendor and improve accountability.
“When you have these big projects, how do you identify the top talent within a company?” said Chiang. “Are you getting their A team, their B team, or their C team? Are they bringing in people they just brought in from the minor leagues?”
Of the larger firms involved in the most recent IT debacles, only SAP would talk with the Business Journal about why they believe these projects had delays and other problems.
SAP’s software was blamed for problems that plagued several government payroll contracts before the controller’s office chose it in 2010 to replace BearingPoint, another contractor the state fired a year earlier. Those projects included flawed payroll systems in Ireland and the Los Angeles Unified School District, each experiencing millions of dollars in cost overruns and years of delays. The state picked SAP anyway.
SAP spokesman Kendzie dismissed the notion that his company has a poor track record.
“Our products and software solutions are successful in tens of thousands of public sector engagements, including hundreds in the state of California,” he said.
More Competition Needed?
Smaller firms also believe the state’s tendency to select large firms also leads to less accountability because those companies often turn around and pick subcontractors to perform most of the work.
In that case, government officials have an even harder time verifying talent levels of people on the contract. “It’s not their name on the line,” Castro said.
Some also believe the process needs more competition and better ways to judge firms for their past performance. Local firms believe they are often shut out of the bidding process because the state favors bigger companies with recognizable names despite their past failures.
“A lot of these large firms, regardless of how poorly they perform, are selected over and over again,” said Castro of M Corp. “I think that’s unfortunate.”
The state’s chief information officer disputes the idea that smaller companies are locked out.
“We do put in requirements that somebody be able to back up their work,” said state CIO Ramos.
One example: The California State Lottery recently sought bids for an upgraded sales management system. Anyone could submit proposals — but only if a firm had previously worked with one of 17 state lottery systems with at least $1 billion in revenue.
Department spokesman Lewis said the state tries to accommodate smaller firms, but that some projects are just too big for them.
“It’s a very difficult balancing that has to be done,” he said. “You want to have as much competition as you can, but you want to have reasonable assurances that these people can deliver.”
Castro said those types of mandates shut out smaller firms with proven track records. He said he thinks between 30 and 50 percent of state IT contracts could be done by local companies instead of multinational corporations with failed work on their record.
“There are times that large system integrators fit better than a local firm, no question,” said Castro. “But there are many projects in the state that could have easily been done by a local firm that would have had a better outcome, simply because those firms live and die by their reputation here.”