Can't donate to charity?
Volunteer computer time
or Support SETI!
R&D Sponsorship Center
October 2004

Home Page
Feature Archive
A&I Column Archive
Production Tools
State Marketing Data
US Marketing Data
World Marketing
Service Directory
Quality Assurance
3D Printing

Subscribe to Advertising & Marketing Review!
Contact Ken Custer at 303-277-9840.

Lessons from the Colorado Benefits Management System Disaster

by Glen Emerson Morris

Popular Columns
The Cost of Creativity
When bright ideas cost too much.
Desktop Manufacturing
Hits the Home Market

Someday print any object you need.
Saving Motion, Time & Your Business
Motion time studies can save you money.
A Gold Mine of Data Goes Online
The Statistical Abstract is now online, 1300+ data tables in Excel format, free.
A Process for Quality
How a formal process can improve quality.
Recommended Columns
The Greening of Expectations
It's not a fad, it's critical to our survival.
The Learning Curve to Prosperity
Buckminster Fuller predicted the resource crunch now hitting us. He also gave us the tools to deal with it.

Considering Colorado has been the home of three of the most publicized computer disasters in the US (the airport baggage handling system, the DMV system and the Colorado Benefits Management System), it won’t be surprising if these events become case studies in some future book.

If any good can come from the last two computer disasters it would be that important lessons are learned about quality assurance. Both incidents were avoidable, especially the Colorado Benefits Management System fiasco. Advertisers can avoid having similar problems with their Websites if they use a little common sense and they rely on experienced computer professionals.

In the case of the Colorado Benefits Management System, the biggest mistake made was that there was no provision for a rollback to the older, and working, system if the new system failed. Even for minor changes on existing Websites, provision for rollbacks should always be made. In the case of setting up a new system, rollback provisions should be mandatory, and the rollback process should be tested before the new system goes online.

Another probable mistake was the apparent failure to do load testing which would have alerted EDS to the slowness of the system and the problems that would lead to. It is true that load testing is relatively expensive and requires specialized engineering talent, but it’s usually worth it. So is automated testing for functionality and usability. A ballpark estimate for load testing software would be about $50,000. Add another $200,000 for automated testing software and the total cost for all automated testing software would be about $250.000, not counting the hardware required to run it. However, given the $200.000.000 cost of the Benefits Management System, the cost of automation software would have only been slightly more than one tenth of one percent of the total cost.

Automated functionality testing would have made EDS aware that the Colorado Benefits Management System ran so slowly that data entry timed out when moving between data entry screens. According to some reports 17 different screens had to billed out and it took up to 24 minutes for each new screen to load. There are several possible reasons for the system running this slow, some related to hardware, and some related to software, and all of them are avoidable. Hardware issues could be from using processors that were too slow, or didn’t have enough memory installed, or were configured improperly. Software issues could be from improper cache settings, improper software configuration, or the much worse issue of a fundamentally bad software design in the first place.

Yet another problem was requiring 17 different screens to process one case. This is a design error that would have never been allowed to happen by experienced software developers. This also suggests that no usability testing was done. This aspect of the system, combined with its extreme slowness, made the system fundamentally unusable.

Extreme work hours may have been another factor. In all probability, the EDS engineers working on the Colorado Benefits Management System were putting in far more than 40 hours a week, and the resulting disaster was at least partly due to this factor.

Since the dot com crash, companies have generally required much longer hours from employees than before. Good software engineering requires many variables be considered simultaneously. However, the ability to multitask is one of the first abilities people lose when they become tired. (This is the reason the FAA limits pilots to flying only 1200 hours a year. Beyond that time, pilots are much more likely to make mistakes, and serious ones.) As management guru Peter Drucker once commented. “Never confuse motion with productivity.”

According an industry insider, EDS has a history of hiring bright new graduates, paying them as little as possible and letting them learn on the job. The result is a workforce with intelligence but little experience. Unfortunately, this is an industry wide problem.

Software quality assurance hasn’t been the same since the dot com crash and 9/11 attack. The software industry responded to the market crash by budget cuts that targeted many of the higher paid, and most senior, professionals in the industry. The result is that nearly everyone who knew what they were doing has been laid off from the software industry.

Probably the biggest example of this is the case of Robin Hedges (AKA Robin Sterling), who founded Apple’s first QA division, and was later Director of QA for Radius, Adobe and eBay. Considered by many to be the best QA manager in the business, she was unable to find employment after the crash and eventually gave up and opened a hat shop. Her case was not alone.

A former manager for a major network security products developer cited a case where a former Apple automation engineer was laid off from a team because he was higher paid than the others on the team. The problem was that the former Apple engineer could write automation code in four hours that would take anyone else on the team two weeks to write, if they could write it all, and his code always seemed to work the first time it was tried.

There has been a management problem in the software industry for decades because most engineering managers have engineering degrees, not management degrees. Unfortunately, the skills required to manage an engineering department are vastly different than those needed to program code, and this usually shows up in the quality of the product.

Every mistake made in the development and deployment of the Colorado Benefits Management System was completely avoidable. A little previous experience and common sense would have avoided a situation that adversely affected tens of thousands of people. The real tragedy is that, given the state of the American software industry, the mistakes made weren’t surprising at all.

Glen Emerson Morris has worked as a technology consultant for Network Associates, Yahoo!, Ariba, WebMD, Inktomi, Adobe, Apple and Radius, and is the developer of the Advertising & Marketing Review Data CD.

Copyright 1994 - 2010 by Glen Emerson Morris All Rights Reserved

' keywords: Internet advertising, Internet marketing, business, advertising, Internet, marketing. For more advertising and marketing help, news, resources and information visit our Home Page.

Back to top

Economic Indicators
Census 2010
Census Bureau
Health   Labor
Commerce Dept.

It's Time to Let
A Robot
Make Your Sales Pitch!
Roy the Robot
Funded by Kickstarter