Can't donate to charity?
Volunteer computer time
or Support SETI!
R&D Sponsorship Center
January 2003

Home Page
Feature Archive
A&I Column Archive
Production Tools
State Marketing Data
US Marketing Data
World Marketing
Service Directory
Quality Assurance
3D Printing

Subscribe to Advertising & Marketing Review!
Contact Ken Custer at 303-277-9840.

Automated Website Testing: IBM and Rational Pricing

by Glen Emerson Morris

Popular Columns
The Cost of Creativity
When bright ideas cost too much.
Desktop Manufacturing
Hits the Home Market

Someday print any object you need.
Saving Motion, Time & Your Business
Motion time studies can save you money.
A Gold Mine of Data Goes Online
The Statistical Abstract is now online, 1300+ data tables in Excel format, free.
A Process for Quality
How a formal process can improve quality.
Recommended Columns
The Greening of Expectations
It's not a fad, it's critical to our survival.
The Learning Curve to Prosperity
Buckminster Fuller predicted the resource crunch now hitting us. He also gave us the tools to deal with it.

It's too early to tell, but one of the best things for Internet advertisers in 2003 could be the purchase of Rational by IBM. Rational is one of the big three makers of automated Website testing software, with a revenue of $689 million in its last fiscal year. They aren't a household word because they charge thousands of dollars for each CPU their software is run on. Though each CPU can then simulate over a dozen users, it still takes dozens of CPUs to simulate a reasonable level of traffic on a Website. As a result, the total bill for all the automated software needed to test an e-commerce site can easily run $250.000, and many Fortune 500 companies are used to paying this kind of price.

Unfortunately, for small to mid-sized businesses, $250,000 is out of the question, but setting up a complex e-commerce Website increasingly isn't. In an ideal world, the cost of automated testing software would be falling along with the price of computers, but in this case, that hasn't been happening, and it's becoming a problem.

Just consider the testing required for a server that supports 250 simultaneous users (as the Apple 4.0 Server did in 1993). Even if you could afford to buy 250 different computers for simultaneous testing, where would you put them? And who'd run them? With automated testing, each computer can be made to simulate over a dozen users (by tricking the server into seeing it as many users instead of just one). This reduces the number of machines needed from 250 down to a manageable 21. In addition, only one or two QA engineers would be required to keep the tests going.

Still, 21 licenses at $5,000 adds up to over $100,000, and that's just for functionality testing. Load testing is additional.

For those not familiar with Quality Assurance terminology, functionality testing is designed to tell whether basic features of the Website work, and answers questions like whether people can actually log in, view the Website properly, do a product search, and order products. Load testing determines how many users can be handled at once doing various combinations of functions, and the exact response time in each case.

Functions are simulated through the use of "scripts," which are essentially programs that are executed by the software testing application. A script can be created by recording the keyboard and mouse actions of a real person using the application, or the appropriate code can be written from scratch. (The competitors of Rational use proprietary coding languages to program scripts, requiring specialized training before the product can be used efficiently. Rational based its code on Visual Basic, which is understood by many software engineers, a point likely to be exploited by IBM.)

When the scripts are run by the software testing application, the target Web server is subjected to exactly the kind of real time traffic it would have to handle when deployed. The Web server would be unable to tell the difference between the testing application and real users because there wouldn't be any difference.

The complexity comes in with the depth these programs can go to. For instance, rather than take a screen shot of a button properly lit up and comparing it to a screenshot in the test, testing applications can simply look at the code for a button and see if it has been set to an enabled state. This ability saves considerable time over the screen shot comparison method, which requires a different photo for each bit depth and resolution covered.

One of the reasons automated testing software is so expensive is that it must constantly be revised to work with the latest Website programming languages, which are in a state of constant evolution. With a relatively small customer base to defray the heavy R&D involved, prices can only be high.

With some luck, IBM will understand that reasonably priced automated testing software is essential for developing Websites for small to mid-sized businesses. Lowering the price of Rational software would allow more businesses to go online in a big way, with all the quality and dependability needed for commercial Websites. This could mean a bigger market for IBM servers, and everyone else's.

By spreading development costs over a much wider customer base, IBM might actually make more money off of Rational than if Rational were to keep its premium pricing. IBM could make automated testing software commodity priced, and it sell it for about the same as most major Adobe titles sell for.

It would take a little re-thinking on the part of the smaller Website production studios and in-house production departments to integrate automated testing with their current development process, but it would be worth it.

Automation has long since become a common practice in Silicon Valley. It's not seen as an answer to all testing needs, but it's seen as part of the answer. Robin Hedges, former Director of Quality Assurance at Adobe, Radius, and eBay, says, "Automate those things that are static, manually test those things that change. And always automate the things that are repetitive. There's no point in turning your testers into monkeys."

However, until prices drop, automation may be a practice limited to Silicon Valley and the Fortune 500. Whether automated testing becomes an affordable option for thousands of businesses is now up to IBM. Even changing the price structure would go a long way towards making automated testing a viable option for smaller businesses.

Let's hope that IBM isn't stuck in a mainframe mentality when it decides on what should be Rational pricing.

IBM could probably sell 20 times the current sales volume if it reduced the cost of Rational software to five or ten percent of what it is now. This may seem drastic, but this would put the cost of Rational test software in the range of Adobe software, and the other standard tools.

So why is automated testing software so expensive? Two reasons. This kind of software is very complex and sophisticated, and very few companies need it.

On a smaller Website, manual testing may be feasible, but it's still expensive.

One metric commonly used in Silicon Valley is to allocate 25% of the total Website development budget to quality assurance. Most quality assurance professionals will quickly point out that it is mathematically impossible to test a large e-commerce Website on that budget, and unfortunately, they are correct. The best that can be done is be reasonably sure all of the major bugs have been found and fixed, and that only minor bugs remain unknown.

Another QA metric is the ratio of program engineers to QA engineers, which at the better companies ranges from four to one, to the industry ideal of one to one. At startups the ratio can exceed two hundred programmers to one QA engineer. Microsoft reportedly uses a one to one ratio, and is still noted for quality problems.

On really complex projects, even one to one may not be enough. There were cases in Webmania that proved one highly intelligent, enthusiastic, and under talented engineer can tie up a several QA engineers for years. And lawyers, too, for that matter.

Over the last decade offshore testing has become an affordable testing resource frequently used in Silicon Valley. India has thousands of testers for rent online, and Indian QA engineers go for about 20% of what American engineers go for. Indian testing services also offer overnight service, since their day is our night.

There is a downside to offshore testing. Services like electricity and Internet sometimes fail, and sometimes weather related problems come up. WebMD reportedly lost contact with their Indian QA team for a week once when the Indian city of Hydrabad flooded. The Indian QA team was trapped on the fifth floor of their office building for a week, and survived by eating out of vending machines. The American WebMD QA team couldn't find this out, and even had trouble finding out there was any kind of problem in Hydrabad at all, and India is a country where English is commonly spoken.

Another problem comes from cultural differences. Offshore testers are not as likely to have credit cards for instance, and when asked to simulate a common scenario like the credit card use of a person on a business trip, they sometimes don't have a clue as to what to do. Not only have many offshore testers never had a credit card, many have never even been on a business trip.

Still, offshore testing can be a useful resource, especially when combined with in-house manual and automated testing used in Website production.

Copyright 1994 - 2008 by Glen Emerson Morris All Rights Reserved

' keywords: Internet advertising, Internet marketing, business, advertising, Internet, marketing. For more advertising and marketing help, news, resources and information visit our Home Page.

Back to top

Economic Indicators
Census 2010
Census Bureau
Health   Labor
Commerce Dept.

It's Time to Let
A Robot
Make Your Sales Pitch!
Roy the Robot
Funded by Kickstarter