How a Start-Up Software Company Outsourced Testing

If you're in a start-up software situation and you finally have a reference customer willing to try out the new offering, you don't want to blow it by giving them a version of the software that's not really ready to be used. Yet maybe you don't have the budget for a full-blown beta testing program. What do you do?

That's the situation the subject of this case study — a software company in Silicon Valley — faced. They estimate the solution they went with — a highly targeted outsourcing project — cost them a fraction of the expense of a full-blown beta program. Plus, they accomplished in a few weeks what it might have otherwise taken them six months or longer to achieve.

Reynaldo Gil, CEO of Commendo Software, has been working in the tech industry for three decades. When he started mapping out his latest endeavor — a company to create tools for "capturing the Web experience" — he knew he wanted to go with what he calls the "fabless model." That’s a term that comes from the semiconductor world, where, according to Wikipedia, a company “may concentrate its R&D resources on the end market without being required to invest resources in staying current in semiconductor technology.” It “achieves an advantage by outsourcing the fabrication of the devices to a specialized semiconductor manufacturer called a semiconductor foundry or fab.”

His company, which, he said, has fewer than 10 people, would focus on their core competencies and outsource everything else.

What was core? "We're creating intellectual property. The core is owned and developed internally. But like any product, 20% is things you have to protect," said Mr. Gil. "Eighty percent is things that work around your software — that's commodity."

He chose Sierra Atlantic as the service provider, since he had a longstanding relationship with the firm. The provider has offices in Fremont, CA, where Commendo is located, as well as cities elsewhere in the US, the UK, India, Malaysia, Thailand and Singapore. It offers services in enterprise application management, software product development and testing, application development and maintenance, software integration and Web work.

The first project Sierra Atlantic undertook for Commendo was a testing process unique to the specific software being developed.

Commendo's tools capture the user's Web experience like Adobe Acrobat captures a document experience. Reference customer Cemex, the huge cement maker based in Mexico with locations worldwide, wanted a utility that would allow executives to record whatever they wanted off the internal Website, then have it in a form that they could pull up on business trips — without an Internet connection, a long list of Web page bookmarks or a collection of files saved to a hard drive. A salesperson, for example, could go into the office and record pieces of the Website that he or she wanted to share with a customer.

In the software business, typically, a product goes into development, then alpha testing, then beta testing. At each phase, the developer opens up the software to more and more users. But some software is so specialized, a beta program might involve only 10 different potential customers, to get a variety of conditions — diverse hardware and operating system platforms, combinations of installed applications, and other variations. Plus, the software company needs to put people on the beta to manage those customer relationships, thereby increasing staff size. "By the way, of those 10 customers, you're probably going to lose 80% of them because they're going to be unhappy with the results," said Mr. Gil.

Plus, the customer testing process is pricey. Mr. Gil estimates that most start-up companies spend $2 million to $3 million getting through a beta program in the field.

The challenge, said Mr. Gil, was that Cemex expected the software to be "mature." Yet they were also Commendo's lone client at the time.

"We had a need to go through a testing process in a very accelerated fashion before we delivered the product to Cemex," said Mr. Gil.

Obviously, the company didn't want to risk losing the goodwill it had with this new customer by putting it through the hassle of trying out a bunch of different software versions.

"Our product has to deal with some piece of Web that's developed by some Java guy, some piece of Web developed by some Microsoft guy, some piece of Web by SAP," said Mr. Gil. Was there some way that Commendo could create a lab to emulate many of the conditions that might surface in Web usage among its Cemex community of users — all the Web development platforms, browsers and multi-media programs in use at the company?

Sierra Atlantic brought an interoperability lab set-up to the partnership, which took the place of a broader beta program. Here's how the project worked.

The service provider put a single technical project manager — based in its Fremont office — on the work. During the actual testing process, it tapped about half a dozen people from its offices in Hyderabad, India.

Early on, the whole team from both groups met by conference call. "People need to know who's on the other side," said Mr. Gil. "I think it's important to establish direct relationships with the people involved. Once the project's kicked off, you can partition off the work. then somebody can manage the relationship." That was the job of the Sierra Atlantic project manager.

The project manager attended on-site status meetings weekly through the six-week duration, and the larger groups continued having checkpoint meetings across teams as exceptions or problems surfaced. The project manager acted as the liaison to the testing team as the work progressed.

Both parties — provider and client — came up with the tests, the combinations of hardware and software, against which the software would run.

"This is the value that you should expect from any partner," said Mr. Gil. "They have to bring their insights. They're experts in their field. I look for the best partner in any category to bring that experience so I can take advantage of it. [Sierra Atlantic] tested things that we hadn't considered, which is one of the values that they brought."

The client created "virtual test environments" that it could then package up and send offshore for the actual testing.

This approach eliminated many of the communication issues common to new teams working together for the first time.

"One of the biggest problems with outsourced testing is if something goes wrong, it's hard to recreate what happened," said Mr. Gil. "You don't have an accurate record of what tests were done, whether they were done correctly, all that stuff. With [this] process we were able to eliminate a lot of those kinds of human factors that affect tests."

Because the software itself captures the process taking place on the Web, the tester could run the virtual package against the actual set-ups in separate windows or separate computers to compare the two. When anything went wrong, the experience was recorded with Commendo's software, which allowed the developers to isolate the specific problems.

"By using these techniques, we were able to eliminate 80% to 90% of communication problems and focus on the real value — which was setting the objectives of what we needed done and clarifying things, rather than fire-fighting," Mr. Gil said.

"It was only a matter of a few weeks of lab times. But it replaced a process that normally would have been two or three iterations of field testing. And so based on the results — and based on how fast we were able to identify problems that had to be fixed — we shaved probably a year to a year and a half off the cycle time of the product," said Mr. Gil. "We estimated that we matured the product three versions, which is pretty significant."

He said Commendo delivered the solution to the customer in 2004 and hasn't had a single service call on it. "We anticipated a bunch of different conditions that the customer didn't have to [deal with], which we were very happy with."

Key to the success of the project was experience on both sides in managing the relationship, said Mr. Gil.

The work was done on a fixed fee basis — though exceptions cropped up, which raised the price. "For all intents and purposes, we stayed within the budget we had set," said Mr. Gil.

Now, Mr. Gil expects to work with Sierra Atlantic on the next phase of product evolution — the development of a Website that will allow Commendo to make its software available to a wider community of users. "We're thinking of giving away the software, and the software will be paid through sponsors," he said. "The site will be a place for people to do self publishing, or whatever their Web experience is, and then there's going to be 80% of the population which consumes what these people are creating."

Useful Links:

Commendo Software
http://www.commendo.com/index.html

Sierra Atlantic
http://www.sierraatlantic.com/

Cemex
http://www.cemex.com/