Agile Outsourcing: Testing and Quality Assurance

	Bookmark This Page Email This Page Format for Printing Submit an Article Read More Articles
	Related Tools & Articles Agile Outsourcing: Software Design and Agile Methodologies Agile Outsourcing: Requirements Gathering and Agile Methodologies Agile Outsourcing: How Agile Methodologies Help Software Development Outsourcing
	Discussion Forum "We are new to software development. We are in the process of putting together an RFP for the creation of a Web service. We have a pretty good idea on the budget for the coding. But what is a reasonable assumption, percent-wise, for software testing?" Contribute to this Discussion

By Nari Kannan

Is software testing the same as software quality assurance? Quite certainly not! Testing checks whether the software is doing things right; quality assurance covers larger ground, namely, whether the software is doing the right things. Software testing is about whether the software is working properly without errors -- system as well as logical errors. Software quality assurance ought ideally to cover other areas, such as whether the business problem is being solved effectively or (on a narrower scale) whether the software as designed and implemented fulfills the requirements set out at the requirements stage. Some experts believe that the scope of quality assurance extends to the formalizing and improvement of the testing process itself.

Successful outsourcing projects are the ones that strike a good balance between testing and quality assurance throughout the lifecycle of the project. Software that works as set out exactly in the requirements document but that doesn't solve the business problem at hand won't be called a successful project, no matter how few bugs there are. Conversely, software that solves the business problem effectively but is unreliable and bug ridden won't do either. Refining requirements as you go invariably may need changes in test plans, and so the testing work may increase the more you do quality assurance!

In this month's column, I explore how agile methodologies offer probably the smoothest way to ensure this balance between testing and quality assurance. I explore some typical problems that arise in testing and quality assurance and how agile methodologies address them effectively.

Typical Problems

Increasing Use of Web Interfaces

Client based interfaces are increasingly giving way to HTML and Web-based interfaces. Most software with client interfaces will probably have Web-based versions sometime soon. Software as a Service (SaaS) for the most part will be Web-based. Ask anyone who has tried porting a rich client interface to an HTML or Web- based one and you'll quickly hear a litany of difficulties in designing easy-to-use software for the Web. HTML and Web interfaces are not as malleable or ductile as client interfaces and invariably require many iterative efforts. This is simply due to the flexibility of HTML as compared to client-based interfaces. It's difficult to nail Web-based interfaces in requirements and design precisely or easily given where we are with HTML today.

Bugs vs. Flow of Tasks in Software

Looking for bugs in individual modules is easy. Predicting problems with task flow in software is hard. In fact, it's a well-accepted belief that in software products only a v3.0 or v4.0 is when the flow of the software is finally understood and fine tuned, for ease of understanding and use. Designing software in the abstract never works well enough for proper quality assurance. You can always test whether the feature is working properly or not. Whether the flow solves the business problem effectively can't be known as quickly.

Regression Errors, Long Development Cycles and Effect of Employee Turnover

Long development cycles invariably mean changes in business conditions and requirements and hence software design. In long development cycles you also face the prospect of personnel turnover, which affects the project that much more. Gaps in understanding of the requirements and design from one employee to the next invariably lead to problems in quality assurance, even if not in testing. Regression errors or errors that were fixed before start creeping back into the code, because of long development cycles and possible miscommunication among personnel coming and going.

Dissatisfaction Due to Changing Requirements and Design

The US Federal Bureau of Investigation (FBI) Web site offers a fascinating case study of what went wrong with its mega-million-dollar, Virtual Case File (VCF) project. (The link is included in "Useful Links," below.) The original project was scrapped after about three years and a new project started. What is interesting is that the new approach includes a lot of experimentation with off-the-shelf technologies and many, many phases of development and testing, just like agile methodologies! Like this project, many an outsourced project gets canned even if the software delivered is 100% up to specifications, has low defect or bug rates and the code is excellent and conforms to coding standards set out at the beginning of the project. They may get cancelled because they may have solved the wrong problem -- perfectly! Unfortunately, the world doesn't stand still while requirements are converted into beautiful code!

Software Reliability -- More Art than Science

Software reliability is more an art than a science. This is true especially in the age of Web-based interfaces and server based software that works across continents and many intervening Internet routers, hops and servers. A user in New York City may not be able to see your online application because his or her ISP's router is malfunctioning while it works perfectly for a user in San Francisco. Designing for software reliability must take into account all of these additional variables that pop into the picture that don't exist when the software you use sits on your desktop like Word or Excel!

How Agile Methodologies Address These Problems

Iteration Better than One-time Enumeration

Due to their iterative nature, agile methodologies provide a way to avoid many of the problems that plagued the FBI's VCF project. Earlier in the development cycle you get to kick the tires, change your mind and fine-tune what you want as you go along. For Web-based interfaces, there's no better way to achieve testing and quality assurance excellence at the same time than to do it over many iterations, observing all the while how it behaves across users (especially geographically dispersed ones!).

More Test Cycles Mean More Chances to Ferret Out Bugs and Fix Them

Simple logic tells you that instead of testing software twice during its lifecycle, if you test it 10 times, you're likely to find more bugs and fix them. This is especially true if the older tests are fully automated and you have the time to come up with more complex combinations of inputs to test. Agile methodologies when properly applied, by design, include many more test cycles than non-agile ones.

Earlier and Faster Feedback about Requirements and Design

The FBI VCF project may have had the chance to do a lot of course corrections earlier on in its project lifecycle if the people behind it had taken the approach that they took later. In the second attempt, they designed as many as 15 phases to the project, many of them involving experimentation with off-the-shelf software and newer technologies. This is probably as close to using agile methodologies that large government projects are going to get. However, private companies doing outsourcing can, of course, do the iterative approach with fewer constraints.

Immunization for Effects of Employee Turnover

Agile methodologies provide a natural buffer against the ill effects of employee turnover. If testing and quality assurance were performed 10 times instead of three times in a non-agile development project, personnel turnover wouldn't affect the outcomes as strongly. Communication problems and problems of understanding can be fixed earlier on in the development cycle with the new employees, minimizing the problems that arise out of employee turnover.

Automation of Test Suites

Automation of test suites along with agile methodologies provide an effective way to increase the range and depth to which testing is done. These days, automation test suites that allow you to specify a test and designate results as Pass or Fail enable the automatic testing of code at a module level, increasing the likelihood of catching regression errors automatically. This enables the testing and quality assurance people to constantly build and enlarge the test suites themselves, testing for an ever larger set of conditions. This builds a level of reliability that may not be possible with traditional methodologies. Agile methodologies may make possible enough time for load testing and stress testing even earlier on if the rest of the functional testing is automated.

Implications for Outsourcing Clients and Service Providers

Outsourcing creates a lot of friction in projects, even if the deliverable is on time, on budget and exactly to specification. Perhaps somewhere along the way, the specification needed changing because the old one no longer solves the business problem appropriately. Agile methodologies provide a new way to think even about outsourced software development.

Iteration helps address many deep issues and problems in both software testing and quality assurance. Agile methods provide precisely this capability. While outsourcing situations may not provide the same level of methodology flexibility that an in-house project may provide (due to contractual and legal obligations), it's still worthwhile to consider how agile methods will enable a service provider to provide software solutions that solve business problems and not just fulfill a business contract.

Many large outsourcing deals fall apart because of inherent problems with some of the non-agile software methodologies and how they dictate the conduct of a development project. Agile methods provide innovative ways for outsourcing service providers to deliver software solutions that are delivered faster at a higher quality at a lesser cost. If only they could come up with bold and innovative ways to restructure contracts so that the goal of the project was attained. The goal is not the successful fulfillment of a business contract but the successful resolution of a business problem. The latter will be what ensures a win-win for both parties in an outsourcing context in the long run!

Useful Links

Ajira
http://www.ajira.com

A statement by Robert S. Mueller, FBI's director, before a US Senate Subcommittee
http://www.fbi.gov/congress/congress05/mueller020305.htm

About the Author:

Nari Kannan is CEO and co-founder of Ajira, a company that designs and develops service process management tools. Nari has 20 years of experience in information technology and started out as a senior software engineer at Digital Equipment Corp. He has since served variously as VP of Engineering or CTO of five Silicon Valley startup companies dealing with a variety of problems in IT consulting, automotive claims processing, human resources and logistics applications. He has written a number of articles in the area of Six Sigma, services and BPM. Contact Nari Kannan at nkannan (at) ajira.com or visit http://www.ajira.com.

Rate This Article:  Current Rating: 3.20   Poor    Excellent                1    2    3     4    5   View all top-rated articles

Copyright © 2003-2008 – Sourcingmag.com, CTQ Media LLC. All Rights Reserved
Reproduction Without Permission Is Strictly Prohibited – Request Permission

---

Publish an Article: Do you have a sourcing tip, learning or case study?
Share it with the largest community of Outsourcing professionals, and be recognized by your peers.
It's a great way to promote your expertise and/or build your resume. Read more about submitting an article.