How To Hire Best-of-Breed Programmers for Your Project

	Bookmark This Page Email This Page Format for Printing Submit an Article Read More Articles

By Dian Schaffhauser

Face it. On the surface, there’s really little to distinguish the typical service provider that offers application development. They have a bunch of developers on staff, project managers, business analysts, sales people and so forth. They offer the promise of scaling up and down as your business needs change (and as long as that’s written into the contract). They provide programming services as a time-and-materials operation or as a fixed-price job. They brag up their quality maturation level. They talk about the 24-hour development cycle, particularly if they’re located offshore. But, really, isn’t a programmer a programmer a programmer?

Maybe not. TopCoder Software, a Glastonbury, CT company, literally calls on the services of thousands of programmers all around the world for its development work -- and only the best results end up getting used in the code for their projects. They believe their approach saves clients money and time.

How do they pull it off? This case study focuses on the service provider rather than a client story because we think its approach to the work is unique.

First, you have to recognize a few b-r-o-a-d generalities about software developers. They can be incredibly competitive. Also, they tend to love games and will devote themselves doggedly to solving puzzles. And evaluating them for hire is tough; they may or may not have great interpersonal skills, may or may not know a particular tool, but still be excellent in other ways; and what’s applicable to network people -- certifications, years of work experience and other measures -- doesn’t always apply to these folks.

Robert Hughes, president and COO, and the other principals of TopCoder knew this going in when they started their company. After all, they had run a software development consulting firm from the mid-’80s, which at its height had about 600 people.

During those years, Mr. Hughes said, “We put an awful lot of focus on recruitment and training only to be relatively disappointed with the skill level of people that were out there and that we could find. We spent an awful lot of time and money and effort on trying to find those people and making sure that they had the baseline logic and reasoning skills and coding skills...to be productive for our clients. Try as hard as we could, we still had relatively high turnover, much of it not voluntary... The people we were bringing in we just didn’t feel right putting out at clients and couldn’t keep our quality level up.”

That company was sold in 2000, just before the tech downturn.

Therefore, the next company he and the others created would tackle the problem of finding top-notch programming talent in a new way: through tournaments and competitions.

Here’s the idea: There are two types of competition. One is an algorithm contest: “Here is a problem. Write a piece of code that when we assault it with test cases is going to return the right result in every case,” Mr. Hughes explained. Not only does it test the programmer’s ability to write code that does some function correctly, but it’s got to be written quickly. These problems are created by a core group of 40 to 50 participants -- “members” in TopCoder parlance -- and then put through a review process.

This type of contest, run weekly, can attract a couple of thousand programmers from around the world in the first rounds, according to Mr. Hughes. Spectators can register as well, to watch the contests and look over submitted entries.

Once the competitors have posted their entries, the company runs the submitted code against a couple of hundred test cases. “As long as it passes all test cases, then it is deemed to be an accurate solution,” said Mr. Hughes. “So really, the competition side of it [tests] how quickly can they come up with a solution that works. There are also some execution limitations to it. It has to execute under a certain time limit so that there [are] efficiency requirements built into it.”

The other track is a component development contest. The competitors -- about 200 active participants -- write reusable commercial-grade software. The specifics for the software come from client projects. In other words, TopCoder gets a programming job, breaks down the job to its components, and then runs an online competition to get those components written.

For the judging of the component side, the things that are submitted for the client applications is a peer review process where judges are given score cards and specific criteria customized for the component being evaluated and scored. “We will come up with an aggregate score that declares who the winner is on the component side,” said Mr. Hughes.

The key to getting both kinds of the competitions to run smoothly -- aside from the attention of a developer membership that numbers above 55,000 -- is in the specifications. “There can be no ambiguity in the problem,” Mr. Hughes said. “It has to be extremely clear, exactly what you are asking for or else you are going to have several hundred developers coming back to you saying, ‘Now, I can interpret this three different ways,’ and they are vocal.” Also, the results need to adhere to standards, “so that it is very easy for us to piece those together and build a complete application for a client,” he explained. That requirements phase will evaluate integration needs, data feeds, output expected, and subsystems that the code needs to work with. (Useful Links below includes samples of component specs and requirements.)

The winners for both kinds of contests gain money and fame. Top contenders have their photos and stats posted to the site. A ratings process “similar to the US Chess Federation” ranks them. “When people compete in chess matches, they have an expected level of performance. Based on how well they do against that expected level of performance, they are given a revised rating each time they compete,” explained Mr. Hughes.

But contestants aren’t the only winners. On the component side, the client gets the software it needs quickly and presumably more cheaply. (A profile of one client, ING Americas, that ran in ComputerWorld’s January 10, 2005 issue, quotes a CTO as saying that the TopCoder approach to one project cost his company $20,000 to develop, vs. $400,000 to code it in-house.) And TopCoder gets components that in many cases -- depending on whether the client considers it proprietary -- can go into its library to re-use on future projects. TopCoder also gets something else, as Mr. Hughes points out: “All the while, [the programmers] are giving us an awful lot of data on their skills...”

That library of components actually supplies 30% to 40% of the code that a typical programming job requires, Mr. Hughes said. The remainder of the project is what is put out in the component competition. TopCoder places a project manager into the site where the service is being provided -- usually long enough to gather the specs. Once the requirements are written, that project manager returns and gets buyoff from the client to verify that the specs match requirements. This is done on a time and materials basis.

From there, the specs are put out to the “community,” which will attract a dozen or two dozen submissions for any given component. Communication among participants and the company takes place in online forums hosted by TopCoder. The company pays out for both the first and second ranked submissions (in case the first choice doesn’t work for some reason). “By compensating the participants, we are also able to take ownership of the submission so there are no [intellectual property] issues associated with it,” Mr. Hughes said. This phase of the work is typical fixed bid for the client.

Of course, this approach to building software has its challenges. “We haven’t found that client yet that doesn’t change [its] mind,” said Mr. Hughes. When that happens, the component requirements are retooled and go back out in a new competition.

The other challenge cited by Mr. Hughes: “If somebody is asking for modifications to an existing system or there is heavy, heavy integration where this is basically a module or a subsystem of some other large application, depending on the nature of it and the technologies involved, there may be some limitations there.”

Mr. Hughes believes the TopCoder approach to software development speeds up the process. “Once we have those specifications in place, we have this enormous work force that can be working at any time on any of the problems and submitting them,” he said. “The component element of it speeds it up quite a bit. It scales it quite a bit. Depending on how large a particular application is... it is still going to take us the same amount of time. We are either going to release 10 new components this week or 20, but either way, we have a large enough work force going at it and beating the timelines that we can do a very large application in a very short period of time. The added benefit is that we are only going to accept the best solution, so we are going to have a large number of submissions and be able to cherry pick the best of the submissions.”

The final phase of the development process -- putting all the pieces together -- is also done in competition. “Once all the components that are necessary for a particular project are completed... we put that out as an application project for the members to work on together,” Mr. Hughes said.

Implementation at the client site is the final pain point in the TopCoder methodology. “Even though we have a high success rate with our clients, that is probably our biggest challenge,” admits Mr. Hughes. “It doesn’t work as much like a machine like the rest of it does. It requires some care and feeding to get that to happen [right].”

Another challenge, which may bother you if you’re trying to achieve gender diversity in your workforce: About 96% of the participants are men. Mr. Hughes said he and others at the company are trying to come up with ways to attract more women, but he acknowledges that “it’s difficult.”

Also, postings in the forums hosted by TopCoder also suggest that the competitions attract a younger crowd than you’d find in a programming department. Students make up a large number of contestants for the algorithm contests.

But an advantage to the TopCoder approach is the international audience it has found. Mr. Hughes estimates that half of the contestants are Americans and half come from other countries. That diversity plays to TopCoders’ benefit, Mr. Hughes believes, since coders tend to be stronger in different skills depending on what part of the world they come from. “The US base is certainly strong and highly rated,” said Mr. Hughes. “India is probably not as highly rated -- certainly in the algorithmic side -- as you might expect. On the algorithmic side is where we see Russia and, in fact, China doing well. On the component side, it is Eastern Europe, Poland, Romania, Ukraine. I think you can contribute a lot of that to familiarity with object oriented development and modeling... I think the Russian competitors can quickly come up with a solution to one of our algorithmic problems. They can figure it out scientifically, very quickly, and hammer out the code to solve it correctly. But in terms of attention to detail and being able to complete the documentation that is required to successfully win one of the component contests, Eastern Europe seems to do exceedingly well.”

One aspect of the TopCoder approach that’s worth noting: Contest winners are paid the same amount no matter what country they live in. Since at least two of the company’s competitions (in-person tournaments held twice a year) pay $20,000 to first place winners, the stakes are high.

So, aside from selecting TopCoder to handle application development work, how can you use what it knows about developers to your advantage? One approach is to hire the company to host competitions specifically to help you find programming talent for hire. Google wanted to staff up an office in India. TopCoder set up online, elimination rounds to qualify people. Participants had 24 hours to come in and solve a particular problem that would be relevant to the type of work a programmer would do while working at Google. The top 500 scorers from that had a follow-on round, which reduced the competitors to 250. A third round shrunk the group to 50, who all came on site for a final round. Each of those 50 received some form of prize for their participation, as well as the chance to win cash. Presumably, Google recruited many of those finalists to staff its India team.

Of course, not all hiring companies are as attractive to potential hires as Google might be. But Mr. Hughes says even those lesser-known companies have had success in wooing really bright people through TopCoder, which puts out job listings to its membership and helps its clients to identify the best candidates and recruit them for employment. In this process, TopCoder evaluates not just the technical skills but also “a general screening in terms of past work, how they are able to work on teams, their general experience level, what they have done with production systems vs. development.”

Interestingly, Mr. Hughes says “soft skills” are over-rated for tech folks. “I think it is relatively popular now for people to say, ‘OK, for [computer science] students coming out, they are going to need project management skills, organizational skills, to be able to communicate well and to do all those things to be successful. I tend not to agree with that,” he said. “I think that they can be strong developers and technologists at their companies and do extremely well. I think they come out more polished now than they ever have before. Everybody has a pretty good handle on those types of things. Being able to work on a team, being able to communicate effectively, being able to manage up and down. What I think it is lacking is the basics. It is mathematics, it is engineering, and it is very low-level base stuff that makes everything else so much easier when they have a command of it. Being able to figure out how an application should work, design systems, do all those types of things, [that] starts with the fundamentals and that is where I think all the attention should be paid.”

A bit of final advice from Mr. Hughes: Don’t judge a programmer by his or her cover. “The more egotistical, the more bombastic they are about their skills, they are also the first ones to over-engineer something that could otherwise simply be done. I hate to propagate the geek mentality, but it is usually the quiet, meek people that -- if you are able to effectively tell them what you want -- will deliver it back to you.”

Useful Links:

TopCoder Software home
http://www.topcoder.com/

A sample component from the TopCoder “Component Catalog.” This component is an “Email Address Validator.” It performs three functions: 1) provides multiple levels of email validation; 2) validates that the address is well-formed; and 3) validates that the email box is valid on the target server.
http://software.topcoder.com/catalog/c_component.jsp?comp=2804393&ver=2

5-page “Email Address Validator 1.0” component specification
http://software.topcoder.com/catalog/document?id=10183850

2-page requirements specification for the same component
http://software.topcoder.com/catalog/document?id=8576053

9-page sample developer profile (136 kilobyte PDF file). This profile shows the kind of information that TopCoder develops about its developer members, which goes way beyond standard resume fodder.
http://www.sourcingmag.com/docs/free/TopCoder%20developer%20profile%20sample.pdf

About the Author: