Coding tests are a fact of life when you interview for a developer job. They’ve been written about plenty, and the conventional wisdom is they’re very useful. For example, the venerable “Joel on Software” wrote about them in his “Guerrilla Guide to Interviewing,” and Jeff Atwood wrote about them in “Getting the Interview Phone Screen Right.”
Well, coding tests make sense for very junior job openings, where you’re looking for just a “coder”. Or, if the candidate has zero prior work experience. Otherwise, they’re worthless.
Here’s the typical coding test
The candidate…
- Visits the company for a day of interviewing, with each interview being 45–60 minutes. At least one of the interviews has a coding test.
- Is asked to code a simple problem. (E.g., “reverse a string”)
- Is asked to code on a whiteboard, without any assistance.
- Does the test while the interviewer remains in the room, watching.
- Has their answer reviewed/judged by the interviewer. Sometimes others are brought in to review the answer.
All firefighters must be able to roast a marshmallow
Does any of this remotely resemble how you develop code in your company? I’ll bet you don’t code standing up at a whiteboard with someone watching the clock, without any tools. So why do you think it’s a good predictor of anything?
Virtually everything about a coding test is wrong.
The social environment is all wrong
A normal job candidate is already stressed out. Coding tests ratchet up the stress by at least an order of magnitude.
The interview slot is, say, 45 minutes long, and he knows the clock is ticking. Most interviewers say something reassuring, like, “It’s not important to get the right answer. I’m more interested in how you approach the problem.” No candidate believes this. Nor should they, because a candidate with the “right” answer will be perceived as better than one without, all other things being equal.
The interviewer usually remains in the room, watching him as he draws chicken-scratch on the board, chases dead ends, etc. I know I do my best work when someone’s watching me, and waiting for me to finish. How about you?
It’s true that a standard interview’s social environment isn’t identical to a normal work environment. But the intersection between this and a normal work environment is ø.
The development environment is all wrong
When I design or code something, I’m at my computer.
With Python, it’s easy to dive into the interpreter and experiment. But no matter what your coding language is, I’m sure you also sometimes use small code experiments as part of your development process.
I can search technical sites, news groups, or forums if I need to. I can even e-mail or IM someone with a question. When I write code, I use a language-sensitive editor, which highlights syntax errors, displays library documentation, etc.
I have a copy of Python Essential Reference at my side. Maybe I also have CSS Pocket Reference or Python Cookbook. No matter what your language is, I’m sure you also rely on some texts when you code.
None of this exists in a coding test. The intersection between a coding test and a normal programming environment is ø.
You’re testing for the wrong thing
The tests must be simple because time is limited. Hence, the tests are “reverse a string,” “find the largest value in an array,” etc. This is simplicity to the point of triviality, and it’s not testing what you think it is.
The candidate is always obsessed with demonstrating that he knows the language’s syntax. (Most interviewers reassure candidates on this point, and no candidate believes them.) This means a large amount of candidate energy is going into syntax checking, instead of solution crafting.
There are aptitudes and skills that coding tests don’t even touch. You won’t see how the candidate documents their code. You won’t see how the candidate approaches unit tests. You won’t see how well the candidate follows check-in rules. Etc.
It can be insulting
There’s nothing like having a great work history, impeccable references, and a resume with tons of relevant experience — and then being asked to write a function that capitalizes the first letter of every word in a string.
Remember, I’m writing here about someone other than a totally green-behind-the-ears candidate.
It can be sadistic
See the “social environment” section.
A good coding test would take too long
In a good coding test, you’d give the candidate a table, a computer with Internet access, some reference texts, a pad of paper, and three or four hours to work on a problem. And then you’d spend at least 15 minutes to review their solution, given the likely complexity of a programming problem that needed three or four hours to solve.
This would allow for a far more realistic appraisal of the candidate’s skills. It wouldn’t be perfect, but it would have a few things in common with the real world, of which the standard coding tests do not.
The problem is that nobody has the time for this sort of test.
Back to the future
Most (nearly all?) other disciplines don’t do their equivalent of a coding test. If you were interviewing…
- A CFO, you wouldn’t ask her to do a math problem on the whiteboard
- A firefighter, you wouldn’t set the wastebasket on fire and ask her to pour water on it
- A hardware engineer, you wouldn’t ask her to solder a chip to a board
- An artist, you wouldn’t ask her to draw a picture on the whiteboard
They instead rely upon the applicant’s references, work history, and answers to questions.
You’re probably trying to discover this with a coding test:
- How the candidate thinks approaches problem-solving
- The accuracy of the candidate’s work history
- The accuracy of the candidate’s representations of language familiarity
- The candidate’s use of good development practices
In which case…
You can uncover all this by asking good interview questions, and doing good reference checks. Given how most companies do both of these, it’s really (unfortunately) a radical suggestion.
Good interview questions should be easy, but many interviewers ask terrible questions. I once interviewed for a VP Engineering position, and was asked if I knew Moore’s Law. I nearly fell off my chair.
Most companies don’t do good reference checks, and many don’t do them at all. A reference gives you a window into the applicant’s performance in a real-world situations, which is orders of magnitude better than a coding test. And you can do a reference check using your own contacts, as long as they’re not at the candidate’s current employer. I’ll take these over a coding test any day of the week.
Updated on 2013-06-01: I’ve never been happy with this ending. It’s lame. I was trying to argue for something, but it didn’t come out right and I didn’t write what I wanted to say. I’ll try again, someday.
47.631545 -122.364657