In defence of the technical interview

The /r/programming subreddit has about 2 million members. Most posts get only a few comments, busy threads get between 100 and 200 comments. But search for “interview” and you’ll see posts with far more.

Many developers have strong feelings about coding interviews. All the articles above lament the trend towards live coding tests as a requirement to get a software job — not a single one speaks in defence of this practice.

In today’s essay I want to do the following:

1. Make the case for code tests.
2. Name some common practices which are best avoided.
3. Explain why interviews always seem to involve pointless algorithmic puzzles instead of ‘real’ programming.

I feel I can write on this topic because I’ve interviewed developers over 300 times in my career, and designed the interview process used at my current employer. The team I hired performs developer interviews on a daily basis. A good hiring process is valuable so this isn’t a full blown tutorial on how to design one; it just mentions some things that candidates legitimately and quite reasonably hate, and which smart employers will avoid.

Firstly, how do we hire at R3? We’re recruiting for technical roles around the world as part of building an open source Bitcoin-inspired decentralised database. The interview process for developers consists of, firstly, a piece of code that you’re asked to send back a quick code review of (this is meant to take about 5 minutes), then a 30–60 minute long video chat+screenshare in which you may join from home and some coding is done in an editor and language of your choice*, then finally an invitation on site to meet the team and talk to senior management. The code test sometimes also includes design and ‘talky’ questions, depending on the background of the candidate and precise job role: it’s not just coding. We’ve found this process to be pretty accurate whilst still being quite lightweight (compared to some hiring processes at least!).

[*language of your choice, as long as it’s mainstream enough the interviewer can read it. We are willing to train developers from alternative backgrounds in the language and platform we use — Kotlin on the JVM]

NB: None of the questions listed in this article are actual questions we ask, just stereotypes of the genre

With that out of the way, let’s start with styles of interview best avoided.

What interviewers should avoid

Some firms do the following things, which I don’t recommend:

• “Robot interviews” with automated assessment. Nobody enjoys being an interviewer and so there’ll often be demand from your team to see if it could be outsourced. But when we very briefly experimented with this the results didn’t line up with human-driven assessments and, more importantly, asking a candidate to interview is a polite request for their valuable time. I find it disrespectful to ask someone to make a commitment of time for free to a company without making an equal commitment of time back. An interviewer spending an hour on the phone with you is an indication that they value your time as much as their own. A flaky automated ‘homework’ web app imposes very disparate costs. We predicted (correctly) that senior engineers in particular would refuse to do this sort of interview.
• Whiteboard interviews. Whilst understandable many years ago, nowadays laptops and screen-sharing are ubiquitous. The natural way to write programs is in an editor, on a keyboard, with a search engine and compiler to help you. Video call interviews allow a candidate to program with their own computer, with their own tools, at home. It’s vastly preferable to asking someone to write code with a fat marker pen whilst standing up in a tiny conference room.
• All day interviews. Many firms expect an interview for a developer to take an entire day, typically with between 5 and 8 separate interviews. This makes it hard for developers who already have jobs to attend (i.e. the kind of developers you probably want to hire). Doing a remote interview followed by something that fits in a lunch break helps avoid candidates needing to take time off work, which means they’re more likely to go through with the process. I’ve personally found that eight feedbacks doesn’t seem to increase team quality much over just two.
• Large scale “homework”. Whilst we have started asking candidates to do a simple pre-interview task lately, the task doesn’t involve writing any code and can be done in a few minutes. I’ve heard of firms that ask candidates to actually write full blown, real world programs outside of interview time. Again, this isn’t respectful of people’s time and many will rightly refuse to do it.
• Making questions up on the fly. Question design is hard. You expect the candidates to be prepared, don’t you? So it makes sense that you’d be well prepared too.
• Fix a real bug or implement a real feature. Beyond the obvious copyright issues if you don’t get an assignment from the candidate, there’s no good way to make this process repeatable, no way to ensure the question always fits the available time, and no way to ensure you see a good range of programming skills (e.g. the feature may simply involve copy/pasting some existing code with minor tweaks). It also assumes you’re hiring Java developers for a Java codebase or Ruby developers for a Ruby codebase: if you’re just hiring skilled devs and letting them learn your local tools on-the-job this style of interview won’t work.

I’ve gone over some things to avoid here, but the basic premise that we should ask programmers to do live coding in an interview remains. Let’s now ponder why this is such a common requirement despite its unpopularity.

Many candidates don’t like being code tested. False negatives, and the feeling that the process is badly designed thus creating lots of them, leads to anger and resentment. Nobody likes the idea they missed out on a job because a braindead interview process prevented them showing what they can do.

Getting rejected for a job you know you could do is an awful experience, especially if the process appears implicitly to be making claims of scientific accuracy.

It might therefore appear that employers could gain an advantage by not doing them and relying on older forms of interviews instead, like having a conversation about past experience. Yet code testing has spread from companies like Microsoft over the years to become standard nearly everywhere, and there are now few software firms that are comfortable with just a friendly chat over the phone before making an offer. Why is that?

Code testing is necessary

Watching a trainee come out of their first shadow coding interview is always a treat. First there’s a glassy eyed, slightly shell shocked expression. Then the ranting begins.

How could the candidate have started coding in one language, then decided actually they don’t know that language and should start over from scratch in a different one? How could someone with 10 years experience on their CV be unable to start a new project in their own editor? How could the candidate spend thirty minutes trying to generate a random number and still fail? This is crazy!

The old hands chuckle, thoroughly enjoying this time-honoured rite of initiation. “Ah, that’s not crazy. Let me tell you about the time …”

Structured interviewing in the tech industry was popularised by Microsoft in the 1990s, a process that became famous for tricky brain-teasers like the famous “Why are manhole covers round?” along with programming puzzles like, “invert a linked list in C on a whiteboard”. The essay arguing for this form of interviewing was Joel Spolsky’s classic “Guerilla Guide to Interviewing”, based on Redmond’s process. When it was published in the year 2000 it was quickly adopted wholesale by startups, like a small two year old company called Google.

But I think the technique was made most famous by Imran Gohry’s FizzBuzz question, brought to public attention by Jeff Atwood’s blog in 2007 “Why can’t programmers program?”. Jeff started his blog by saying:

Everybody has to start somewhere. But I am disturbed and appalled that any so-called programmer would apply for a job without being able to write the simplest of programs. That’s a slap in the face to anyone who writes software for a living.

Dan Kegel is an esteemed former colleague of mine at both Google and on the Wine project. In 2006 he said this in “How to get hired”:

A surprisingly large fraction of applicants, even those with masters’ degrees and PhDs in computer science, fail during interviews when asked to carry out basic programming tasks. For example, I’ve personally interviewed graduates who can’t answer “Write a loop that counts from 1 to 10” or “What’s the number after F in hexadecimal?”

It’s tempting to write off such stories as exaggeration, or chest-thumping by developers looking to prove their own superiority in the interview room. I’m here to tell you it’s not like that. Everyone who builds a team of developers, and I do mean everyone, rapidly gets used to people turning up to interview who cannot actually program computers, even under the most generous definitions of the term.

There’s a widespread perception that companies do code testing because they’re stuffed with algorithm obsessed nerds who are all trying to locate the most academic candidates possible, in the hope of being the next Google. This isn’t the case. Companies do code testing because they have encountered so many candidates who look good on paper, and may even be able to talk about computers convincingly, but can’t actually write a program when asked. Any program. At all.

If you’re a job candidate, being able to tick these boxes already puts you ahead of the pack:

• If you claim to know a language, be willing to actually use it in the interview. I’d guess a good 50% of candidates who list C++ as a skill on their CV point-blank refuse to write a program in it when asked (but are willing to use an easier language).
• Be comfortable with your tools. If you’re a Java developer, that means knowing how to create a new project in your chosen IDE, write some code, run it, and then locate any outputs the program has created. You should also know how to write and run tests and use the debugger.
• If you’re asked to bring your own tools to the interview, actually do so. Don’t try to install things during the interview itself.
• Be comfortable with the basics: Collections. IO. String manipulation. Loops. Data types. If you’re a functional programmer understand how to actually mutate lists in place (I’ve seen a failure rate of 100% for Scala programmers on a question that had a naturally imperative solution).
• Know the language you’re going to try and use. Don’t be flummoxed by ordinary errors about syntax or type-safety.

If these things sound basic, welcome to the world of interviewing developers.

Code testing is important for another reason. Experienced and skilled engineers — you know, the type you want to hire — are aware of this problem and don’t want to work in a team that lets just anybody in. A robust hiring process is not merely a chance for the candidate to impress the interviewer, it’s a chance for the interviewer to impress the candidate. Some of the best people I’ve worked with were attracted to their place of work because of the difficult interviews. Striking the right balance between efficiency, false positives and false negatives is a part of the art of building a software company.

Still, most horror stories about interviewing from the candidate’s perspective aren’t from people who fell at the first hurdle. Most people who get annoyed at code tests are annoyed because they feel they’re being asked to solve strange problems that are fiddly and unrepresentative.

Why are questions always algorithmic?

Write your own sort function. Invert a binary tree. Do a graph search.

Job interviews can feel like being sent straight back to university. All the years of experience you have writing real programs are suddenly tossed out of the window and it’s like you’re sitting in a lecture hall once more, being asked to prove your mastery of theoretical topics that are — in practice — always solved for you already by the rare sort of developer who actually enjoys reading Knuth.

When reading discussions the most common explanation I’ve encountered for this situation is that employers are idiots. The second most common explanation is that employers all want to be Google, and whilst Google evidently needs computer science experts, nobody else does. But they cargo-cult Google’s hiring process anyway in the hope that’ll make them billionaires.

The real explanation is much more prosaic. When you sit down to write an interview question for developers there are many constraints you have to satisfy to create a good question. The sort of question that satisfies all these constraints tends to look “algorithm-y”.

This is OK. The goal of an interview is to extract information about the candidate as quickly as possible. There’s no requirement that a good interview question be representative of daily work. To see this more clearly, consider interviewing a pilot. After establishing basic bona fides, it would be reasonable to ask the candidate about what to do in various emergency situations. Emergency situations aren’t representative of the daily work of flying, but safety is important so nobody would accuse such an interviewer of asking irrelevant questions. Yet this accusation crops up in the software world all the time.

Given the goal of extracting information about the candidate’s skills as quickly as possible, what are our constraints?

Firstly, the finished program needs to be very short. Unless you want to spend your entire interview on one question, a competent developer should ideally finish within 30 minutes or so. You can’t write much code in 30 minutes. Already this constraint knocks out most kinds of “real” program that might deliver practical value.

Secondly, it needs to be completable in isolation, with no complicated setup or site-specific knowledge. Every second spent explaining the question is a second you are not watching the candidate demonstrate their skills. Sometimes I see online comments of the form, “company Foo does bog standard business web apps so why don’t they ask me to make a REST API instead of writing my own sort function”. But try to write down a question of the form “make a REST API” that a candidate could actually succeed or fail at, and you’ll find it’s huge. What does this API actually do? Where’s the data? Do you want me to connect to a database, or serve data from a file, or maintain it in memory? Etc. Ideally, an interview question can be explained in 30 seconds or less. Some companies do ask questions like that, and can succeed if they’re only willing to hire developers specialised in their specific framework of choice and that framework automates all the boilerplate. At R3 we are willing to hire developers from a variety of backgrounds and specialisations, so we have to stick to questions that can be reliably answered without any particular framework or technology.

Thirdly, it should actually convince you the candidate knows programming. Writing a few functions that returns some fake HTML isn’t going to cut it. At minimum you’re going to want evidence the candidate can use loops, collections, classes, do IO and is familiar with their standard library (by which I mean knowing roughly what’s in it, not that they’ve memorised every API).

Fourthly, it should give great candidates the chance to excel. The skills I’ve mentioned may seem ludicrously basic. That doesn’t mean the company has low standards. A good code test has enough depth in the problem that a great candidate can rapidly and impressively create a much better solution than a beginner can. With sufficient practice interviewers learn to distinguish between experienced developers and beginners even when deploying the same set of questions to everyone.

Finally, you want some evidence the candidate can take a problem that doesn’t have an immediately obvious solution and think their way through to an answer, rather than just freezing up. The exact nature of that problem doesn’t really matter, just that it’s not solvable by blindly pattern matching against things the candidate has seen before. This is the most ambiguous part of the process: what does it really mean to test the candidate’s thinking ability? It’s also the one that excludes most solutions of the form “use code written by someone else to do it for me”, even though that is often the right approach in real work.

Short, easy to explain, uses only basic language features, allows both poor/great solutions and doesn’t succumb to boilerplate. I argue that all of these are fundamental for hiring a competent team, but trying to satisfy them inevitably leads people to apparently unrepresentative algorithmic puzzle questions. The goal of these questions is not to see if you actually remember obscure algorithms from the depths of a computer science course you maybe never took. It’s just an excuse to get you writing a bit of code that contains most of the basics.

So don’t stress about whether your answer has ideal computational complexity, at least, not on your first attempt. The interviewer probably doesn’t care. Instead, focus on the chance to write clean bug-free code in a quick and competent manner. Then if you have time left over, offer to go back and optimise it.

Conclusion

Hiring developers is a significantly more rigorous process than seen in some professions due to the absence of any globally recognised certification bodies, and the need for concrete technical skills. The process has improved enormously over the years from the perspective of employers. For candidates who are comfortable doing these sorts of interviews the reward is more evenly balanced teams without useless fakers, which used to be a pretty common complaint (and still is, in organisations that haven’t adopted this practice).

Despite all that I must finish by observing that regardless of our best efforts in the industry, hiring is still largely random. Well designed interview processes make it slightly better than random, which is why we do them. But given the high cost of firing someone who doesn’t work out, there will always be a bias towards false negatives over false positives and thus there will probably always be horror stories of candidates who could obviously have succeeded at an organisation getting rejected by mechanical procedures. As Spolsky’s suggested alternative of replacing interviews with internships shows, for all its flaws, it’s a hard system to beat.