The problems with live coding interviews
garrettdimon.comA perspective from the other side of the desk (playing devil's advocate here):
There's a fundamental skill that a good programmer has to have, and that is to be able to take a novel problem that they haven't seen before and break it down to solve it in a sensible way.
There are plenty of programmers who fake their way through a career without having that skill. They just copy stuff and never really understand it. They're fine for some types of programming career, but if your business involves solving new and novel problems then you have to know which type of programmer you're hiring.
A contrived live coding exercise gives you a strong signal on this. It does have a decent chance of producing a false negative, but only a very small chance of a false positive, and that's the trade that has to be made with this kind of approach.
Is a better option to not do this kind of assessment, hire the person, find out that they can't do the hard bits and then fire them within 6 months? I'll take some convincing of that...
> “take a novel problem that they haven't seen before and break it down”
But who does that professionally as a stand-up performance, clock ticking, a judge breathing down your neck who has been equipped with a script that tells him things like “if candidate doesn’t do X within the first ten minutes it’s very bad”?
Doing well in that situation depends more on social performance skills than problem-solving skills. You’re essentially trying to infer the interview script that the interviewer has in mind and act it out convincingly.
If someone says “I’d like to start by taking ten minutes alone in another room and think it out first”, will they get hired? Probably not. It’s not the expected performance even if the solution is fine.
Yes. I did this type of interview for a previous job, and don't want to do it again. I'm a software developer, not a salesman.
Requiring salesmanship to enter a company may well correlate with requiring salesmanship to advance and be recognized at the company. (I don't regret that job, because the rank-and-file were great, but I don't want the same structure of overlords.)
This time around, I am fortunate to have effectively lifelong runway, so I'm now trying to handle an interview as if I were discussing a problem with a co-worker. If that filters me out, that's probably best for me, too.
Given two candidates who can break down problems, I'd prefer to hire the one who can do it with a gun to their head (metaphorically, of course). Lord knows owners, ceos, board members, managers, and angry co workers would often enough like to put a gun to my head.
I’d suggest not working for people who like to put a gun to your head, euphemism or not.
What's wrong with hiring for interviewers to be this nihlistic.
Working well under pressure is a skill too.
Why? We don’t have animals hunting us and a constant need to fend for food and water like our ancestors did. Why in 2023 should anyone be under pressure for anything to do with their job?
Ideally, there would be no pressure involved. But in reality, there are pressures associated with deliveries and conflicting goals. You don't have to look far for these, and hiring for the ideal conditions is unrealistic.
That’s why he says this approach is more likely to produce false negatives than false positives
False negatives are always better than false positives. Every company who does leetcode interviews agrees with that.
> If someone says “I’d like to start by taking ten minutes alone in another room and think it out first”, will they get hired? Probably not. It’s not the expected performance even if the solution is fine.
When interviewing I always start with a discussion about the problem. On the whiteboard. I'm explicit with the candidate that this is totally expected.
I'm actually measuring something perhaps more important than the code that will get written later: can this person formulate a plan and have a technical discussion.
But what you don’t know is whether people who didn’t give a stellar performance would have gone on to be adequate or even excellent hires.
As for the time limit, it's relative. All things equal, if you can complete the problem in 10 minutes and the company can hire somebody that can complete it in 9, then objectively the other candidate is the stronger one.
As for nervousness for being forced to partaking in a stand-up performance, I'd argue that "social performance skills" can work against you, since the more "antisocial" you are, the more you can ignore (or are oblivious to) what other people think about you and you can focus on your task. It's only people who actually have the minimum requisite "social skills" that would be conscious of other people intensely watching them "perform".
But even if you're right -- interviewing is inherently a "social performance" activity. If you're not coding you're doing some other stand up performance to present yourself anyway.
> All things equal, if you can complete the problem in 10 minutes and the company can hire somebody that can complete it in 9, then objectively the other candidate is the stronger one.
They are stronger in the specific problem space you’re testing for. If your business has a lot of novel coding problems that need to be solved in under 9 minutes, then sure, this is a great measurement.
On the other hand, if your business has a lot of hard problems that take days, weeks, or quarters to solve, measuring someone’s ability to solve a 9 minute problem is a terrible metric. If anything it’s a counter signal, since it tends to select for candidates that optimize short term thinking over long term planning and problem solving.
> if your business has a lot of hard problems that take days, weeks, or quarters to solve
I suppose your ideal interview as an interviewer would be to give the candidate a take home task and ask them to spend 2 weeks to work on it?
> If anything it’s a counter signal
Or perhaps give the candidate a task that normally takes 30 minutes, and hire the candidate if they take 60 mins to finish it?
I mean, you do you, but I hope you (and everyone else) can see why I'm not convinced otherwise.
No, the ideal interview is to ask them questions about what type of problems they've solved before and ask them to walk through what they did. Also, have a conversation with the person to get to know a little bit about what they're looking for in a job/company. It's an interview, not a tryout.
Metrics like a 10 minute task, or a 30 minute task is all relative. Do they know they language, the IDE, the operating system, the documentation, any experience in whatever abstract problem topic you choose, and personal comfort levels will all come into play.
If you want to filter someone to do a very specific thing then post to hire a contractor with the specifics of what you need. If you want a developer that can grow within your organization then pop quizzes area good way to dismiss good candidates.
> No, the ideal interview is to ask them questions about what type of problems they've solved before and ask them to walk through what they did.
This just doesn't work. Lots of people can bullshit very convincingly, and even if they can't produce novel solutions, can very well explain them in a matter that makes you think they for-sure know how to do it.
Hell, I can probably still prove lots of interesting things about set theory in math, as can many other people who studied math - doesn't make me the equivalent of the world-class mathematicians who actually came up with the proof.
Empirically, "talk to someone about what they did" doesn't give me hires that actually know what they're doing.
If they pay me at contract rates double my current salary, and in my particular case my current IP assignment clause/the problem/ law makes it ok. Sure, pay me money to do something like what you guys need done.
Probably most useful if it's a contribution to open source.
I don’t get why I have to prove to some strangers I can write code when I’ve been doing it for the last decade. It’s a ludicrous and broken system.
Because those strangers don't know you.
Any idiot can write things on a resume and say they did things they didn't do (aka lying). You would never do such a thing, of course, but as crazy as it sounds, there are people out there who would do just that! So because there's no professional license to write code, the only way to prove to these strangers that you actually can write code is some sort of exercise where you prove it to them.
I really don't get why this is so hard to understand, either. I get that live coding in front of someone else is a crazy stressful situation - I've failed multiple interviews because I couldn't perform on demand and answer the interview question in the interview setting, when I could easily have done so after taking a proverbial shower to have a think, so I'd love to get rid of them too. But unless we all band together and start a software developers guild or something, the live coding interview is here to stay. (Though, Triplebyte, now Karat, and others did take a run at improving the process, so there's that.)
I know what I know, but you don't know what I know. It's only by communicating, in a sufficiently unfakeable way, like a 45-minute in-person interview, that one can pass or fail the unwritten "can program" shibboleth.
My issue is that these coding interviews tend to turn up a lot of false-negatives and sometimes even false-positives. And of course they do. There’s so much more to what we do than being able to implement A* in 45 minutes without a real IDE could demonstrate.
You don’t get why you have to prove your ability to do a job when you want them to pay you to do that job?
Yep. I realize that’s not going to resonate with many or even most people, but that doesn’t mean I don’t believe it and that doesn’t mean it doesn’t have merit.
Law firms don’t ask lawyers to litigate in a mock courtroom setting before hiring them. Hospitals don’t ask surgeons to perform a little test surgery on a person before hiring them. Engineers aren’t asked to build a bridge to prove they know how.
I think the reason we are so unique is because we CAN demonstrate our ability to quickly and easily, and without risking human life. CAN of course does not equal SHOULD.
Law is a protected profession - you have to pass exams to be able to call yourself a Lawyer, and a law firm will verify that you have passed the relevant exams.
Same for medicine, engineering...
Any chump can call themselves a software engineer. That's the real problem here.
Normal interviews are mostly a scripted performance where people largely regurgitate answers to predictable questions.
So the issue isn’t that it’s a stand up performance it’s that you need to split your attention between the performance and solving some trivial problem. Becoming really good at programming requires being able to focus on an actually difficult problem not work through a simplified example while entertaining an audience.
There might be a correlation between solving trivial problems quickly and being able to figure out a heisenbug from some multithreaded monstrosity, but it’s not strong enough for minor differences in performance to be meaningful.
> Becoming really good at programming requires being able to focus on an actually difficult problem not work through a simplified example while entertaining an audience
There are multiple ways to "become really good at programming".
A company of any size needs people who can churn through well-defined problems quickly, and most live coding tests are relatively well-suited to selecting for that. They also need people who do other things well:
* tackle large, ill-defined problems, alone or in a small team * identify, triage, and (if necessary) solve problems as they emerge * refactor existing implementations that have outgrown their initial architecture * identify trends and estimate when they will lead to scaling issues in the future * communicate complex problems to people without the necessary context * break down complex solutions into discrete, well-defined steps that can be tracked to completion
They also need people who can accurately estimate how long all of the above will take... unfortunately, as we all know, such people do not exist.
If you're hiring someone to churn through well-defined problems, live coding interviews are likely a good fit out of the box. If you're hiring someone to reinforce a weakness in your organization's ability to do any of the other things above, live coding exercises are at best a loose framework that we're all familiar with to try to get at whether or not they have those skills.
I'd even argue it's a prerequisite.
If one doesn't even know how to complete well-defined tasks reasonably quickly, I can't imagine how they would be able to break down complex tasks to these smaller, well-defined tasks, and accurately estimate the expected effort needed.
This sounds almost Dilbert-esque. It seems like a lot of people here want to be a pointy-haired boss (or pointy-haired tech lead)!
> I'd even argue it's a prerequisite.
You can know how to do something without being able to consistently do it.
Maintaining velocity is something that I've never been very good at. I always struggled as a junior and mid-level developer with having periods of very high productivity followed by periods of very low. I've been fired for it.
On the other hand, if you measure my output on a monthly or quarter time horizon, I'm very consistently at the top of the class.
My strong suits are around helping others get through tough problems, quickly switching between levels of abstraction (including within a single conversation), understanding how a part of a process fits into the whole, and in guiding/limiting architectural discussions by managing the scope of the problem space under consideration.
This fits very well with my current "staff level" role, but was a real struggle getting to this point.
> This sounds almost Dilbert-esque. It seems like a lot of people here want to be a pointy-haired boss (or pointy-haired tech lead)!
I can see why you'd say that, but I think there's a big difference: PHB was incompetent in addition to having a different role from his team. I've found a sweet spot for myself where I can fill the parts of the "PHB role" well that actually do help the team, but I'm also more than capable of switching contexts and knocking out some tickets when there's a crunch. When I'm especially "in the flow" I work on that sort of stuff and do well. When I'm not, I run interference and keep my team from having to deal with the organizational stuff that doesn't directly help them get their jobs done.
The best functioning teams I've experienced all have one thing in common: they set people up for success by allowing them to do play to their strengths. As a team grows, you can find gaps in those strengths. I see hiring as an opportunity to find someone who is strong in the areas the team is weak. Let the new person come in and do the things they like to do and are good at!
Coming up with your own tasks means understanding the problem implicitly. Solving other peoples trivial problems means searching for the interviewer’s hidden gotchas and unspoken requirements.
Ie if someone asks you to find the median value of an array the number of elements and how to handle nulls isn’t obvious, but when you want the median value in an array the context is clear.
> Normal interviews are mostly a scripted performance where people largely regurgitate answers to predictable questions.
Ah, so you prefer being asked "where do you see yourself in 5 years?", and regurgitating the text generated by ChatGPT instead?
Alternatively, as I replied in the sibling thread, do you prefer to be handed a difficult take home task that is expected to take 2 weeks to complete?
I think I can live with the first, but I'd be insane to work 80 hours on a task just for a small chance to get a job.
“Where do you see yourself in 5 years?” is asking a perfectly reasonable question about what if anything you’re working towards. If you think regurgitating something from ChatGPT is appropriate, I can see why you might prefer more limited interviews.
> then objectively the other candidate is the stronger one
...if the job is to complete abstract problems in live coding interviews, sure.
By your logic, there's no point in doing any interviews at all, because it is nobody's job exactly to sit as a candidate in an interview session.
Is that your point? Because I'm getting tired repeating the point about take home tasks in the sibling comments by now.
What are the "all things equal" here, though? I would tend to disagree that someone who can complete the coding challenge faster is the objectively better candidate. Especially if we are talking about leet coding, where there are people who study their asses off and memorize the common leet code problems. Or maybe the coding challenge isn't leet code, but just so happens to be something that the candidate has done before, or has done recently. Just because one candidate lucked out and has done this very specific thing, doesn't mean they are better than a candidate who could figure it out in half a day, if given a little more time to do so.
You people are so bent on arguing the unarguable that the arguments are so unnecessarily argumentative.
The interviewer and the hiring company does not have a God's eye view of the candidate and must work within the limitations of the hiring process. The best one can do (without resorting to metaphysical processes) is to make a decision that has the highest expected value (or highest chance of hiring a good candidate).
The fact that there could be a candidate that lucked out because they were prepared for the specific set of questions you asked is not specific to live coding problems. You could have asked them anything and they could have lucked out and just happened to be prepared for the answer you were expecting.
That said, I can tell you for certain that if given a task that the average candidate takes 10 minutes, and a candidate completes it in half a day, you're looking at a 0.1x developer right there. That's by definition.
They’re only objectively the strong candidate if that’s the sole thing the company values. It’s unusual for that to be the case.
With the "all things equal" qualifier, it's sufficient that completing the task faster is a positive signal rather than a negative one.
I mean, this is very elementary.
a single example of a one minute difference in solving a problem that takes ~10 minutes is noise
Yes, so given that all things are equal, are you going to hire the slower person then?
Or are you going to, as I mentioned in sibling comments, give them a take home task that takes two weeks to complete, just to be sure?
I’m also most often on the hiring side of this. If novel problem solving is what you’re actually testing, I would agree. But too many interviewers just think they’re testing that, when what they’re actually testing is whether a candidate has seen a certain memorizable trick before.
I’m talking about the “detect a cycle in a linked list” kind of question. If you ever actually need to do that in practice (though I would question the choices that lead up to that), it’s easy enough to google. The hard part in practice is figuring out that your messy practical problem decomposes into an algorithmic question, but that skill is rarely tested in interviews.
Detecting a cycle in a directed graph is useful (in certain domains I've worked in), but doesn't have a cute trick answer.
[Edit: I in mind ‘Detecting the cycles’ rather than ‘Detecting a cycle’ but wrote the wrong thing. Mea culpa.]
Your actual job will be more like implementing a maximally performant directed graph in safe Rust. (Not really. It won't be that interesting.)
> Detecting a cycle in a directed graph is useful (in certain domains I've worked in), but doesn't have a cute trick answer.
Doesn't the standard cute trick answer for a linked list (tortoise and hare) work fine? Do two parallel breadth-first traversals of the graph starting at the root node, one at twice the speed of the other. (If there's multiple root nodes, introduce a unique super-root that points to the roots.) If there's at least one cyclic path, both iterations will enter the same cyclic path because iteration order is the same, at which point it reduces to the linked list problem.
Yes, I phrased my comment incorrectly; I had in mind identifying (all) the cycles, and that for a linked list they're the same thing. For a DAG it doesn't generalize because you can't find more cycles without storage to exclude the earlier ones.
Fair enough. I imagine a simple (but O(cycles) space, not O(1)) way to do this is to still tortoise-and-hare, and on detecting a cycle, break the link behind you (by recording it in a table) and continue the BFS. If O(cycles) = O(n) this isn't terribly interesting compared to other approaches, but if O(cycles) < O(n) it might be helpful.
Actually, now that I spend more than ten seconds thinking about this, I think this requires O(1) space; you only need to remember the single most recent link behind you to avoid following it again. Remembering a single break is sufficient to get you to either the next cycle or (if no more) terminate the BFS; and once you detect the next cycle you can safely remember only /its/ break since BFS will never take you back to the first cycle. Assuming streaming output (e.g. print identification of each cycle as you detect it -- including e.g. printing each node until you see the break point, which will identify each node that is part of the cycle), this is O(n) time and O(1) space, which I believe is optimal.
if you can mutate the graph and mark a node as traversed that's the easiest way. If you can't then save the addresses to a hash set. Both are extra memory but you don't have to deal with "parallel BFS," which honestly is just over complicated imo.
With "parallel bfs" comparing "layers" of traversal has runtime cost that effects the big oh so I think the above solution is better overall even though it feels cheap.
But the whole point of the "cute solution" is to solve with O(1) memory; if you're willing to allow O(n) then the linked list also admits more sane solutions.
I'm not sure what you mean by comparing "layers" of traversal. Tortoise and hare is about provide a termination condition in the case that a cycle exists. In the case that there's no cycle, every algorithm must inspect each node, so is O(n). In both the linked list and the DAG case, if there is at least one cycle, (a) both pointers will enter it in at most O(n) time, and they will match in at most O(n) additional time, so this stays big-oh optimal.
I'd argue the point of the exercise isn't to implement an O(1) solution, but to verify that the applicant understands how the logic they're writing actually executes, and is capable of minimizing complexity.
Whether they arrive at an O(1) solution doesn't matter a bit to me. Most of the time, I don't even care if they phrase it in big-O notation - or have even heard the term!
Given the choice between someone who completed the exercise with an O(1) solution in half the allotted time and someone who never wrote a single line of valid code, I'll choose the person who is able to articulate the problem and potential solutions best every time.
It's far easier to search the web and find a "cute solution" than it is to learn how to think about and communicate complexity.
> But the whole point of the "cute solution" is to solve with O(1) memory; if you're willing to allow O(n) then the linked list also admits more sane solutions.
Yeah but the "parallel BFS" solution is actually slower. Overall people sacrifice memory for speed in these algorithm problems.
Additionally the input is already O(N) so if you count the input as memory the addition of O(2N) doesn't shift it. If you don't count it well you're deliberately removing real world actuality and making the problem harder.
>I'm not sure what you mean by comparing "layers" of traversal. Tortoise and hare is about provide a termination condition in the case that a cycle exists. In the case that there's no cycle, every algorithm must inspect each node, so is O(n). In both the linked list and the DAG case, if there is at least one cycle, (a) both pointers will enter it in at most O(n) time, and they will match in at most O(n) additional time, so this stays big-oh optimal.
It's layer based traversal via BFS. You have to compare the layers produced by the tortoise and the hare. Let's see for a binary tree:
BFS traversal happens in layers of breadth. The height of a full binary tree is log2(N + 1) - 1. The length of the last layer is 2^height. So the amount of nodes in the last layer is (N+1)/2For traversal: hare: O(N) tortoise: O(N) There are N total nodes so traversal via bfs or dfs will always be N.Because at each stage of traversal from the turtle and the hare we have to compare the "layer" of traversal between the hare and the tortoise to see if they overlap there is a roughly O(((N+1)/2 )^ 2) comparison as we cycle through each node in the layer to see if it exists in the other layer being compared. This reduces to O(N^2) processing time.
So total is O(N^3) because you have O(N) traversal time, and O(N^2) comparison time on each traversal step. You can reduce the layer comparison to O(N) if you save one layer to a hashset and then compare that set to the other layer, but that has a memory cost and only reduces the total big O to O(N^2).
There's an error in my calculations. Apologies.
The comparison happens at every level and NOT every node so, it's O(N) for traversal then when a layer is completed a comparison occurs at a cost of O(N^2). The amount of times this happens is equal to the height of the tree which can be simplified to logN. So total big O is O(N) + O(N^2 * log2(N)).
This simplifies to O(N^2 * log2(N)) which, while better than O(N^3), is still really bad.
I forgot to mention, BFS requires a queue. So if the graph is size N and with one root node and all the rest of the nodes as children that queue will go up to size N. So your solution doesn't actually get away from the memory issue at all.
A fair point; the only reason the questions are asked is because they have known answers. And unless the questions are trivial, their answers are essentially little tricks. Some candidates have already seen the answer, others have not.
But, testing if the candidate has seen a problem before may be a proxy for something meaningful.
The most fun interview question I've answered is "list as many unique ways as you can to detect a cycle in a linked list"
I'm also on the hiring side of this. I follow the process the company gives me.
Honestly speaking, 70% of the candidates can do the job, but I fail 90% because they can't pass the coding question. It's a filter with a serious problem. Yeah it's all we got, but that filter rate justifies a big discussion because it's crazy.
> that is to be able to take a novel problem that they haven't seen before
But with platforms like leetcode and more, the candidates are split into those who have had the time to prepare (and see the problem before) and candidates who can’t put in that time. And most modern interview processes do not give you credit for thinking through it. You only get through if you solve it the right way, and fast. There is no time to “think and solve a novel problem you’ve never seen before”.
> And most modern interview processes do not give you credit for thinking through it. You only get through if you solve it the right way, and fast.
Interview processes that do it that way are fundamentally broken. I wouldn't want to work at such a company. Interviews are two-way streets, and that would be a case of the company failing the interview.
The real benefit of this sort of interview practice is not learning whether or not the candidate can arrive at the correct answer. If we're at the point where this level of interview is even happening at all, you have (or should have) reasonable confidence that the candidate is capable of solving the problem. Whether or not that they do so in the interview is irrelevant. The real benefit is in being able to see how the candidate thinks through the problem.
While I think these whiteboard exercises are not the best approach, when I've been the interviewer at companies that required it, I chose a very difficult problem, gave a time limit, and told the candidate that I do not expect them to actually arrive at the solution within the given time. I just want to see how they approach the problem.
I used to do this in interviews -- give people problems not because I cared whether they got "the" answer, but because it gives you a chance to see their brain work.
The problem with this is that many, many interviewers do this badly. They think they are looking for someone who gets the right answer. Or they are not good at assessing how other people break down and analyze problems.
It is artificial. All interviews are artificial. This is why everyone would prefer to hire candidates they already know a lot about. But a good interviewer isn't looking for a perfect performance.
I think there's an analogy here to GRE scores and other standardized tests. With GRE scores you can predict someone's GPA, but you can't necessarily predict their success at doing original research. That's because success as a researcher depends on personality traits that are hard to measure like creativity, independence, self-discipline, perseverance, and passion about a subject.
Programming puzzles are the equivalent of the GRE for programming jobs. It might tell you something about the candidate, but I'm skeptical that it measures the most important traits and skills that really good programmers have. It's an empirical question, but your skill at solving programming puzzles might not predict your ability to systematically break down large problems, write readable, robust, well-documented code, or design and debug complex systems.
I also have my own theory about this which might just be my own personal bias. I feel like standardized tests encourage conformity in thinking. By relying on standardized tests, you are selecting for individuals who are primarily motivated by external measures of success and approval. On the flip side, you are selecting against individuals who are intrinsically motivated to learn and build things, but who may not care about those external factors. My theory is that organizations with too much of the former and too little of the latter are going to have issues with groupthink and will be less likely to be innovative.
"I feel like standardized tests encourage conformity in thinking. By relying on standardized tests, you are selecting for individuals who are primarily motivated by external measures of success and approval. "
Totally agree
> be able to take a novel problem that they haven't seen before and break it down to solve it in a sensible way. ... A contrived live coding exercise gives you a strong signal on this
The author suggests take-home coding exercises with "maybe ... a live call to discuss their solution" as giving a stronger signal than a live coding exercise.
The author also suggested to "present them with a complex technical problem that requires architecture-level decisions across various parts of a technology stack, and have them talk through how they would approach it" as a better signal than a live coding interviews.
So even if what you say is true, you aren't presenting a devil's advocate position. Instead, it makes it seem like you think the alternative to live coding is no assessment, which the author very clearly argues is not true.
> but if your business involves solving new and novel problems then you have to know which type of programmer you're hiring.
Its more that businesses don’t know what type of programming they do and are just hazing people for no reason
More like one person is just hazing people due to a strongly held unobjective opinion like yours
Don’t know the better option, just the limitations of this one
I think we're seeing the consequences of not having a properly regulated field that you might see another more strongly regulated engineering disciplines such as electrical and mechanical.
Since there is no CS equivalent of a professional engineering exam, employers have no guarantees that applicants will meet minimal qualifications, and so we continue to see an arms race of increasingly complicated whiteboard / leetcode interview processes.
Is there any other means of evaluating for this skill?
Recently I took a leetcode screening, my first ever, and failed to complete one single question in the allotted time due to a minor panic attack and a sudden loss of focus. The whole time I couldn't help but think about the state of the economy and how "this is it, this is my only chance." Half the time I was fighting with the in-browser IDE and struggling to remember names of JavaScript builtins that usually come easy.
Immediately after the time was up I magically started to realize how much I'd over complicated my one almost finished answer and quickly came up with much more efficient answer. Suddenly it started clicking how simple the first two problems were and how easy it would have been to crank them out if I wasn't panicking about the tech bubble collapse.
The recruiter later asked me how it went and I grumbled something about how it was a leetcode test. He said "oh well they need to make sure you have the skills for the job." At that point I was over the whole thing and honestly pretty fired up.
Over the next week I proved to myself that I do have the technical skills for that job, and that's honestly what counts.
¯ \ _ ( ツ ) _ / ¯
I always bomb the whiteboard coding portion. I've been doing this for over 20 years. I've been a senior engineer since before the iPhone existed.
Fortunately, I interview well and can discuss many, many technical concepts deeply. But put me in front of a whiteboard and I couldn't tell you my mom's name.
I imagined you standing in front of a white board, flustered, and then writing "mom" on the board. Made me laugh.
No matter what the interview format is you can just have an off day.
I have been in the software industry for 30 years as of this month. I have never had a gap of longer then 6 months and the shortest time with a company was 4 years 8 months and the longest time was 16 years. I have worked on the following technoclogies. Companies range from on of the Big 3 consulting firms to startups. Here are the technologies I have worked on
- C/C++ on Win16/Win32
- Assembly language development with Z80/8051/ARM on embedded microcontrollers
- Java (core java, Servlets, J2EE)
- Ruby on Rails
- NodeJS / Javascript
- Worked with AWS tech (the full stack)
- Relational DB (MsSQL, PostGres, MySql), NoSQL db (MonoDB)
- Coded for Linux/Unix, MacOS, Windows 16/32, PalmOS, iOS
I can provide reference for each of those skillsets form my past colleagues.
You know what - I probably couldn't pass half of the insane coding puzzles these interviewers throw out. Not because I can't solve them, I just don't remember enough of the syntax or library semantics of the top of my head.
At my experience can we just assume that I am a competent coder (maybe not the top 1%, but at least in the top %20) and talk about the job and how I can contribute ? I mean its almost insulting if you ask me to make a linked list/reverse a binary tree or other such nonsense looking over my shoulder me with a time limit.
> At my experience can we just assume that I am a competent coder (maybe not the top 1%, but at least in the top %20) and talk about the job and how I can contribute ? I mean its almost insulting if you ask me to make a linked list/reverse a binary tree or other such nonsense looking over my shoulder me with a time limit.
If there was a way to verify what you say reliably, then of course that would be better. But there isn't, and writing down that list is extremely easy - in fact, half of the CVs I've ever seen look very similar in terms of the length of the list, amount of technologies mentioned, etc. Even for people with far less experience.
There has to be some way to check whether someone actually knows what they're doing. For sure some of the time a strong reference is enough proof. That's why people in the industry a long time with many contacts will often go from job to job without even interviewing anywhere - they just move to places with former colleagues that already know them.
But for a new place that doesn't know you, thinking it's insulting to show what you know is... weird. Is it going to be insulting on day one when you actually have to do the work?
Makes me think of the quip along the lines “the expert has forgotten more than the beginner knows”
I don't get what's insulting about it and I'm glad I had the chance to prove it instead of people just binning my resume with experience the were obviously skeptical of.
Maybe it's not insulting, but it sure as shit is pointless.
Based on my experience interviewing people I don't agree with that either.
Yeah there's plenty of people who would get badly hurt if they fell from the level they show in the CV (or the level of their ego) to the level of skills they can actually show.
> Relational DB (MsSQL, PostGres, MySql), NoSQL db (MonoDB)
> At my experience can we just assume that I am a competent coder and talk about the job and how I can contribute ? I mean its almost insulting if you ask me to make a linked list/reverse a binary tree or other such nonsense looking over my shoulder me with a time limit.
I might inclined to agree had you not gotten all the names of the databases in your list wrong.
Ha! You are right - I still can't type :)
Try to look on a hiring problem from another side. Companies getting a lot of applications and at least some either stretch the truth on their CV or just lie, so trusting CV is not something a company can rely upon. Instead a company need an universal for all applicants way to filter out ones who likely cannot write code and then spend time only on candidates who've passed this test. Unless difficulty of this coding interview is miscalculated any experienced developer should be able to pass it, may be with a few days of coding interview practice. And if you are not ready to spend a few days preparing for interviews it's also a bad sign in eyes of an employer.
Very on point and I can truly relate to all of this. I'm so frustrated by the whole process that I've started to decline live coding interviews and any take-home exercises that take more than half a day to complete. Although I have over a decade experience in my field, have been a tech-lead at some of the largest corps, contributed to open source, mentored many Engineers, and really enjoy software, this whole experience is pushing me away from the field. I kind of don't care to prove anything to anyone anymore over a 45min interview... will probably start a small biz and move far away from "silicon valley" types of people/mindsets.
I don’t really. If it didn’t work the way it did I guess most of my employers would have crumpled up my resume and thrown it in the garbage for not having the right credentials on it but because it is a level playing field for everyone I got some exciting opportunities.
> any take-home exercises that take more than half a day to complete
jeez, i would reject if they took more than 5 minutes
I've been on both sides of live coding interviews multiple times. I generally agree with the author, but I don't think any of the issues he brings up are insurmountable.
When I'm the applicant, I make it a point to take control of the narrative by saying something like:
> If it's alright with you, I'd like to approach this as an opportunity to expose how I approach problems in general rather than how I'd solve this specific problem. I'll speak stream-of-consciousness as I go through it so you can get an idea of how I'm thinking about it. Feel free to ask questions if you'd like; I'll rely on you to decide whether it's more important to you that I complete the task or explain my reasoning. I'm happy to switch to pseudo-code or just discuss potential approaches if we run short on time.
When I'm the interviewer, I open with pretty much the same thing. My goal is to put the applicant at ease (to the extent that I can) and make it clear what I'm trying to get out of the session:
> First, let me say that it doesn't matter to me if you complete the exercise or not. At this stage of the interview process I'm confident that you're more than capable of solving the problem, so lets use this as an opportunity to get to know each other and see if the way we think about logic is compatible. I'd love it if you could point out things you'd change, but don't worry about trying to 'finish' or end up with production-ready code. It's just a means to an end.
i do pretty much the same as an interviewer. i am interested to see how the candidate communicates (down to language skills if they are not native speakers)
i had one who could not deal with that at all. he preferred to show me some of the code that he had worked on. fine, i let him do that instead. i passed on him not because he refused the coding session but because we didn't communicate well enough for me to be confident that i could work with him.
as a candidate i would skip the introduction though and just start talking like i would when pair programming, mainly because i'd be uncomfortable to ask for permission first.
I’ve never tried this exactly, but I wonder how “live coding” interviews might go if the interviewer was the one doing the typing/coding and the candidate’s role was to verbalize what the interviewer was doing (and why) - and also guide them to some degree.
It’d be more akin to pair-programming, which some of the interviews I’ve conducted have evolved into, depending on the strength of the candidates.
Would that capture enough to get a sense whether this person gets the gist of the code being written such that they could replicate the same output in a less contrived scenario?
My first job split the programming interview in two. 45 minutes where the candidate drive (typed) and 45 minutes where the interviewer drove while the candidate guided them. The first half disqualified a lot more candidates than the second. It was usually pretty clear how the candidate would do by the end of the first half. Occasionally a candidate would be too limited in their communication to do well while not typing.
That job had a lot of people working in pairs very frequently so it made tons of sense to do it that way there.
my experience with junior programmers suggested that they are not familiar with doing this. it seemed easier to give them the keyboard and let them work on the problem how they saw fit while observing and asking them questions in order to get them talking.
there is also the problem that instructions given by the candidate may not be clear and then i have to decide whether i just assume what they meant (because i already know how to solve the problem) or if i try to take them literally leading to frustration.
This is exactly how a company I used to work at interviewed. We did all the typing to remove that layer of stress and focus on their problem solving ability. It worked extremely well.
I actually was interviewed this way once. It worked pretty well.
These debates have been done to death and mostly I also think that the article is a fairly typical strawman argument. Take the worst way to do a live coding test, point out the problems and then dismiss the whole idea.
What else are we supposed to do? Take the fact that you can talk a good game as enough of a signal to invest 10s of 1000s? Assume that everyone with 20 years experience is as good as everyone else?
The problem is that there are no reliable signals. Most Developers I have interviewed have a massively inaccurate ability to judge their own ability (in both directions). I've lost count of the number of times candidates have rpomised that they can just learn whatever they don't already know and haven't been able to do it to any degree.
Qualifications are meaningful in some contexts more than others but most people in the UK don't have comp-sci qualifications.
So yes, I will use various coding exercises because depending on the level, it shouldn't phase someone to be given something quite simple and to see how they approach it (do they write tests first? Ask some good scope questions? Explain why they've done something the way they did?)
I have failed one of these tests in the past thinking I was a good Developer (I am!) but I don't blame the test or the process, I realised that my approach was haphazard and not an objective good look to an Interviewer so it was actually helpful.
These interviewees want: "Trust me bro. I'M JUST A BAD TEST TAKER. I can't show my awesome skills to you right now because I'm terrified and I'm panicking. Just give me a chance, even though I've shown you nothing."
Interviewer: "Gosh darn it, you're hired!!"
This presents as a bit of an unintentional strawman against demonstrating any kind of coding during an interview process. Live coding, if earnestly used only as a filter, does not inherently need to focus on speed or performance. It doesn't have to be a hard "galaxy-brain" problem. It doesn't have to have any tricks to it. It can literally be a simple task that one would expect any working software developer to be able to complete without too much fuss.
Whether this is actually the case comes down to the hiring manager.
If the hiring manager decides to optimize for performance in the coding interview then yes everything said here is true. They will typically hire people who can perform well and fast at simple coding test type problems above other any other desirable attributes.
If however, they simply evaluate the ability to work through (at any reasonable pace) a fairly trivial coding task, but make the hiring decision on bulk of the rest of the interview, then it shouldn't be a problem.
The problem is most hiring managers have not been selected for their ability, or even trained, in interviewing. A coding test is easy to set up (or copy from somewhere), administer and evaluate. It is often literally the least they could do.
What this post describes is simply hiring managers who lack interviewing skills. Personally, I would probably want to avoid reporting to such a person.
I still feel like interviews are entirely weird for everyone involved. It's not a real reflection of somebody's performance. Perhaps some people cannot work well with someone watching them think. I'm one of those people. I get a lot done at my job, but I need time to mull things over and sit with things in order to come to something that makes sense. Interviews only reward 1 of 2 things: people that are fast, efficient thinkers (this is good) and people that have very good social skills and can convince you of something they are not.
People who cannot perform socially or technically on the spot in a completely unnatural setup are sort of left in the dust. I'm not sure i have a solution other than throwing out technical interviews and actually trusting peoples prior work. I have work in the public (published academic papers, patents) but none of that seems to mean anything in an interview lol. Its only about can you perform right here right now for 1-4 hours.
I've said it on other threads. (1) trust peoples past employment, use background checks or something to make sure they actually worked where they said they have worked (2) look at their body of public work if they have any (3) just hire the people that look good off those two metrics and youll probably end up with a good employe 90% of the time and save countless hours and dollars. I'm almost convinced random choice + team vetting of resumes + a little background check would be just as effective as endless technical interviews.
> It can literally be a simple task that one would expect any working software developer to be able to complete without too much fuss.
Unfortunately this varies dramatically from person to person during a high pressure interview.
I interviewed a very senior developer once that actually used to be my current manager's boss. The coding question was pretty simple and most people didn't struggle too much with it. He BOMBED it. Like, you could have taken someone with 30 minutes of coding experience and they would have gotten as far as he did.
It was also very obvious he was just incredibly nervous so I put him through to the next round anyway. We hired him and he was great.
> It can literally be a simple task that one would expect any working software developer to be able to complete without too much fuss.
I agree, although this is why I feel a small project and participation in code review is a more realistic and useful gauge for real world ability.
Can the person understand the code base well enough to propose a sensible change, and how collaborative are they in the code review process? Do they document/comment their code, add tests, do they answer questions and take criticisms patiently, is the solution simple, does it work?
You'll learn about zero of these things watching them write a fizzbuzz in a shared editor window.
> unintentional strawman against demonstrating any kind of coding during an interview process
I don't think that's a fair statement. One of the alternatives, take-home coding exercises (with possible live call afterward), is a kind of "coding during the interview process", yes?
The essay more specifically concerns a certain specific type of coding evaluation, with live evaluator who is unknown to you, in a setup which inhibits many normal problem-solving techniques (pacing, drawing notes, thinking silently for 10 minutes, etc.)
> It can literally be a simple task that one would expect any working software developer to be able to complete without too much fuss.
I think the two of you are in agreement even for that point. That is, the author seems to agree, writing "They likely do a great job filtering out people who are incapable of programming".
The point that's often overlooked by people who complain about any sort of interviewing process (this one being an example) is: somebody _does_ pass the interview, eventually, or they change the process. The poster here seems to be suggesting that live coding is an impossible hurdle that nobody can overcome, but that's obviously not true - people overcome it and get hired all the time. I've auditioned as a musician in the past and it's tempting, when rejected, to say, "well, their standards were just too high" - but I have to find the humility to recognize that they just went with somebody better, because they did end up going with somebody.
>hey just went with somebody better
Yeaaaaah, that's not how any of this works. I've done 100+ interviews for MegaCorp. Who ultimately gets hired is ultimately a dice roll no matter how "data driven" we call it. Did you get a good loop? Was one of your interviewers in a bad mood? Did you end up with a hiring manager who "used to code" and now measures everyone against whether or not they use out dated OOP techniques everywhere? Ah crap, did you get Gary? That guy sucks so much. He asks "hard" questions to make himself feel good.
The sausage is what you'd expect if you remember one key thing: it's humans on the other side of the desk. They're finicky and arbitrary ceatures.
Flashbacks to losing out on a dream job because the husband and wife interviewers had a nasty argument after a long plane flight right before the interview. Sigh.
> Who ultimately gets hired is ultimately a dice roll
Well, I'm not sure I believe that's universally true (or even particularly common) but even if you're right, what difference does it make? You still have to be good enough not to bomb the interview but then just "hit the tables" often enough to get a lucky roll. That would be the case for any sort of interview process, whether it was live coding, an informal discussion, or tests of strength.
Sometimes someone is passed by the interviewer, which is not the same thing as passing the interview. That is, they just pick someone, who may or may not be the best candidate.
Your experience sounds horrible, and unfortunately those sort of "tests" are just a horrible thing. Generally, do NOT do take-home exercises. That's your time, which has value; the company doing the hiring contributes nothing. At least, in a live-coding exercise both you and they have some skin in the game. As with all tests, good or bad, part of the test is actually doing the test. Often it's less what you know, or could do; and more about "can you do the test?", and mostly "doing the test" is a poor proxy, and the whole exercise becomes a waste of everyone's time.
There's also the issue of scalability.
Grinding leetcode at least scales horizontally to a huge number of companies.
Homework is typically useless outside of the single company you're doing it for.
I generally refuse all takehome assignments unless:
1.) It sounds uniquely interesting and fun to do. 2.) The company is prestigious enough, or pays well enough, that making any effort to try to get the job worth it.
Completely agreed. Also in my experience the takehomes are even more susceptible to being rejected for absolute bullshit reasons. Someone told me the thing I coded up in less than an hour to solve their problem was “overengineered” so they wouldn’t be moving forward.
Also... I think at 100% of the companies that have given me a homework assignment, it has always been in addition to, not in place of, a standard leetcode whiteboard round.
Same here. Though it’s hard to blame them since some people quite obviously didn’t do the takehomes themselves.
Why would you be interviewing at all if the criteria for (2) isn’t met? Unless you’re just trying to get interview practice or keeping them as a safety option.
I wouldn't dismiss an employer out of hand just because I wasn't already familiar with them but I'd be quicker to back out if I saw something that didn't inspire confidence.
Sometimes people just need a job (especially in the US, where jobs are so coupled to health care). I've been in that position, and I'm very glad I'm not now.
The last time I did a take home exercise for a position I wrote something that was beyond the team's skillset. That was awkward.
That's a great reason to do the take home test. You learn a huge amount about the team by how they react to your code.
Or they just evaluate among themselves and never tell you what they thought beyond thumbs up or thumbs down.
Or just completely drop the ball and never get back to you.
> Often it's less what you know, or could do; and more about "can you do the test?"
It really depends on how you create the test and how you review it. You can prepare one where the actual solution is 10 lines of code and anything else is exactly for showing what the candidate knows / could do.
I used something like "read data from CSV, write it to sqlite, treat it like a mature production app, feel free to use placeholders (usage doc goes here), go nuts". Then the test was really about knowing about error recovery, encoding issues, documentation, error reporting, monitoring, ci pipelines, etc. Many badly organised tests don't make the whole concept of a take home test bad.
Your test sounds like what people complain about. You give them instructions but it's "really about ... ". Just ask them what you want them to know. If you hired this person and they turned in code without error recovery or encoding, you'd tell them they need to add it and they would. No need to obfuscate your intentions and have them read your mind in what you're looking for.
No take home test is even needed. A simple conversation would let you know if they understand the importance of those things.
> Just ask them what you want them to know.
You can't really ask about those things directly. If I ask "(how) would you add monitoring to this?" you can make something up on the spot. If I ask "what else would a mature production app contain?" that will give me a better idea of what you're familiar with.
> you'd tell them they need to add it and they would
Different levels. We were not after juniors in that case, but people who can do independent work. The question would be way more specific otherwise.
> No need to obfuscate your intentions and have them read your mind in what you're looking for.
It's not about mind reading. For that role, you either know how to create/manage serious applications or not. Your specific ideas may be different than mine and that's fine, but the general areas of interest will match. E.g. whether you register some event notifications, do text logging, persist decision journal, or do something else — you'll likely mention that. And you may forget/miss one or two things - that happens. But you don't miss them all if you're a good candidate for that position.
When you say make it up on the spot, I'm hearing they can answer questions when you ask. If you ask them to describe how they'd process input and send it to a database they can just as easily tell you what they generally do. If they don't mention any of the things you're looking for, then you have your answer. If they answer some then you can decide to probe about what's missing, or thank them for their time. You haven't explained why a take home test is needed for that.
> When you say make it up on the spot, I'm hearing they can answer questions when you ask.
Or they have enough surface knowledge to risk an answer that could potentially work. It depends if that's the threshold acceptable to you.
> You haven't explained why a take home test is needed for that.
We're commenting on the blog post about this very question. Making the initial stage of this test take-home is at least partially solving points 2,5,6,8 from the list.
It sounds like we're now back to mind reading the answer you'd like them to provide based on overly broad questions. That is a poor way to assess whether they know about a topic. Ask them what you'd like to know. You don't have a relationship with them and they aren't familiar with your version of "normal" or "mature" processes.
It doesn't matter if they're familiar with my specific concept of a mature app. (Although just reading the job description would cover 90% of it...) There's always going to be a lot of overlap of concepts anyway. The question was broad exactly because there are many good answers.
(FTR, the answers were pretty consistent in what elements they included, so it wasn't confusing for the candidates)
That's like a trick question given that interview questions typically want the opposite.
The instructions were longer and clear - nobody tried to second-guess them. So not an issue in practice.
That’s not always accurate that the company doing the hiring contributes nothing. (Also good companies could offer to pay for that take-home test time.)
Having done take home before from the hiring side, it was incredibly time-consuming for us. We had someone anonymize the three finalist submissions, and then we had three people each individually review and comment on each one, and then we got together to discuss and choose the final candidate. Once we agreed on one, only then was it de-anonymized.
All-in, it took way more time than a single developer doing three live coding interviews. But my guess would be that most companies wouldn’t be willing to be that deliberate with take home.
There’s no guarantee the employer isn’t in final phases with someone else that will lead to them never even evaluating your work.
One of the better laid out interviews I had consisted of them pitching a problem prompt via email, followed by an hour and a half to complete the problem, followed by a ~30 minute code review. Realistically, the task was like a 20 minute job, so plenty of margin for stress-related slowdown. I think that's a happy medium between unbounded take home exams that monopolize your time, versus in-call leetcode whiteboard sessions.
The author did write "better yet, let applicants do a take-home, and pay them for their time. ... think of it as another chance to be selling them on joining your team over some other team."
I added that after-the-fact in response to this comment. So that's on me for not including it originally. It's a great idea and one I wholeheartedly endorse because it forces the company to put some skin in the game and helps limit the number of applicants that they would request perform the take home test while also recognizing that applicants' time is in limited supply.
I think we're going to see a pretty large industry shift away from take-home tests given how easily 90% of these tests can be gamed using chat GPT.
I'd think we'd have already seen this with GitHub Copilot. There was an interview I was part of late last year where the candidate had Copilot turned on during the live coding, and he didn't turn it off even after it was obvious that's what he was using. What I was more surprised by is how I thought this was a bigger deal than everyone else. Like, why come up with these elaborate tests when the candidate is just going to use autocomplete the whole time?
Maybe programming in another few years will just be glorified autocomplete and little more.
And perhaps testing people on how to write code was a mistake to begin with. It's one thing to write code, but reading code is another.
> Maybe programming in another few years will just be glorified autocomplete and little more.
It won't. For the same reason code generation from UML diagrams hasn't replaced programmers [1]. Or why business owner/analysts don't write acceptance tests in Gherkin (Given-When-Then).
Though some tasks that programmers had to do manually in the past may be automated (not the first time).
If they are effective at using it to solve problems, why not?
The real challenge is to your testing mechanism and not the candidate
Take home tests are often paid and those are the only ones I will do anymore after being burned a few times.
That solves the skin in the game problem.
If they're not paid and you still wanna do it, make it into an interesting problem you wanna work on and have it be presentable enough to include in your portfolio.
At least, that way you might get something out of it instead of nothing, although I definitely do agree that being paid for it is the only way to solve the skin in the game problem.
I keep seeing people claim that no one offers alternatives to live coding in interviews, but not only does the author provide an alternative (take home coding test/project), there are obviously many alternatives to live coding. I've been working in this industry for over two decades, and I've interviewed more candidates than I can count. I can assess a candidate with a 20 minute conversation, because a candidate's understanding of the context and theory of how and why things are done the way they are determine how good they are at the job. Ask them what happens when you visit a website. The amount of detail they go into, and what details they focus on, will tell you an enormous amount about them. Ask them whether they prefer an OO or functional approach to organizing code. The good candidates will have thorough, well-thought out opinions, and you're handing the bad candidates enough rope to hang themselves.
Of course, in order to be able to assess the answers to these questions in an effective way, you need to be able to very knowledgeable yourself, and that's the real root of the problem here.
No, they're not saying there are no possible alternatives. They're saying there are no better ones. Different claim.
Software development is not a performance art (like playing the piano). Putting someone in that kind of a situation is, well, sorry, as dumb as can be.
What's important is how people approach computational problem solving, not if they can write a solution in 1 to 45 minutes. Really, who cares?
One of my go-to examples is trying to work from home. Which is great, yet has its challenges. We have three German Shepherds. They are lovely. However, when a delivery arrives or the gardeners are out nearby, well, it's mayhem for a few minutes. I've come to understand that I should just take a coffee break when that happens. I can't even do mid-level math, much less focus on a difficult CS problem during those moments. And it takes a good 30 minutes before my head can be back on task.
Stop treating software development like performing art or athletes having to perform in the heat of a game. That is not what we do. At all.
If I do a live coding challenge, it is absurdly simple and is meant to root out people who have literally no business being in the room.
Write a function that takes in a string and returns 1 if the amount of letters in the string is odd. It returns 0 if the amount of letters in the string is even.
If you can do this, we are good. I dont need NP hard or whiteboarding or algorithms. I can tell how good you are just by talking to you about your background.
Some candidate, done, sweating: "I wasn't sure if the limericks, speeches and open letters count as letters here but the criteria used are easily replaceable" /jk
I can never remember how modulus works, I use it so infrequently.
Just use n & 1 then. An odd number ends in binary 1; and’ing it with 1 yields 1 if its odd.
You don't even have to get that cute. Do integer division, multiply the result by the divisor, and subtract the product from the original dividend. Surely you can't say you don't use multiplication, division, and subtraction.
Sure, but you probably remember division.
"I can't remember of n % 2 returns 0 or 1 on even, so in place of that I wrote (if ((n/2)* 2 == n) for even, assuming integral division"
highly recommend learning how division works within assembly. If you can understand sending this data into registers, you will never forget how mod works. Honestly, I dont even care if you use modulus or not. Solve the problem within a function demonstrating you know CS101 concepts we are good. like, really. No judgement, this is just an idiot filter.
I think implementing a function that does what a modulo operator would is itself a fairly straightforward question.
Which kind? For example, can you name a difference between Python and C semantics for % op (if any).
the best fit jobs I've ever had have also taken this approach. Absurdly easy programming problem, then 95% talking about my background.
Bonus if you use n & 1 instead of n % 2 != 0
The problem with live coding interviews is that they shouldn't exist.
Another problem is that we're prone to thinking that being able to do well on tests equates to doing well in life and work--despite a stunning lack of evidence in support it.
So what do you propose? Picking names out of a hat?
We expect too much of 'interviews', so we've built an entire structure around them: 'behavioral' questions, whiteboard exercises, resume keyword scanners, long, tortuous lists of 'qualifications', elaborate processes of multiple interviews, etc.
After many years of sitting on both sides of the table, I've have come down to this: "hire lightly and fire lightly". In practice, this means beyond the (very) basics, hiring is not algorithmic; it's a crapshoot.
Well if I have two openings and five people what way do you think is more fair to choose among them?
On average, any random company will be filled with solidly average engineers. Which means, because of how numbers work, most of the engineers are neither good nor bad at engineering. They’re just mediocre.
Companies need to realize this and come to grips with it. Additionally, this holds as well for the following: beyond a certain skill threshold, a company will not really be any less competitive by failing to hire the best or even second best candidate. If you are a regional telecommunications firm, and hire the 3rd best candidate, the 1st and 2nd candidates will likely get hired in unrelated firms or unrelated regions (especially in the remote work era). And even if this isn’t so, because most everyone is mediocre, the competitiveness of your firm won’t drop appreciably.
OK. How does this answer my question, since presumably I would still want to identify at least the mediocrities rather than the complete jokers who have nothing to contribute?
Get them all to roll three 6-sided dice, and pick the one with the largest sum
So we’re actually back to picking names out of a hat.
Have you tried picking names out of a hat, and comparing the results over time? I'm being serious.
No, I'm not in a position to conduct such an experiment. Even if all the interview process does is identify people who were conscientious enough to prepare well, it doesn't seem unfair to me.
Also a bunch of us have autism, so coding, presenting, trying to gauge the audiences, existing understanding, ensuring remaining available space on the whiteboard, working out, went to talk, and went to type, making sure you were in gauged with everybody in the room, and all the other aspects of presentation at the same time is difficult.
I think reviewing code is a much better method.
It's a real skill you'll actually employ. You're coming at the code cold, which is actually a realistic scenario you'll encounter on the job. Your ability to catch bad ideas and prevent them from getting literally codified is a valuable skill. And all of that is worthless if you're in a state where you can see a mistake, but are too afraid to speak up; this gets tested too.
It might not be so great for newbies and people fresh out of college, but even they should be able to read the code and discuss it.
I agree, and this is what we have been doing for a while at my current job (I went through the same process, so it's been done for at least 7 years).
The candidate would come to the office (when we were still doing in-person interviews) and meet a couple of the developers for a little bit of chatting. They would then be given a couple of printed pages of Java code with a few basic classes representing banking accounts with some typical methods for depositing, withdrawing, persisting to a database etc. (with all of the details stubbed out) - and they would be given these instructions:
The candidate would be left alone with the papers and a pen and would spend the next 15 minutes looking over the code by themselves.Read the provided Java source code files and identify any problems that you can find. The problems can include things such as actual bugs, design problems or code quality issues. You do not need to search for syntax errors - the code does compile. You have 15 minutes to find as many problems as you can.The rest of the interview would then be spent discussing their findings. Most candidates would find the obvious problems in the logic, missing null-checks etc., while trickier things like synchronization issues were missed by quite a few. Even though we had a list of all the bugs/issues that had been put into the code, the important part wasn't for a candidate to check off as many of these as possible - the important part was the discussion about the issues that followed.
After the candidate had told us what they found, we would start hinting about the remaining issues and eventually tell about all of them. How quickly someone would pick up on an issue when it was pointed out told us quite a lot. It was a way to get a feel for how the candidate thought and reasoned about code, without the pressure of them having to actually write code with someone looking over the shoulder.
Using whatever tools would be handy.
Throwing it through find bugs is always a good option, and should find something
That could be interesting. I had a co-worker (at a different company, while on an open-source project) whom I highly respected because of his excellent code reviews. I think it would have to be done with an established build chain (e.g. formatters, linters, CI) so as not to degenerate into trivia and bikeshedding. I think it would be most interesting (assuming a good interviewer) if the approach under review were actually somewhat defensible.
Agreed and even using the same code for different levels can work well.
The juniors spot obvious errors. The seniors also spot logical flaws and the conversation quickly moves into "how could this be done better, overall".
Well stated. I seem like the exact type of dev as described in this blog post - I get performance anxiety and the audited interview process doesn't fit my particular style. I wouldn't describe myself as slow, but my steps to solving a problem are often not linear and sometimes difficult to measure in such a setting.
as a devops engineer as my main job, I also try to explain to interviewers that on any given day I am reading and writing half a dozen languages of vastly different paradigms, and although I'm very proficient in many of them, I definitely need to reference things that maybe shouldn't "need" to be referenced (like confusing bash/python loop syntax is very common for me as an example). This rarely ever slows me down in reality, but will definitely cause me to fail interviews I shouldn't.
If I was an interviewer, I wouldn't care if a dev knew whatever esoteric language syntax or API calls by heart. I'd just expect them to know how to use them intelligently. The former does not always imply the latter.
For what it's worth, I do interviews for my employer and we allow candidates to use Google during the coding interview as long as they screen share their browser window.
An impressive number of people still fail the interview despite the questions being pretty simple (in leetcode terms, probably on the easier side of medium).
> Companies may deem these acceptable concessions relative to having a consistent and un-biased process....
YES! Yes, exactly. A consistent and unbiased process that reliably weeds out people who is very bad at programming is incredibly useful. I'm not convinced live coding interviews are either of those things, but assuming they are, they are absolutely worth the listed downsides. Do they filter out lots of great programmers along with the bad applicants? Yes, totally, and that's a major waste. But all of the listed alternatives are less reliable, less consistent, slower, and friendlier to cheating.
I would love to eliminate live coding interviews where I work. I hate the things. But I have never encountered a mostly consistent and kinda objective solution that compares. I was hopeful that the essay was leading to a proposal for one and disappointed once again at its lack. Please, someone tell me what the giant tech companies should use instead, and I will gladly throw these "please reverse this linked list" interviews into the trash.
I have a very simple rule that I’ve been following for years and it always worked for me. I ask a recruiter about the interview stages, and if I hear “live coding” I immediately decline this opportunity and switch to the next one. Eventually I was able to find really great places to work at, that made me happy for the next years.
its extremely distracting. its like attaching an anchor to my brain. its like driving in traffic while simultaneously trying to text back and forth with someone on your phone. both suffer
and very unnatural
and insulting to an experienced person (whos been promoted, and whos survived many layoffs in the past, etc) with many accomplishments. someone who has clearly solved and shipped, repeatedly. and one who has artifacts visible out in public. and has praise testimonials from former bosses, coworkers, clients, etc
Amazingly, this latest take on this shopworn topic, which says nothing we haven’t all heard over and over, actually anticipates my response, “what are the alternatives?” and then fails to make much of an argument for any of them. Next.
"Here are reasons why this thing is bad. I don't have a better option."
This type of interview related blog post seems to hit the front page every week. Almost everyone agrees that interviewing doesn't feel good and seems overly complicated/difficult/whatever.
I hate that interviewing is a skill that has to be developed, in large part, separately from other engineering skills. I also don't really enjoy SQL/Database work. But I've gained enough competence in SQL that I'll be fine in most jobs. The same is true for my interviewing skills.
I don't have a singular, one-size-fits-all better option, but I explicitly included several options that I've seen work well either as the applicant or from the hiring side. I just wasn't going to presume that there's one perfect replacement that will work for every team or role.
You just listed off a bunch of ideas. A lot of thought and argument went into the current arrangement and you’re just giving us a brainstorm.
Everyone complains about the state of SWE interviews yet nobody seems to be able to come up with a legit better alternative. Even the author recognises this in the article.
This format is popular because it's the best time/effort trade-off for both the company and the candidate. It's massively flawed, but everything else attempted so far turned out to be even worse.
How do other industries do this?
Is programming weird because you can just ask someone to prove they know how to use a hammer? And so other industries just have to hire based on work history and/or bias "culture fit" during the interview? And they suffer terribly from people who can talk the talk but not walk the walk?
Or is programming weird because there's so much propensity for people to be able to talk but not walk? We rely on nerdspeak and jargon so much that just being able to prattle on in a dilbert-esque way would otherwise convince someone to hire you?
> How do other industries do this?
There's lots of other ways, but some of them suck.
Some fields attach a lot of weight to your alma mater, maybe they only hire people who studied law at Yale or Harvard.
Some fields require not just a degree, but also years of study under an industry veteran. Sometimes that also involves hazing like working 70-hour weeks, for some reason.
Some fields require work-sample tests where you show up at a given location and demonstrate your abilities on demand.
Some fields require not only a degree, but also years of working for free in order to break into paid work. And the paid work is far from guaranteed, that free work only pays off for 10% of people.
Some fields don't offer permanent employment, instead hiring people for much shorter periods - so bad hires can just not be rehired for the next project.
Some fields have a literal test you have to pass to get a license which allows you to practice that field, and some even have laws that criminalize practicing in the field without said license.
Given surgeons don't have to do live pancreatectomies as part of the interview process, I'm not convinced that's such a bad thing.
I am not sure. I think part of the reason is that programming lives completely on the digital plane, while other fields tend to have physical components that can't be faked and allow you to grasp someone's knowledge in a much more straightforward way (e.g cooks produce physical food that you can see and taste)
Probably a little of both. A staggering number of candidates are just completely unable to solve simple problems.
Many years ago, I interviewed a candidate at Amazon for an SDE3 job (aka Senior SDE.) I asked him to write a function that takes 2 sorted arrays of integers and merges them into a single sorted array of integers. He just couldn't do it.
This is very simple problem to solve, and I think anyone who can't do that is not suitable for SDE3 (or 2 or 1, in my opinion)
One could argue about very difficult problems - can a candidate really be expected to invent the solution on the spot, or is it just because they saw it on LeetCode? Is it luck or ability?
But for simple problems, they are a strong negative signal and a very weak positive signal: just because you can merge sorted arrays, doesn't mean you're qualified, but if you can't even do that, you're most likely not qualified.
Triplebyte and a few others in that space were able to do pre-screening for companies that chose to go though them, which skipped the recruiter call and pre-screen and got you on site faster. Worked pretty well, but they failed to figure out a business model and we sold to Karet which just shut down Triplebyte's candidate vetting process.
I've had good experiences as an interviewer doing code review style interviews.
The candidate gets some code and a ticket description of what the code should do.
If he spots an issue, he can fix it right in the code or simply explain it.
I'd that testing the ability to read and understand code is both less stressful for the interviewee (no "judging every key stroke" stress) and more helpful for the interviewer.
Spend 4 hours coding, while being paid by the interviewer, watched by a peer, working on a bug fix or new feature.
Works great.
> Another unavoidable facet of live coding interviews stems from effectively requiring someone to just start typing. For plenty of people, that’s not a problem, but for people who tend to be more tactile, visual, or kinesthetic, just banging on a keyboard isn’t a natural way to begin problem solving.
In my experience conducting these live coding interviews, it's almost a universal rule that if the candidate starts coding immediately, they will waste a lot of time on irrelevant details, or take a long time to see that their approach can't work.
I always try to encourage the candidate to talk through their solution or draw a diagram if it helps them. Candidates who follow this advice always perform better.
There are problems with not doing any live coding too. I've interviewed people for programming jobs who can bullshit their way through a phone screen but literally can't write a for loop or a simple fiz buzz program.
> In my experience with live coding interviews, the companies and teams seem oblivious to the gaps created by using live coding interviews as one of the key signals.
To me this is the key to the whole problem. For many companies live coding interviews are a cargo cult approach to interviewing. Since the people that do them don't understand how to evaluate others they become a checkbox exercise that only evaluates if the candidate would tackle the problem the same way the interviewer thinks they would tackle the problem.
It’s as simple as this: refuse any and all interviews that require you to dance
I refuse to take those interviews. I had one company that promised they wouldn’t give me a live test, and when I told that to the interviewer who was trying to give me a test he said “hm, well we’re going to do it anyway”
I passed the test and was given an offer which I shot down for the company which respected my terms
In the end the other company was left high and dry when they needed the new team member
Just do it. Enough of us see this for what it is. If we just act how we think we change the industry
Here's the problem. If the company offers you 600k salary then the dance is now worth it.
The other problem is after lay offs, if all companies ask you to dance then you have no choice. Dance or stay unemployed.
Obviously nobody wants to "dance" and nobody would if they had the choice.
So the only thing you can say in response is giving excuses why people must dance?
Sometimes you’re desperate and will do things you otherwise wouldn’t. That isn’t and should be thought of as the norm
It's a negotiating term. Ask for more to dance
People don't have to dance. Incentive to dance is very strong though.
Here's the thing, when you look at the behavior of large groups of people the overall incentive is the overall outcome. As an individual you can resist but groups act predictably and follow straightforward patterns.
Think of it like drugs. You going to tell people to stop doing heroin and end the problem forever? No. You gonna tell people to stop dancing and end the problem forever? No.
The only thing that will bend the industry in favor of your idea is software unions. Centralized control can influence individuals to act in cohesion. Without leadership the short term incentive wins.
You’re saying people are weak and can’t enable change if they chose to behave a certain way and instead we need more governance and powers above us
I fundamentally don’t agree. It’s simple: prioritize dignity in yourself and the world will give it
I don’t accept interviews where i’m required to dance. My career has only gotten better since I made that choice.
Even on the individual level you can make this right. It just takes effort.
>You’re saying people are weak and can’t enable change if they chose to behave a certain way and instead we need more governance and powers above us
Yes, I am saying people in aggregate are fundamentally weak. Certain individuals can overcome weaknesses but in aggregate we can't.
>I fundamentally don’t agree. It’s simple: prioritize dignity in yourself and the world will give it
Then you fundamentally don't agree with reality. Tell that dignity thing to a drug addict. Tell him to Prioritize his dignity and stop doing heroin. End the drug problem because people are strong. A few people will listen. In aggregate, nothing changes.
>I don’t accept interviews where i’m required to dance. My career has only gotten better since I made that choice.
On one hand you have control. On the other hand maybe you don't dance because you can't dance. There are people where "dancing" is equivalent to donating a dime to charity. A Trivial effort. To them the interview process is the equivalent of giving a dime in return for half a million dollars. Why the hell not?
>Even on the individual level you can make this right. It just takes effort.
The individual level is the only place where someone can refuse to "dance." For the industry to change it has to happen in aggregate. The aggregate change is impossible. There will always be people willing to dance, and there will always be people where "dancing" is a zero effort hand wave.
It’s not impossible, you’re just giving up and refusing to be a part of the solution
The path of least resistance long run is demand dignity. The path of more resistance long term but ease short term is accept undignifying circumstances
You can make your situation and the worlds better by doing the right thing. Or you can sit and argue to a stranger why it’s ‘impossible’ to tell a potential employer to pound sand when they ask you to do something demeaning
Choice is yours
>You can make your situation and the worlds better by doing the right thing.
Did you not read what I wrote? You or I both doing the right thing changes nothing.
People doing the right thing changes things. This isn’t about me reading it’s about you making a false claim repeatedly for some reason.
Tl;dr but this debate has been done, a million billion times already.
The author concludes with a decent summary of the issue (ie. Is this the best method among the worst?). But doesn't actually find a better way of doing things.
And in the million forum threads on this issue, no one else has either.
Of course there are. I've hired a person once because I had known him 20 years ago and he knew his skills were top notch. (And to be clear, we haven't been in touch for many years since.)
But I don't know that many people. Properly conducted live coding is the least worst alternative out there for complete strangers.
Pay the interviewee, have them fix a real big or add a real feature, do it in a pair with another dev.
Works great.
If you're grading someone during the coding interview exclusively on their code, sure. I use a pairing interview to make sure that the person can collaborate effectively, think out loud, and shake out ambiguity.
There is a nice side effect about doing this as a coding interview; there are often obvious indicators that a candidate is a poor fit -- for instance, big knowledge gaps in the standard library of the primary coding language.
More important is how the candidate structures how they would solve the problem, and how they communicate it to the person on the other side of the discussion. Are they taking testing into account? Do they iterate rapidly, or have a monolithic solution they have a hard time conveying?
For lack of an effective whiteboarding solution with remote interviews, coding interviews are here to stay. They should be reframed, though, as focusing on collaboration -- not on being a leetcode champion.
After having been through this crap and now being on the other side of the desk, I've come to the conclusion that this is simply standardized testing for adults, with all of the same myriad problems. It doesn't identify what you're actually looking for, gives you more false signals, and alienates the talent you actually want.
The way we do these at my current job is extremely productive for us. We look for two things, apart from basic competency: problem solving, and asking for help.
It's structured as a two 90 minute pair programming sessions. Interviewee shares their screen, and we work through the problems together. Obviously it's pretty hands-off, but we guide and nudge where appropriate. Here and there, when they use something that relates to a deeper topic, I'll ask questions to gauge how deep their knowledge goes. Like asking if they know how C#'s foreach works under the hood. Not as a selection criteria, but simply to get a sense of how much they know.
Use of a search engine is openly encouraged. A lot of the time, we don't even care if the program actually runs. If they struggle with syntax or the correct function overload, we'll help them out after giving them a little time to find the solution.
I also throw in a problem designed to get them stuck, and ask questions I expect they can't answer. A good programmer asks for help and admits when they don't know. A bad one bullshits their way through.
We want to hire programmers who can do a real job in the real world. Implementing red/black trees on a whiteboard blindfolded isn't a job skill, it's a party trick. That's not something a programmer will ever need to do.
In the real world, real people use google and stack overflow. They don't have encyclopedic knowledge of the entire language's syntax. They ask their coworkers for help or opinions.
Our interview process is designed to show us how a person will function in a scenario as close to the job as possible. Because that's what we're hiring them for. We look for their ability to work through a problem with the resources that everyone always has. We look for how they work with others and how much they lean on coworkers.
This has worked out extremely well for us. We've hired some very talented individuals, and have totally avoided the archetypal shitty dev. The people we hire immediately mesh with the team, and learn and grow the way all programmers do.
Granted, we are a small company and we have the time to have our own programmers giving interviews. We also have a much higher need to be so selective. But every single person who has made it to the technical interview has remarked unprompted that it's the best interview they've ever had. And I mean 100%.
It's because we treat candidates the way we'd treat our own employees. They get to know what the job is like, and we get to know how they'll do the job.
Seems fair, kudos
I feel all discussions about the coding interview are missing two critical points: first, how we got here, and second, why are we still here.
First, how we got here. I didn't experience this first hand, so I'm just making a recount based on what I've heard and read from "old timers". My understanding is that before whiteboard, leetcode-style interviews became the norm, tech interviews were mostly unstructured and quite informal. In the really old times (pre 90s) you could get a job just by knowing how to use a computer. I believe this wasn't that unusual in the 90s and early 2000s. I still remember hearing a founder-CEO bragging that his interview process was a 30 min chat with each candidate, and if he liked them, he would hire them.
From what I could gather Microsoft is the first big company that started using these "coding challenges". Then they became wildly popular thanks to Joel Spolsky [1], the publishing of Cracking the Coding Interview [2], and Google made brain teasers world famous.
Second, why we are still here. First, I believe there is a huge cargo-cult factor. Companies want to copy big tech, and alumni from these companies go on to found their own. This kind of interview has been honed and polished over the years, landing in a local optimum. An entire industry of websites and products has been created, and there are many entrenched interests. People might hate it, but the process works well enough for tech companies that they don't need to worry too much about it. Another under-appreciated factor is scale. This kind of interview sort-of scales well, which is important when you hire at a massive scale. That's why things like "have the candidate come to work one day and pay them" won't work for companies that need to screen thousands of people a week. Lastly, a standardized and "well known" process introduces some guardrails that can avoid some obvious pitfalls. The book "Working Backwards" explain how the Bar Raiser program was created at Amazon when a bad senior leader hire used the unstructured approach to hiring to build an empire misaligned with the company. At a big enough company this is bound to happen sooner or later.
Third, where do we go from here? I find it extremely unlike existing big companies will change their methodology any time soon. It might not seem like it, but for a company like Google it would be a massive undertaking to overhaul their hiring process. It would take years to achieve the level of efficiency and effectiveness of the current system, and surely there would be tons of opposition.
I believe the only way forward is for new companies to experiment with other methods of hiring, particularly at the beginning when they are nimble and can experiment freely. As they grow they will face challenges scaling, polishing and standardizing their process. At some point they will become the next generation of Big Tech, and the cargo cult wheel might spin again.
In any event it seems we need some sort of structured approach to hiring where we assess the match between company and candidate.
[1] https://www.joelonsoftware.com/2006/10/25/the-guerrilla-guid...
Agreed.
About 1, big companies need regular standard hiring sessions also for anti discrimination, and simplifying comparisons of larger numbers of candidates. Bureaucracy can come about from involving big numbers of people.
Google, Facebook, Amazon use it for a reason. And they are top engineering companies. Period.
If their process for hiring was so spectacular, you'd think they'd have been a little more reluctant to do mass layoffs.
Why? They weren’t laying people off because they were bad at programming. And in fact they were likely hiring so aggressively in the first place in part to keep top engineers away from competitors.
The process for hiring senior management is different!
I worked at one of the companies in your list and will tell you from experience that they've moved on from the old and tried process for many types of interviews. My technical process with them back in 2019 was:
1. Tech phone-screen: work through a simple but relevant to your job exercise (no leetcode stuff and code didn't need to run... i.e. could be pseudo code).
2. Take-home exercise: where I got the chance to write quality code at peace and a technical design doc describing my process and findings. I'm certain the take-home made up for 80% of the decision.
3. On-site: chatted with both engineers and non-engineers about the exercise, and how I would've approached new hypothetical but real-life requirements.
Everything past the take-home exercise mostly revolved around that exercise. If let's say I had cheated the whole exercise, I guarantee you it would've been evident immediately for many reasons – including the fact that when I was working on the exercise I discovered and reported a bug in their own production software.
I worked at one, alongside others who like me had gone through their (in)famous process, and alongside others who had not (they came via acquisitions). There was no difference.
Similar has been said about Yahoo, MySpace, IBM and other companies that completely failed to innovate after adopting this erroneous line of reasoning.
Live coding interviews are the future. The risk of false impersonation was low enough for a while to rely solely on virtual screens, but widespread access to and awareness of large language models has made any virtual exercise unsustainably corrupted.
Sure, you can often tell the folks who know nothing when asking them to explain the code, but it's increasingly difficult to tell a great engineer apart from a mediocre one that's cheating, and once you start hiring cheaters the toxic effect to culture sets in fast.
One of the author's points is that the live coding questions are often the types of problems which, in the workplace, might best be done with GPT.
"So at that point, do they want to see you muddle through it, or would they rather see that you know to have ChatGPT run through the initial pass and then refactor?"
"If a company is evaluating engineers with questions that can be easily answered by AI in seconds, what are they really evaluating for? Perhaps they’d be better off hiring a chat bot."
If a candidate said that they would use ChatGPT to answer the problem, then I'd start to talk to them about the legal issues surrounding that approach. What would they do if the hiring organization forbade the use of ChatGPT for anything?
I would hope that all SWE candidates would understand software licensing to some extent and how to behave in a way that would not put the hiring organization in a legally risky position.
Given your hope, shouldn't you be asking software licensing questions to every candidate? Irrespective of them mentioning ChatGPT? Otherwise, what happens if two months in they start using ChatGPT?
ChatGPT isn't the only legally risky software licensing issue.
A lot of people, including me, will use a search engine to find a solution to a given problem. That solution may be covered by copyright, and subject to a license which is not compatible with the company's business model. Some companies prohibit the use of GPL software. I consulted for one company which required authorization before using any open source software.
So it seems to me you should already be asking people about these sorts of issues.
I think that’s not really relevant, in the same way that it doesn’t much matter when they take a football prospect’s running times that someone could go much faster on a bicycle or in a car.
I'm pretty sure I heard the same point made 20 years ago against using IDEs, with their auto-complete and tool-tips and style feedback.
And 40 years ago in the debate over letting kids use a calculator in school instead of calculating by hand.
A footballer follows a very constrained set of rules. If the footballer were allowed to use a car then 1) it would be easy to score a point, 2) the field would be ruined, and 3) people wouldn't pay to watch or support the team.
If a programmer uses ChatGPT to get a handle on a task with a new API, and saves a day of futzing around, how is that NOT relevant to job performance? (For the sake of argument, let's say that experiment showed that API doesn't scale well enough, resulting in a decision to scrap that approach entirely and use a different API.)
We’re not talking about whether you should use them at work; we’re talking about whether it makes sense to have an evaluation where you don’t use them. Closed-book exams are similar. There’s no real-world situation where you wouldn’t be allowed to refer to whatever materials you like, but the evaluation uses somewhat unnatural circumstances to gauge how well you’ve assimilated the material in a limited amount of time and with a consistent process that’s fair for everyone.
"It makes sense" leads us back to the linked-to essay, which argues that live coding a solution to "arbitrary and unrelated topics like creating a script to handle scores for bowling or something equally irrelevant" do not make sense as a way to
If the evaluation doesn't make sense, then the conditions placed on the evaluation don't really matter.
Similarly, "assimilated the material" only makes sense if the live coding interview really does cover "the material". To use your analogy, measuring a football prospect’s cycling times aren't that good of a test of football playing skills. I mean, yes, there's some overlap, but there are more useful ways.
And one of the example tests was "handle scores for bowling", which is far from most work-related issues.
"consistent process that’s fair for everyone."
The author addressed this idea at several points, including "Any belief that a live coding interview is a consistently reliable way to make an objective assessment represents willful ignorance at best."
Picking a name out of a hat containing potential employee names is also consistent and fair.
Just because it's easy to measure doesn't mean it's an effective predictor.
I suppose I fundamentally do not agree that the skills are unrelated. I think they are the same skills, but applied to a much smaller problem that fits in 45 minutes. The author does not make any case that the smorgasbord of alternatives he offers are any better. And none of this has to do with your “well I can just Google a well-known problem like this in real life so it doesn’t matter” argument.
And then your football team will lose to players who are all on bicycle?
No but it’s not like in a football game the other team is just going to let you run in a straight line without trying to impede you either.