In defense of coding interviews

If you have to prepare for the interview, doesn't that suggest the assessment isn't representative of day-to-day programming skills? In fact, it suggests the exact opposite.

With zero preparation, I could go to an interview with you right now and tell you anything you wanted to know about designing a database schema, how to get any kind of data out of that schema, how to make that data available via an API, how to cache the data and perform permissions checks, how to deploy code to a cloud environment, how to set up a build pipeline, or any other question you might want to ask about the day-to-day process of developing and deploying web applications. I can answer those questions because I already work in the industry and I broadly know what I'm doing.

However, I probably couldn't pass a FAANG interview without months of prep work, precisely because whiteboard interviews bear almost no resemblance to the reality of software development.

As someone who has done the interview rounds at FAANG and many dozens of other companies in SV - you either work in algorithmic work or got lucky with your questions.

The questions that are asked have little to no basis in my daily work as an engineer. Same is true for the hundreds of other engineers I’ve worked with who have gone through the same hoops that I have.

> I think the assignments were completely fair examples of what I might experience on daily basis.

In my experience, you have a lot more than an hour to do it in the real job. And while perhaps 1% of SW folks will deal with in on a daily basis, most will not. I certainly have gone years without needing to use graph algorithms (including even DFS/BFS). And I've needed a BST only once in a decade - and I didn't need to code it, just needed to understand the data structure and complexity to know that "binary_search" in C++ was a superior option than using a map for my use case (even though the complexity is the same).

Some context: At work I recently solved a problem where I had to build a graph (DAG), and do some queries on it mildly efficiently[1]. I did it quickly thanks to wasting time on Leetcode, but no one knew how quick, and people were impressed that I "got it done in less than a day". The reality in most SW work is that whether it takes you a day or an hour to solve this problem, the overall rate of progress will not be impacted.

[1] Mildly efficiently because for the first POC pass, I didn't memoize, and then I got lazy and decided I'd optimize it only after getting a real world data set and seeing if it was slow. It wasn't, even for a decently large data set. The reason? Business logic dictated that the max distance between two nodes did not exceed 8, so from a complexity stand point, you're not increasing the work N-fold by not memoizing, but it's just changing the constant factor.

Somehow, I suspect that in half of the FAANG interviews, had I coded this and argued that the complexity didn't change, I would not get a pass.[2]

[2] Not trying to be cynical: I did argue this for a different problem in a Google phone screen - I had been asked to write some code + data structures to handle an Manager/Report relationship, and the interviewer pointed out my data structure was less than optimal. I countered with "In most companies, the number of employees a manager has under him/her is usually bounded by a small number." Surprisingly, the interviewer gave in.

This comment reminds me of a comic with two vultures who question Mr. Mouse’s claims about the Owl being a predator. It’s not like he ever bothered them. See here: https://twitter.com/nathanwpyle/status/999294987195035649

> preparing for the interview

What type of questions were there?

> I've just interviewed for Google and Meta. In total I went through 8 individual coding interviews.

Curious, were there any questions about what you think of their company ethics, and if you have a problem with it?

Congratulations, there is no chance in hell I would put myself through that.

The problem is when a significant number of companies all cargo cult the same process- that way you can’t even write them off as a one-off bad place.

The style you propose probably has a high correlation with intelligence and adaptability without requiring too much upfront knowledge / experience. It's open enough to allow for mistakes or answers which show intelligence without requiring perfection. Comparatively, LC-style interviews and "do you know the compiler of this language"-style questions are looking more for knowledge than figuring things out on the spot.

Knowledge generally isn't a great indicator of future job performance, intelligence is. There might be some correlation between the two (you probably need a level of intelligence to acquire the knowledge for LC-style questions), but most jobs are layering proxy upon proxy to test something that has little to do with job performance. Being able to fuddle syntax or search basic solutions and mutate, is leagues different from what LC in an isolated environment is.

Then you add onto that the alternatives tend to be even worse. Personality testing works when done right, but the vast majority of interviewers play pretend-psychiatrist and make the entire thing way too subjective to be effective beyond filtering obvious red flags. Take-homes work, but most companies won't compensate for time and there's no framework to show which take-homes individuals have succeeded (kind of like a certification process), making them an incredible time sink.

And then there are always the cases who have little experience yet manage to pick everything up in 2-3 months and become anywhere from average to top performers.

> The false negative is the obvious downside of the coding interview, but I wouldn't say that it means there's no merit to the excercise.

Tell that to the people who are always the false negative.

I notice there seems to be a "No true Scottman" issue here where the critics is stating one problem and the proponent is like "yeah but what if we change it to this". I'm guessing the process at your place is much more tolerable than what people have been complaining.

I wonder at what point does it stop being "whiteboard interview" and just a "few sprinkle of algorithmic questions to test logical skill".

Also when the talents pool is large enough, obviously you can do as many brutal questions as you can and still fill the headcounts with good developers. So I think saying "but it still works" is not very productive.

It seems to me that a false negative is worse than a false positive if the goal is to hire talent. Have never understood the mentality that it is better to mistakenly turn away a high skill individual than to mistakenly hire a low skill one, you're effectively guaranteeing the former to avoid the latter. Would you write a search algorithm that behaved this way?

The question is whether they are actually decreasing the number of false positives. So far I've not really seen evidence of this beyond some mere anecdotes. For such a bold claim, one would expect evidence to be readily available.

Which becomes even worse when realizing "false positive/true positive" is too binary of a definition for a spectrum such as job performance.

And I say this as someone who's in favor of giving people a quick check on whether they can actually do FizzBuzz.

> You don’t seem to have thought very hard about what employers are optimising for.

Have you ever seen a hiring process that's simply not effective?

The undercurrent of this irks me a bit. It assumes a kind of omniscience on part of the employer, and enough time for them to develop a good interview process. I think this is really hard to do.

I think you're generally right. And a lot of the time, employers are empathetic, pragmatic, self-aware and really are doing us a favor when they deny us. Some take into account I-O psychology stuff, too.

I (and others probably) are eager to hear what companies you feel do a good job at hiring.

> Everyone realises that programming interviews produce lots of false negatives.

Let's assume a bad scenario. A company can disregard evidence of competency, like having a portfolio, to basically have a rolling, year round tech Olympiad. To them, it's about fronting as a strong candidate in events, even at the expense of being good at the role.

You can be a pro at pole vault or discus throwing, but a poor carpenter. But there's a mentality out there - with some - that a strong, fit candidate can handle any technical challenge.

Can a chemist be a great chef? I bet, I also bet some could overthink and/or overengineer preparing a simple meal. Does the interview process do anything to check for soft skills, like curiosity, simplicity, etc? Hopefully.

> The goal is minimizing false positives. Failing to hire a strong candidate carries a much lower cost than hiring someone who can’t do the job.

I think some companies - not all - can create a "squid game" contest out of candidates. Worse, I feel candidates can unduly suffer and be humiliated in these cases.

The company gets a superb interviewer, who is no doubt a very intelligent person, but not necessarily one that can adapt to technical reality, contingencies, ambiguities, etc. Some interview processes end up focusing on theoretical and rote knowledge, not integration and synthesis, when they deeply need that. Turnover happens.

But we're back to the beginning, did the company: 1. really know what they needed for that position, 2. interview process check for that? My guess is when you wrote that you were assuming situations where both were true?

But the adversarial puzzles and whiteboard programming are not "the job".

Hire for strengths, not to avoid weaknesses.

Maybe you're the enlightened one here or maybe you're just rationalizing an awful and lazy part of the hiring process for many companies.

Your problem has gone off the rails if that level of performance difference is important.

If the choice is relevant to the task, then sure. If it’s an arbitrary choice then the question is meaningless. It’s like asking “why are you wearing a blue shirt?” when you would have asked “why are you wearing a black shirt?” had they been wearing a black shirt.

> perf characteristics of different iteration styles are often significant to the task at hand.

This ridiculously untrue.

It's also a great example of how dumb current interviews are. Someone like this poster could easily interview you, and now what? Do you just play along or begin to explain how incredibly wrong they are?

And yet correlation is not causation. And a survivor bias is probably in play here. There are so many more factors contributing to the quality of FAANG work and their performance as businesses. Hell, most of them due to their monopoly-like status could easily use any hiring technique and still rake in tons of $$$. To me the streamlined / cargo cult hiring process seems much more a symptoms of the bureaucratization of these large companies than an actual key to their success.

I've been at companies where we receive hundreds of applications per role posted, and one where we receive 10s. I've posted about this before but in both cases the pipeline was the same - initial pass eliminates about 70-80% of the candidates because they're just not worth following up on - think things like IT staff applying for mid level development jobs with no experience, education or anything to indicate they could write any code. Of the remaining we end up filtering out based on comparing applications - this bit sucks, but pretty much stack ranking the resumes and picking out the top 20 or so[0]. At that point, everyone gets a phone screen with a set of questions picked from a list and we proceed through that. The same process works mostly because the size of your interviewing group scales with the number of engineers you hire.

[0] side note - I work in games where we're often filtering resumes for specific roles, e.g. a gameplay programmer or an online programmer. The filtering here is most of the time bucketing candidates based on their experience and sorting that way.

Not OP, but I would love to share some: I relate strongly to the anxiety induced descriptions in OPs text and thus try to avoid these when I interview devs as an engineering manager. Of all the candidates I have interviewed over the years I have never done a live coding interview because it is essentially worthless for me as an interviewer and only serves to discards potentially very clever people whose thought process does not work this way.

Instead I try to find an isolated problem/task we have in our backlog that the candidate should try to implement on their own time, so I can review what kind of solution they will actually deliver once hired. We then go through the solution with them and do a code review as they would go through if they got hired. You then see both the problem-solving skills as well as how they take feedback.

I would recommend checking out DuckDuckGo's excellent "How we hire" guide[0] which describes some of the best hiring process I have come across (albeit it may be too extensive to do in full for some companies).

[0] https://duckduckgo.com/assets/hiring/how_we_hire.pdf

This is really not rocket science. We have implemented in our company.

Step 1: An initial phone interview with the candidate regarding technologies of choice, war stories, etc

Step 2: A small coding exercise that is similar to what we work on. The candidate can solve this on their own time and email us the solution.

Step 3: Review the solution together and ask questions about their methods.

Step 4: Meet the boss & offer.

The coding exercise is only sent _after_ the initial phone call, so we don't waste their time or ours.

The only problem here is that we can't virtue signal that we follow FANG whiteboard interviews and pretend that everyone working at the company can invert a Trie Tree in a 30 minute coding session..

I would like to see the hiring process like this.

1st interview - Go over the role etc, both parties ask question to see if they are a good fit. If both parties are happy to move to the next stage then there is a take home exercise to do.

Tech Exercise - Something simple which touches the core competencies that the role requires.

2 Second interview - Go through the exercise discussing the design choices etc. If everyone is happy offer the job.

I think the probation period should be used to determine if the candidate is the right fit for the role. That way the business gets a much better idea of who it is they are hiring and the same goes for the candidate.

If we have to give Leetcode questions I'd at least like there to be an option to do it by myself, no one watching me type or asking questions if I'm thinking more than a minute. This interview style is so unrealistic and stressful, and turning a normally stressful event to a real horror show. Yes you'll have to somehow see to it that I don't cheat if you leave me alone - which makes it harder, but it will be much more humane.

You can just ask them to talk about code. You can ask them to compare and contrast some frameworks or languages of their choosing. You can ask them to do design tasks, or just talk about the work the company does and let them ask questions.

One of the things I like to do, especially when a candidate is super anxious, is just ask them to talk about a recent project they've done that they like and why they like it.

Interview technical folks at a FAANG and I don't asking leetcode questions. It isn't a SWE role but people should be SWE material. I do ask them to

    * read code
    * extend some working code with a new feature
    * parse some semi-structured text into a data structure

What folks should do is practice coding in front of someone. Writing little exploratory snippets and understand the base library of the language they claim to know. I don't even ask people to vocalize while they are working on the problem. And I solve it at the same time they are and if they get stuck, I show them some of my code.

I had one person start crying they put so much pressure on themselves, I had to take a two week break from interviewing after that.

Many people with genuine 10 years of experience do not care for stupid leetcode question they could obviously solve given some quiet and time.

The idea that everything boils down to answering some intro to programming questions quickly is absurd.

It's is also easy to bullshit a coding interview. The problems solved at interviews are just too small to verify code organization skills.

Great analogy. I think the rub is I wouldn't want to study for and take the LSAT before every interview, and especially not after working in the field for 10 or more years.

Especially if the interviewers make comments like "um how much do you actually practice law?" if you stumble on one of their pet quiz questions.

> The analogs of the traditional coding interview in the law school world are the LSAT and classroom timed memory tests.

How are these analogous? LSAT is the Law School Admission Test. It gets you into law school. It doesn't get you a job as a lawyer. (The bar exam doesn't get you a job as a lawyer either.) And you only have to take it once, not for every job interview.

Aren't those one time things that define the future of your career? Sort of a pedigree being established?

You can go to a top tier university, land a great job for several years, have positive references, and still be required to do leet code for a recruiter... that seems like ongoing gatekeeping, rather than a one-time effort/pass.

Sure but I can get a much better gauge of intelligence from OSS. I’ve seen some of the worst developers I know beat Leetcode because it’s really just memorization not actual intelligence

What was your journey from engineer to lawyer like?

Well I didn’t mean to offend anyone. I guess that just makes you a better technical interviewer than most.

But to be perfectly unbiased, the process would have to double blind. And I don’t know anyone who likes to hire people (or anyone who wants to be hired) following a double blind process.

So at least there is _some_ non-technical bias that gets in the process.

> It’s hilarious listening to people review candidates on third coding interviews. Usually it’s “this person didn’t know this one thing I think makes me really smart”.

Yup. I just went through a round of interviews at multiple companies. In the past I successfully interviewed at a FAANG.

I agree completely.

> This is why I lean so heavy in OSS to review candidates. I can see how they interact with people, and whether they are actually able to build software that _people want_

This.

There is absolutely no need to ask such ridiculous algorithms and advanced equations questions that you will never use or implement from scratch yourself, or asking the candidate to create a formal proof by rote for a entry-level software engineering position or even a senior level position.

Such interviewers who ask these questions have both googled the answers to these questions before the interview and also have never used them practically, but only do this to appear smarter than they actually are. Worse part is not even FAANG asks any of these formal proof questions, but only some certain startups that the last time I have checked, they have failed to make money and have shutdown. Why? Because they seriously thought they were a FAANG company.

Open-source cuts through the algorithms nonsense, saves time and gets a realistic idea of what the candidate has actually worked on in the open. I can ignore the typical hello world, or demo projects but can ask for if they have created open-source libraries or contributions to any serious projects like compilers, OS projects, or any related project to the job description etc.

A take home project is also a fine metric, but asking questions about algorithms and finding out that the interviewer (or the company) doesn't even use them tells you that the interviewer doesn't know what they are looking for.

https://people.wku.edu/sally.kuhlenschmidt/whimsy/oss/ossstr...

It's about speed of solving medium leetcode not coding skills.

leetcode is not coding

Most growth companies have an endless bucket of work.

Somebody who produces 2x at same quality of work is worth 2x the other person, fairly objectively.

It's common to find people who produce up to 10x the average. Anyone in a startup or growth oriented company is smart to optimize for these people.

The most proficient and fast coders often write higher quality code too, in my experience.

People who over index on comp sci theory often are lacking in theory underlying good code. But it's not mutually exclusive. Just that if you have to optimize for one, coding proficiency matters much more for startup or growth type companies.

Exception might be like an AI/ML based startup with a lot of theoretical challenges. Most companies much more business/rule oriented though

> Crappy code

Being able to come up with an algorithm in a reasonable time doesn’t make it crappy. No wonder you can’t pass those.

I mean that's what we do, right?

some people can't even do that.

Anything that anyone can find in a couple of seconds on a search engine is probably not worth an interview question. And that's valid for the interviewer as well don't ask something that you could find by looking at the resume.

You have a task to implement a new product feature and it takes you 1 week instead of 1 day.

In an interview context, I've seen cases where somebody fully understands the logic underlying the solution, yet it takes them 30m to write it, while another person can do it in 5m or less.

Essentially how long it takes to translate from mental understanding to working code

I think this is a great example of the kind of implicit expectations that always pervade these discussions. I had a similar interview in my last job search, and for me it stood out negatively. At the time I’d never parsed any part of an HTTP response in my professional career, so I made some mistakes that look kinda silly if you know what you’re doing, and it was clear that the interviewer didn’t believe me when I explained that this isn’t a problem that comes up often in my field. I don’t want it to sound like I’m complaining, it’s not a big deal, but the interviews a web developer sees as neat and low-prep are exactly the ones I need to prepare extra for.

You miss an important criterion though which is “does my interview has anything to do with the actual work?”, and lc fails miserably. And this one criterion probably classifies lc even lower than take home test or coding questions.

> 1. Take home assignments.

> Meritocratic: good

I'd go with "iffy": because of the "take home" part there's no guarantee it's the candidate that did the work. This can probably be determined with a follow-up conversation during the in-person part, but without such an addition I don't see this format as "good" on this metric.

Ya know, I've heard this repeatedly over the past 15 years, but I've never actually encountered this mythical "senior programmer who can't code." Never worked with one. Never interviewed one. Never met one at a meetup. I think it's just a boogeyman used to frighten hiring managers.

Does the industry even need engineers who can code?

This is a hyperbolic rhetorical question but consider how much of modern work really is CRUD API gluing, and how much of that work is not rigorous engineering, but tedious implementation as per documentation and wading through hastily-thrown together legacy cruft written during previous crunch time.

How much of that work involves novel programming and memorized understanding? Most of it requires study of existing codebases, not given a blank REPL to hash out something new.

OK. What novel problem is the senior engineer solving using a hash table in a coding interview?

Then tell your company to not to reach out to recruit senior & experienced candidates if you aren't sure if they can code from looking at their experience, github, etc. I'm personally very tired of answering recruiters & hiring managers for specialized roles, then the interviews end up being some unrelated CS quiz, mostly from people with less experience than I have.

Nowadays when I interview people, I try and come up with an example that feels like a "real" problem you'd realistically solve on the job, but isn't too hard, usually something involving building an index or some basic concurrency. It's still not perfect, but I do work pretty hard to try and make the problems feel fair and actually something you'd do on the job. I also generally don't concern myself with compiler/syntax errors unless they're extremely egregious, since I think those errors can typically be explained by nerves and realistically won't be much of a blocker in a real job.

I really hate the contrived problems that interviewers pull from HackerRank and its ilk. Even if you could glean useful information from watching someone program a knight hopping around a phone dial pad (a real problem I've gotten, which I'm highly skeptical of its utility), it comes off as pretty irritating to the interviewee. You're not solving little programming riddles on day to day work, you're solving engineering problems, and I think that these leetcode questions don't reflect that.

I think there's evidence of this too; you can study for these leetcode questions and get better at them, so what exactly are we testing? You're not testing the person's experience, you're testing how long they spent on hackerrank the night before, or how many interviews they had recently.

I jumped through the hoops plenty of times. Just because I have to do it doesn’t mean I have to like it, or think it’s a good use of anyones time.

I think one can fairly accurately determine my level of skill from my resume, and thus I don’t think that the leetcode interviews are all that useful. I keep hearing about senior engineers who can’t code, and maybe I’m just lucky, but that really has not been my experience when interviewing, and I don’t think “spent the previous night practicing on hackerrank” really gives you any useful information.

That incredible dearth of junior positions is stranding a lot of new grads and juniors, as seen in discussions like this:

https://news.ycombinator.com/item?id=31537711

You are unintentionally correct, most engineers don't really know their programming language, standard libraries or basic APIs. They find _some_ solution on the internet that's good enough and run with it. Sometimes that is fine, sometimes it is not.

I am indeed asking that as well, given the completely unorganized and nonstandardized state of the industry.

The thing is, the practical field is littered with countless different technologies & endless subjective opinions. We'd need 3,000 different tests to test everyone's practical knowledge, to let them pick between. Oh, you use Go and SolidJS, here's a test for you... and who is going to be able to properly give you a score other than works/doesn't work? This person likes functional style but this person is more OO... the base truth is: code is artisinal & subjective 98% of the time. Most code is soft. There's poor metrics to assess & judge it by. Many many many things work. What we do is cultural, is social, is subjective.

I'm not sure how to make this come through but I'm rather sympathetic. I think practical experience & can-do coder know-how count for most of the marbles. But I also think it's impressively hard to assess by, that most of what we do is just taste & path dependence on the specific techs we've seen that happened to leave good impressions on us: not science, not knowledge, not truth: most coding, most practice is almost entirely happenstance.

But again I want to empathize. Because I think it's cruel to disregard engineers that have seen so much. But I also think you're hyperbolizing overmuch & discrediting the discourse when you make your argument so sharply pointedly. Because there's some truth, absolutely, but it's also no-where near as lopsided. For example, you talk about fresh/current undergrads at Stanford. But this is a pretty regular, basic CS path that any university student should be familiar with. I only had two undergrad classes that really talked to algorithms (Algorithms and then Data structures; OS kind of somewhat), albeit there was some computational thinking already in play at that point. Understanding complexity & algorithms is understanding the basics, it really is. If you can't see time flowing, don't know the tally & impact of the work you're asking for, you're lacking knowledge to avoid a lot of bad decisions. This isn't made up fake shit taught only to the elite.

What's alluring about this system is that it has some hard facts & theory underneath it. It's not just opinion & engineering pop-culture. The answers don't change, the rules don't change, the material stays the same, and it's all rooted in being able to analyze and understand problems, rooted in comprehension. Being able to see & understand & analyze, being able to apply some basic principles: this is a pretty sure way to find people who will be able to Not Mess It Up, who are capable of looking, assessing, & navigating through scenarios computationally.

And last and perhaps most key to me: it's also material that someone with even a modest bit of aptitude can cram for. Pick up a book like "Cracking the Coding Interview" and you can semi-accurately reproduce the output of four years of undergrad in two or three months of occasional light practice & trying. Projects like Leetcode can get you the experience. You may feel bad that what you want yourself to be measured on doesn't count here, but to people trying to go get hired, it's enormously helpful to them to have well defined expectations, to have specific kinds of tests to expect & be able to prepare & study for. I see so many protests that it's not fair, that it favors only those with the luxury of time, but those views to me massively undermine how wonderfully vastly accessible it is that there's a known, well-described, well-supported subject-area we can study. The industry overflows with things to learn, study, & know, but here's something concrete & specific, which undergirds it all, which alone might not determine your coding skill, but which does indicate you at least have some raw basic intellectual capacity to go understand problems put before you & apply some sensible computational thinking to tackling them.

There's tons of things not tested for, but by having a well defined, technology-neutral set of computational thinking tests, based on actual science with objective, factual answers, rather a much wider corpus of future/current/legacy pop-engineering-of-the-day built on nothing but wishy-washy subjective opinion, I think we achieve one of the best possible wins we can against an enemy we both hate: class-based discrimination, cruel tests that reject outgroups.

> How many people if asked to do this on a job rather than an interview a year later would then go and implement it from memory without double checking they didn't forget some edge case on stackoverflow?

An excellent question.

If the interviewer is to ask such algorithms questions looked up on leetcode, etc, then they must be prepared for when the candidate asks if they also use it themselves on the job. If the interviewer admits they don't use it, then they also admitted that they don't know what they are looking for and really are wasting the candidate's time.

If it were me, I would look at open-source contributions (no hello-world or demo projects) where that is enough proof for me to evaluate an entry-level candidate and cut through the algorithms nonsense and ask questions to the candidate based on that which will save everyone time.

Yes, leetcode skills don't help every day. But I definitely enjoy & respect more fellow coders who have computational thinking skills, who can break down problems, who understand costs.

That this isn't the bulk of what we do doesn't change the fact that these challenges do test computational thinking acuity. Having the ability to see & speak computationally is a good skill, one that connects our day-to-day abstract practice with actual real processes. Being able to break down problems & analyze how to tackle them shows an objective ability to assess & work through problems. I want to work with people who can be clear, who can model & explain & step through situations.

And these skills are, generally, learnable, and relatively quickly. I disagree that these abilities fade, but yes, some re-familiarizing & re-training is probably important, especially because, as you point out, the sample size is indeed often N=1, and that's pretty wild.

Some people are ready to trade dancing for a lot of money and a resume that will open them the doors of where you really want to be.

Do you have any suggestions for a simple online coding portal? I know some companies have used a simple Google Doc for this reason, but that seems far from ideal.

If it takes months of practice, probably one didn't learn very much in programming classes. But you'll learn a heck of a lot with those months of practice, and that'll pay off handsomely with that $250,000 offer.

I am not a civil engineer, but I can imagine a situation where a wooden bridge could be used as a stopgap to pull off a more tricky construction.

Maybe not in the 21st century however, and I don’t think your analogy is perfect.

It might be more like wanting to hire an architect who is also great at civil eng. The skills definitely synergize and can help prune the architect’s search space of good designs, but such a candidate is probably just going to go for an engi job, and lean on their architecture skills there: for example an algo dev that can do mock-ups.

3 problems I can see with this

a) Its like saying a Tobacco company has a paper saying smoking is healthy

b) Google measures things that matter to them. The operate on a scale that 99% of companies aren't. Their predictor of success only applies to companies of their scale.

c) What is their definition of success? And how does it apply to smaller companies less than 1000 employees?

Did they hire people who failed their interviews in order to see how well they did?