How to hire engineering talent without the BS
jes.alAll of these "this is how you hire" posts are utterly worthless without data afterwards to prove that their selection process results in high performance. Does any of these posters actually correlate interview scores with performance reviews? No. It's just people very confidently announcing their own biases beliefs without any reason except their overconfidence.
At least the reason why we have Leetcode questions is because Google did the research and came to the conclusion that those are were good at algorithms ended up being more successful at Google, and THAT'S why we are all suffering through LC. But now that LC has been gamed, I would love to see what the results are as to what makes a successful interview.
I think part of it is, within software engineering, what is “high performance”?
Is high performance making economically valuable contributions? If so, there is huge inherent inequality between the economic value of projects, so the metric is worthless for judging engineering talent.
Is high performance getting good performance reviews? In my experience performance reviews are wrought with favoritism, ageism, and probably other subjectivities that create a very low signal to noise ratio, so that metric is worthless.
Is high performance someone who is good at writing software? There are engineers I know who write very good software but lack the decision making or vision to do so in valuable areas. Thus, another metric is worthless.
The reality is there is no general “high performance”. There are smarter and more motivated people, sure. I think testing for general intelligence, personality type, etc., are valuable. But I also think a lot of high performance is about finding someone who is a good fit for the organization in terms of the individual’s experience and skills and the organization’s needs.
Favoritism can be reinforced when assigning projects:
- assign low effort, high impact, high visibility, low risk projects to the most favorite engineers.
- assign high effort, low impact, low visibility, high risk projects to the least favorite engineers.
Favoritism in toxic orgs is the reward for "loyalty".
Another way is by reorganizing teams, assigning favorites to a team just before the completion of a project. Or assigning undesirable employees to a failed project.
Yep. I got a taste of that in my first gig, where the guy who always attended the "optional" office twice per week ended up getting the high profile project that drives the company, and they didn't even tried to hide it because they switched the team assignments a week before it began.
> The reality is there is no general “high performance”.
I've heard that claim times and times again.
Often it's from companies believing in "Best Cost IT talent" or whatever the current buzzword is on LinkedIn. Chances are, they never even worked with an SV caliber dev.
We've all seen engineers that are simply running around in circles around others. Just think of John Carmack. If any of these organizations managed to luck out by hiring someone from, let's say Apple or Google, it would most likely change their beliefs that there's no "high performance".
if performance reviews are worthless why do people have them at all?
if they are regularly practiced at a company, this means the company puts enough faith in them to overall promote those who deserve it and to decide peoples' and company's fate based on that. Which then makes it quite a decent interview outcome metric.
If on the other hand performance reviews are as bad and worthless as you claim, they should be stopped completely, and everyone running them should be instantly fired as a bunch of ageist favoritist pigs who are scamming the company.
As a dev, high performance is velocity, number of bugs and cordial behavior. Additional parameters need to be considered for more senior roles like ownership, cross org influence, mentorship etc. It’s not rocket science.
A few years back I worked with two engineers who sat side by side, over the course of a month the one girl fixed one bug, while the guy next to him did 22. Having worked with them both I believe they were actually about equal despite the numbers. The girl randomly drew a complex bug that was hard to figure out, while the guy got a bunch of easy ones (including one the is not spelled t-e-h)
This variance is why it is so hard to assign objective numbers.
>number of bugs
Bugs are not created equally.
So anyone who puts in lots of bugs in the first place, will look great in the numbers, because they were so great at fixing all those bugs.
And is Google itself effective? We all copied Google but do we have their problems? Do we need the type of engineers they needed?
Google is beginning to show its age, I’ve seen a number of past coworkers get hired there lately that weren’t effective at their job, but I’m sure they crushed some Leetcode.
I find it ironic that Google’s search results have been gamed to the point that they’re about as useless as their competitors were 20 years ago but companies are still cargo cult copying a hiring process that has also been gamed to the point of uselessness.
At this point the whiteboard gauntlet seems more indicative of a lack of critical self evaluation than anything, “we only hire the best because we’re the best rah rah”. Miss me with that shit.
There is an implication with this post that Google is dumb or unskilled, or stuck in a cargo cult, but that ain't the truth.
Google's search results aren't because they suck, they're doing it to get maximum money. This is about getting maximum cash for minimum overhead. The Goog is a publicly traded company, they're about selling you the crappiest product at the highest margin. Until someone can come and seriously eat their lunch in search, it will never get better.
To paraphrase Cory Doctorow, "en-shit-ification" is explicitly the point.
They hired the best and the brightest not to make search better, but to find ways to optimally increase the stock price; good products are entirely tangential to that.
> Google is beginning to show its age,...
Yup. Goodhart's law strikes again. I am curious as to what the next as-of-yet-ungamed effective hiring metric is.
Maybe analytical hobbies. Hire anyone into lockpicking, rock climbing, or music theory.
I think the last couple decades of software suggest that all these indicators, if adopted by companies that significant numbers of people wanted to work for, would quickly become checklists of things to do/fake.
See also: college admissions. (No, that kid probably didn't do that volunteering activity because they wanted to help people, but because they were told it was a checklist item, as they were coached through college admissions. And they probably didn't learn anything, and they were coached on what to say about that in the essay. And, normally, actually productive volunteering probably wouldn't have looked like each one of those kids starting their own duplicate initiative so each could spin it as demonstrating leadership. And it's more an option for kids from families well-off enough that the kids didn't have to work jobs that would contribute better to their household income or college expenses. And don't get me started on how the well-to-do decided that "travel" would be a plus on applications.)
Next level: You take interview vacations, where you work 3 days for every final company in your interview round. It’s an exchange: You check the cushions on the sofas of the HR ad, they check that you can actually take a bugfix to production.
Then they give you your real chair.
Terrible next level, that's unpaid work and it's already a thing in interviewing at some unethical companies.
This doesn't have to be unpaid.
Nobody said unpaid.
Even so, it's probably not going to be super-effective, unless each workplace is willing to have enough useful small things that do not require knowing substantial chunks of the existing code base.
It is probably realistic to expect someone to write something useful (although possibly small) in three days. It is less realistic to expect someone to write a useful component integrated in a large system that they have to learn, in three days.
They need to be contrived bugs. Unpaid work is illegal in the US, and most other 'civilized' counries. Plus you don't want your competitors to send someone to interview just to look at your code.
He said "work", not "unpaid work". Pay that work.
>Unpaid work is illegal in the US
Is that actually true? It's probably uncommon in tech but I understand it's fairly common in creative fields and--while it was a long time ago--I worked a couple unpaid internships when I was in high school in chemistry labs.
The trick is it can't be work, only work like things of no value. You can have unpaid internships so long as you are getting training and not doing valuable work.
A railroad engineer intern can move cars around the yard all day, assembly trains, and other such things. However if they move the car to a loading dock where it is loaded/unloaded that is valuable work and must be paid. If the train they assembly leaves the yard for the next that is valuable work (unless they all make it back to this yard unused).
Most companies choose to pay interns rather than figure out how to not break the rules. A few have a separate intern program where they carefully figure out tasks to assign interns for training. If your intern program did valuable work you should talk to a lawyer.
Surely there is some economy of scale here, where a standardized first round interview'taking a bug fix to production' fits multiple companies. Noone should have to do more than one technical interview
I'd prefer switching metrics every decade to never having a good hiring system.
or introduce some level of randomness to dull the effect of the metrics
"We only want to hire lucky people, so our hiring council only looked at your resume if everyone flipped a coin and all landed heads"
It worked on Ringworld.
Introduce randomness to see if there are other metrics that might be signals worth considering.
Makes me think, the Real Book might be musical equivalent of Leetcode..
Why not skill at writing 8-legged essays[0] instead? Either way, you're so far away from anything that maps directly on to "is this person likely to succeed as an X level software engineer at Y company" that it's all pretty arbitrary.
Not to mention, you'd have to have a precise and well defined notion of what "analytical hobbies" are. Preferably one that doesn't carry an inherent classist[1] (and maybe even racist or sexist) bias. That part alone seems pretty tough, forget the actual definition part.
---
[0]: https://en.wikipedia.org/wiki/Eight-legged_essay
[1]: https://hbr.org/2016/12/research-how-subtle-class-cues-can-b...
Sounds to me like a monoculture filled with the sorts of people that I just…wouldn’t like working with.
I assume that's a joke? Sounds like going back to manhole covers.
I honestly don’t know if I was joking or not but the breadth of replies is interesting.
> I’ve seen a number of past coworkers get hired there lately that weren’t effective at their job,
That's the truth for me, I was shocked some of our worst* interns went on to work at Google after the summer.
* Unproductive, slow, needed lots of hand holding
Probably ADHD, often makes you unproductive and slow when you aren't in high stakes situations. There is no interview to weed out such people since the problem isn't in their understanding, intelligence or skills.
Edit: I have ADHD, I couldn't do work for shit in any job, but I could ace every interview. Then I got a diagnosis, got pills and became a star employee. This is absolutely a thing. I don't see why people would downvote this.
How is laziness differentiated from ADHD?
Too be clear, I’m not accusing you of being lazy, but your description seems to apply to me and I consider myself lazy.
I did consider myself lazy as well and that is what I told people, but then I got pills for ADHD and suddenly I wasn't lazy any longer. So yeah, maybe you aren't lazy and you just have ADHD, or maybe you are lazy, I don't know, but it is worth checking it out. It changed my life, if being lazy is a huge problem in your life I'd suggest to look into ADHD more and maybe try get a diagnosis if it seems to fit.
Interns are supposed to be slow, unproductive, and need lots of hand holding! They’re fucking students for fucks sake.
An intern's productivity sort of lies in "how fast they learn". Can they pick up things by just being pointed into the right directions or do they need everything spoon-fed, in small doses, with repetition, to then still get it wrong?
At least in our place, we do not really expect them to be productive in the usual sense, but they definitely have ample opportunity to give us a feeling of their place on the bell curve.
(as an anecdote: We currently have some 17 year old apprentice which puts some of our seniors to absolute shame, myself included. The amount of motivation, learning hunger, quick thinking and ability to ask the right questions is really astounding, especially given that he essentially just started and has next-to-no education in our area to speak of as of yet)
Really? Not the parent poster, but have observed the same thing.
All interns need handholding but some need 10x more than others. And yeah, they often end up doing another internship or so at a faang.
To be cynical, there are no dumb questions, only dumb people. Some people ask the same question over and over, and others have different questions, rather rapidly getting to very interesting questions.
There are many dumb questions. Asking me about ancient Chinese history for example. (this is a particularly dumb question to ask in the context of this thread where it is an off topic question asked to someone who knows very little about the topic)
Wouldn't this be a natural case of "check Wikipedia"?
Curiosity and motivation do not require experience.
We had some awesome interns, very interested in learning and working! I'm talking about those who weren't and surprised Google (and Facebook) hired them soon after.
So how is your interview process better than theirs, since you keep getting these lousy people? I saw many worthless people in the places I worked at before Google, then at Google I didn't see anyone close to that bad, so to me their process seems to work. They were very open about their data around hiring and scoring hires as well, and that data was very clear that the interview they arrived at was the best interview, they tried and experimented a lot and arrived at it.
Alternatively maybe those people were just unmotivated to work for you and spent their internship preparing for Google.
HIRE THE BEST RELIABLE AND TRUSTED IOS AND IPHONE HACKER FOR HACKING SERVICE ALSO UPGRADING OF RESULTS AND HACKING OF NEARBY MOBILE PHONES, dont miss this opportunity !! CONTACT.: techspypro @gmail com
I work for a big tech, and I have yet to see any evidence that leetcode-style interviews lead to better hires. If that data is being tracked anywhere, it's kept very secret. In fact the metric that recruitment uses for success is "inclined rate". It's easy to see how that is a lousy metric: I can just say yes to all candidates, but that doesn't mean they will do a good job.
I think it has to do with headcount requirements once you're Google scale too. Leetcode style interviews become yet another "process to avoid disaster at a big company". Disaster here means hiring a bad engineer and then having to fire them for performance reasons or whatever. Google's whole concept of hiring committees like it was graduate school also reinforces this. They probably do this to minimize nepotism at the hiring stage.
All these processes become items on a hiring checklist, but to your point, it doesn't guarantee hiring a great engineer. Just one of those tragedies embedded alongside other big co bureaucracy processes.
> I work for a big tech, and I have yet to see any evidence that leetcode-style interviews lead to better hires.
I worked for Google and at least back then everyone working there could read the studies showing that the interviews worked. The effect was very clear, with there being no upper bar where the interview score stopped mattering.
If you haven't seen any data then you just have a set of anecdotes, so your position is very weak. I have seen the data, but a person like you who is convinced it work wont believe me of course. But I'm not sure why Google would lie about that, or why they would keep the interview format for so long if it didn't work. The old article talking about Google interviews talked about what they did before they arrived at this format, what they do today is the best interview according to their data.
That's a very judgmental reply that also misrepresent my comment.
I don't work at Google, so I cannot speak for them. My company does have studies and experiments around interviewing, but again, the metric they use for success is "inclined rate" e.g. doing ABC leads to x% increase in inclined rate. I haven't seen any study linking interview outcomes with on the job performance e.g. yearly performance rating or promotions. In fact many people have requested this data over the years.
> If you haven't seen any data then you just have a set of anecdotes, so your position is very weak. I have seen the data, but a person like you who is convinced it work wont believe me of course.
This comment seems quite rude, and again, it misrepresents my position. I didn't say LC style interviews don't work, I'm saying that I don't know, because I have seen no data to prove that they work better or worse than other alternatives.
Also the landscape has change, and what might have worked 5 or 10 years might not anymore. Tech interviewing has "professionalized" in a Goodhart's Law-esque fashion. People have become very dedicated to game the process, which might affect the quality of the outcomes. But just to be clear again, I want to see evidence one way or another.
> I haven't seen any study linking interview outcomes with on the job performance e.g. yearly performance rating or promotions. In fact many people have requested this data over the years.
But that is exactly what Google did, they linked the interview results to later performance evaluations and that is the correlation I'm talking about. The better they scored on the interviews the better the performance reviews they got later. The correlation was very stable and didn't just appear at low levels.
Your company might not have done such a study, but Google did.
> Also the landscape has change, and what might have worked 5 or 10 years might not anymore. Tech interviewing has "professionalized" in a Goodhart's Law-esque fashion. People have become very dedicated to game the process, which might affect the quality of the outcomes. But just to be clear again, I want to see evidence one way or another.
That is true, the data I saw is nearing a decade old now, but at least the data back then showed that it worked.
> The better they scored on the interviews the better the performance reviews they got later.
One possible explanation of this is that people who did well in the hiring process were valuable employees. Here's another: people who were good at gaming an evaluation system run by Google were good at gaming an evaluation system run by google.
So his position is weak but you’re not in the strongest position either.
Basically, the same depth of evidence (maybe less) that saturated fat was causing all this heart disease, so let’s keep everyone on very low fat diets.
I just remained unconvinced that we know anything at this point. Google is allowed to do what they’re going to do and to do their own studies and make their own decisions based on that data.
However, that study is not a normal lock for the rest of the world and people SHOULD be questioning it, even with anecdotal evidence, at least till we have enough studies proving their point in different situations and companies.
I agree, we don't have enough data. And a common mistake is extrapolating too much from limited data, but here's the most important data point and why we continue to use these practices: these companies are worth trillions of dollars and they hire this way. You want the industry to change its practices? Find an equally successful (impact, market cap, innovation) company that hires differently in software.
I haven’t seen this mythical closed data, but I have used Google products, so I’m inclined not to believe that whatever they do leads to acceptable results.
What is inclined rate?
"Inclined To Hire", meaning someone who either interviewed the person directly or reviewed their hiring profile and all the interview feedback is saying yes, we should hire this person.
Basically the equivalent of the Roman emperor thumbs up/thumbs down except it's a small group and you need a certain number of thumbs up to move forward. The number of people you need Inclined to Hire differs from org to org that use this method.
> At least the reason why we have Leetcode questions is because Google did the research and came to the conclusion that those are were good at algorithms ended up being more successful at Google,
That's just part of the process, though. They (and many others) have coding questions, system design questions, and behavioral questions. Taken together, they answer the questions of whether you can write code, design/build/contribute to a coherent larger system, and are someone who can be effective at working with others in a company.
Over time, there's been a massive (IMHO) over-index on the coding portion, since it can usually result in an objectively correct or incorrect answer. With massive numbers of applicants, companies really don't lose out on being over-aggressive with cutting people based on these things. As they tighten their hiring belts, I think this will get worse.
Yeah. It's absolutely rational for companies/universities/etc. with a bigger pipeline than they know what to do with, to establish an objective (whether or not wholly relevant) and easy (for them) high bar even if it means passing over people who could have done a perfectly good job and even a few who could have been amazing--so long as most of the people they do let in are pretty good.
> posts are utterly worthless without data afterwards
Same goes to any subject where a multivariate analysis is almost impossible due to the lack of control over your experiments. I seriously doubt that anybody has any data that could prove anything.
As an interviewer who did this for many companies for a decade, I hire for attitude. I can teach anybody hard skills (software engineering, sql, python, etc.) with enough time but I can't teach attitude.
A person who I can trust because the persistent drive towards goals, can deliver a lot more than a naturally talented programmer with shitty attitude. This is based on my anecdotal data that I accumulated over the years.
Funnily enough this is similar to what the Seal Team figured out about recruiting in a completely different field independently.
Actually I remember Norvig saying that they found no such thing - scoring higher in interview panels did not correlate with performance on a job. The only positive correlation they found was gpa
+1
If I remember correctly, it was actually slightly negativity correlated with work quality because extremely talented competitive programmers would smash the interviews but write competitive programming style code (aka not enterprise quality).[1]
1) https://catonmat.net/programming-competitions-work-performan...
no wait. this is about live competitions, not about job interviews.
the interview is about the interaction on a CS topic. how do you express yourself, verbally and in code? that's not: can you crank your the highest performance solution in the shortest amount of time.
> At least the reason why we have Leetcode questions is because Google did the research and came to the conclusion that those are were good at algorithms ended up being more successful at Google
Isn't the story more complex than that? I certainly didn't just get leetcode questions when I interviewed there, and when I asked, the interviewer said the group hadn't been happy with hires that only knew algorithms.
When was that? In 2008, the candidates who were good at leet code kind of questions were generally unusually smart and skilled: nobody at the time was grinding these problems to prepare, so people who could offhandedly solve them without any special preparation were extraordinary. Now, 10-15 years later, everyone is grinding Leetcode, and as a result, being good at it is not indicative of all that much. When a measure becomes the goal, it often ceases to become a good measure.
I interviewed in 2007-2009 and there was a lot of preparation as well. It wasn't as widespread as later, but anyone that did 'programming contest' style problems had a huge advantage at interviews.
I had something like an 80% offer rate because of this. Ironically I didn't actually get a Google offer though (not sure why, I did solve the problems).
Nobody was literally solving them offhand in the sense that they had no background in algorithmic problem solving. Those who didn't explicitly "prepare" were the people who liked competition programming and solving these problems for fun in their spare time. Though I'd agree those candidates were probably better quality on average than those who just grind problems solely for the interview
Yes. These days onsites have at least three categories of interviews: leetcode style, system design and behavioral. For L5 level (senior) and above, the latter two tend to weigh more heavily.
It's also more common now to have two coding sessions, and even two system design ones, under the rationalization that it reduces interviewer bias.
Google explicitly links to leetcode/hackerrank/etc as suggestions for practice resources in their pre-interview email, implying that practicing artificial DS&A quizzes is now table stakes.
I would be curious as to whether these changes have been data driven, or whether the bar is being increased purely out of necessity to weed out candidates more strictly.
I certainly have seen my share of candidates that do fine in leetcode sessions but then go on to fail in the other sessions for various reasons that allegedly correlate to work performance.
1. What types of questions did they ask instead?
2. Did you get an offer? Did you accept?
> Google did the research
Google also operates with their own sets of constraints. What works for them isn't necessarily what works for different types of companies.
And because they pay a shitton of money, they have a constant flow of candidates and can afford to miss out on many great ones.
almost all of Google loses money and makes products that get cancelled. so if you want to lose money and make products that nobody uses or gets cancelled, then i guess leetcode is a good idea.
That’s not a development fail. They need better product people maybe? Also, google has money to spend so the bar for a launch may have been much lower in the past.
thats my point. what if leetcode tests filter out everyone who is good at making useful products or services, in favor of people who can memorize how to solve puzzles.
The people who make those decisions aren't doing leetcode tests. Do you think that Thomas Kurian passed a leetcode test? Of course he didn't, the people who make business decisions are in another pipeline that is just a normal management interview.
so engineering is completely separated from management?
Google is one of the greatest companies in history, measured both in technology and business performance. Sure, not everything is perfect.
When did they start doing leetCode style interviews? LeetCode looks to have been founded in 2015; of course, though, that doesn’t mean Google couldn’t have had a similar in-house format that predates that founding…
But it is hard to say how much leetCode style interviews helped. Post-2015 Google already had quite a bit of momentum.
My understanding is that LC style interviews largely replaced "brain teaser", "how would you move mount fuji?" type questions so at least you're testing CS concepts and basic coding ability.
Cracking the Coding Interview came out in 2008 and had a short section talking about the Google interview process, which I believe was system design + a coding problem that would probably be close to a modern LC question.
I am cusrious what you mean by "LeetCode-style". My understanding is "present a problem, only take the resulting code, judge that".
I did a fair number of coding interviews (for SRE positions) at Google (I don't actually know how many in total, but 25-30 is probably a safe guess) and, yes, it started with a small problem that should be solvable in well under half the interview time. I only used problems that passed my "is this fair to the candidate" screening (look at problem, try to write a working solution, if that takes more than 7 minutes, the problem is not fair).
The value in the interview comes from (a) listening to the candidate narrating their thought process during the coding, (ii) discussing the solution with the candidate, sometimes in terms of complexity, sometimes in terms of correctness, depending on what is on the board, (3) refining the code written (add more functionality, change functionality).
For a language I knew, I tended to overlook small syntactic mistakes (a whiteboard does not have any syntax highlighting) and if there was any questions about library functions, I would give an answer (and note down what I said, so that would be the standard used when judging) and the bulk of the score came from the discussion about the code, not the code itself.
If that matches what you mean by "LeetCode-style interview", that's certainly been in place since at least 2011. If it does not, things may have changed, but at least the aim wit ha coding interview back when I was still at Google was less "get some code judge, on the code" and more "get some code, judge the discussion about the code". It is also entirely possible that interviewing for SWE positions was different frmo hiring for SRE positions.
They always used to be famous for dumb questions that made you think like how many golfballs can you fit in a school bus, why are manhole covers round etc.
Android code is such a good example of that hiring process works in practice, it is just brilliant piece of art.
Where would we be without such hiring processes that managed to have such set of genius work on Android.
/s
It's a pretty successful software project.
I really wish I had the reference, but I did the "paid trial" for a week for out interviewing process (startup, 40m evaluation) based on an article that said that standard tech interviews gave you an 18% predictability and that paid trials gave you about 72% (it was only one study).
I can tell you that at our company we definitely dodged candidates here and I was very happy with everyone we hired (but you could call that bias, as I was the hiring manager -- but I also worked withe veryone I hired)
I can imagine that works fine for fresh-out-of-university candidates but I can't imagine a time in my career where I could (and would want to) take 5 days PTO for a job interview.
FAANG-like hiring has the question "What info do you get from the 5th interview that you didn't already have after the 2nd?" - for the paid trial, what info do you get by Friday that you didn't have by the end of Monday?
I agree it's much harder with someone further in their career (and we worked with candidates to figure out what made sense, either shortening the trial, allowing other time to be included, etc.)
To answer your last question.
On Monday, it was had to fully tell how someone interacted with the team (there was a particular incident where a candidate ended pushing some extremely strong politics on the rest of the team on Wednesday). I was okay with it (I think people should be okay to express opinions), but the rest of the team was not -- and I value the rest of my team (ended up making the decision not to go forward as they weren't comfortable with the hire).
Similarly, when they gave an estimate on Monday for the work they could accomplish on Friday (but failed to do it), depending on their reasoning (did they just grossly miss-estimate, or were there legitimate reasons, are they learning from errors). They had time to keep us updated throughout the week, to readjust, etc. All things that you typically learn with time.
This assumes that performance can be meaningfully measured. Which is incorrect, in my experience.
> At least the reason why we have Leetcode questions is because Google did the research and came to the conclusion that those are were good at algorithms ended up being more successful at Google
[Citation needed]
(Or you are doing what you say others are doing)
> worthless without data afterwards to prove that their selection process results in high performance
> Google did the research
Did they? Where is the proof? Pot, Kettle, black
Were the new hires more successful or were the hiring managers more successful?
Note the hiring managers are the ones picking the criteria.
My guess is a dev who passes leetcode but fails to produce is not seen as reflecting upon the hiring manager. It doesn't matter if leetcode correlates or even negatively correlates with the success of the new hire; what would matter is not being able to blame the hiring mgr.
> But now that LC has been gamed,
How was it gamed? I wasn't aware of anything. Is this about LLMs being able to solve some of the questions?
It’s that it doesn’t really test your depth of knowledge of CS topics, it more just tests that you’re able to memorize categories of problems and how to solve them.
Sure you need to understand some basic CS to pass a LC question, but it’s more about recognizing a pattern and having memorized the proper algo for that pattern.
If you were taking an algorithms interview 20 years ago, you’d have to truly know a huge breadth of CS knowledge. But now you can just memorize problems for a few months and never learn the true meaning of “why” a tree is the best data structure in x situation.
It was “gamed” when people learned that you could pass the interviews without truly understanding the material, if you just memorize the rules of the game and can recite them when asked.
That doesn't mean it's not testing for the right things. Pattern matching, memory, and understanding are all attributes that help be an effective software engineer.
It's also not as easy as you describe - just memorizing patterns. Maybe for the easy-medium questions that ChatGPT can solve, but Google stopped asking such questions a decade ago. The algorithm questions that all the tech companies ask today require real problem-solving.
I think it's only gamed in the Goodhart's Law sense. The original intention of hard CS brain teaser questions was to filter for people who can reason through the questions on the spot. Now because everyone knows the questions and answers, the filter is probably more likely to net you people who put in the memorization hours than high IQ folks.
> Does any of these posters actually correlate interview scores with performance reviews?
Big assumption that performance can be reviewed and scored in some objective way, or if it is that this actually happens. I've seen lots of review processes that pretend to be objective, but absolutely none of them actually were.
> why we have Leetcode questions is because Google did the research and came to the conclusion that those are were good at algorithms ended up being more successful at Google
Did they? I remember hearing about a research they did and it wasn't so clear cut to its advantages as a "job success predictor"
coding competitions are no predictor.
oral exam style algorithmic interviews are.
Also - does it really really matter? 101% worker vs 95% worker but one is an insufferable pr*k who makes everyone else 75% workers ...
Everyones approach looks like it works because if everyone is between a 90-110% worker and the team gets along and works together you will get solid results.
>Google did the research and came to the conclusion that those are were good at algorithms ended up being more successful at Google
They could have been fooled by randomness, like many "intellectual" types often are.
As a startup CTO that has done 15+ years of this, I believe very strongly that your hiring process relfects deeply on the core software belief system you hold on. That's why bureaucratic/non-engineering organisation will tend to over-emphasize references and tests, big tech will over-emphasize CS40021 style exercises and whiteboarding "shame on you", and the rest of us, other stuff. My advice for job-seeker: look very deeply why they ask things during the process and you will be able to fairly predict your future there. Make sure it matches you needs and wants. For the process-builder: are you sure those deeply-held beliefs are filtering what you need or is it filtering what you want...?
I’ve been seeing and increasing mismatch in the challenges companies present you with and the role they actually want filled.
At this point, the cargo cult of Leetcode is often more of a hinderance to startups. Everyone wanted to follow Google thinking it’s the best system without asking themselves if they have the same problems or are in need of those kinds of people.
What's especially ridiculous about imitating Google's hiring practices is that the value in Google's search/ad business was largely built by people hired there before these interviews become the norm. They hired through social networks, like, hey, you and X who works here had the same PhD advisor.
There’s nothing ridiculous about it.
Second approach is just not scalable in the long run. And you need someone to do grunt work.
The ridiculousness isn't that Google has changed their hiring process to reflect their current state, it's that other companies are copying what Google does now with the hope that it will allow them to replicate the success that Google had 20 years ago.
If your premise is "We look a lot like Google circa 2015, and I really like what they've done since then, how did they do it?" then it might make sense to replicate their hiring.
But if you want to achieve what Google achieved between 1998 (founding) and 2004 (IPO) then asking what they do now (or have done over the last 10 years) isn't particularly relevant.
Absolutely agree, and smaller companies can even afford to fully disclose their core values and non-technical expectations towards the candidates to make sure the process is a win-win for both parties.
We update and share these points within the team and with the applicants even before they make the first contact with us:
https://stratoflow.com/our-recruitment-and-onboarding-proces...
Nice list. What does "Distanced from themselves and their source code." mean?
The code has bugs and might cause a new pandemic. So they follow social distancing.
That’s all nice in theory, but new college grads want to make as much as possible. They don’t do that by “arguing that gravity shouldn’t exist”. They do that by playing by the rules as they exist.
That means “grinding leetcode and working for a FAANG” (c) r/cscareerquestions.
I am 25+ years in the industry. But if any new college grad asks me for my advice. That’s what I tell them.
I definitely wouldn’t tell them to take a chance of working for any non public company hoping their “equity” may be worth something because they heard that early engineers at Uber struck it rich.
No. I don’t know how to say it more clearly.
The vast majority of people I’ve hired want to be happy, and I have completely qualitative conclusions from having hired such developers for decades: It works well for everyone.
If you want a job at FAANG badly enough to suffer it, go for it. But don’t ask me validate “that’s what everyone is doing”. It’s not — you’ve been sold a bill of goods, or you’re greedy. Either way, you be you.
You realize you’re replying to someone who is 49 years old on their 8th job? I’m far from a naive college grad.
But I bet you dollars to doughnuts that if I told any of them that they could double their compensation by “grinding LeetCode” they would trade the “equity” in a private company that statistically won’t amount to anything for real stock being in their brokerage account every six months.
As far as being “greedy”. Are you running a charity?
Yea, this is an interesting thing. I see plenty of smaller companies still doing the shady stuff that FAANG does and then trying to offer 50% the salary. I'd maybe take that if I was trading money for feeling like I was doing something Good, but if it's just "which privacy violation do I want to support" then you don't really have an advantage to offer me to offset paying me a lot less.
And, if they want to live in any of the same locales where FAANG people happen to live, show them `levels.fyi` for FAANG, and tell them those are the people they're going to be competing with condo/house offers and apartment rents. (Well, in addition to the other parties snapping up real estate.)
Students might know starting salaries, but maybe not know just how huge the income difference can get in a few years. Nor understand how adversely the relative difference can affect their quality of life.
Doesn't that make your sample size very small?
> If you want a job at FAANG badly enough to suffer it,
I mean, you have to agree, for a lot (if not most) people, the ROI is massive.
>The vast majority of people I’ve hired want to be happy
You can't possibly know that for sure. Are you making that judgment based on what candidates tell you during interviews? Because the meta-game of a job interview is literally just figuring out what the interviewer wants to hear and then saying it.
Having worked at a couple big tech companies I’ll say money isn’t everything in life. I’ve been much happier working at well paying startups with a great group of people making half the money.
Would you say the same thing coming straight out of college with $0 in your bank account, student loan debt and no experience?
As I said in another reply, I could afford to ignore BigTech recruiters at 45 making about as much as a returning intern makes where I work now. I already had the big house in the burbs, retirement savings, a family and a son graduating from high school.
I would never tell a new college grad to ignore the same recruiters I did.
I could and can take jobs I enjoy, actively run away from promotions, etc.
It's cultural and personal. I started my career getting paid less but doing fun stuff. I paid my way through university with no debt. I only started to think about money after having kids. i.e. I couldn't care less about myself, and I didn't need a lot of money either, but I did care about my family.
I had a great time, and learnt the most, at smaller places and lower paying jobs. Then I also had a great time at some better paying startups. I probably enjoyed my "big tech" better paying jobs the least. All that said, you're generally so much above the average as a software developer that you can't even imagine what most people are struggling with.
YMMV
You saw the part about my spending my entire career from 1996-2020 doing your bog standard enterprise dev?
In 1996, I graduated from an unknown state college in South Georgia and I made $11/hour as a computer operator working on DEC VAX and Stratus VOS mainframes.
Because of…poor career choices …I didn’t hit six figures until I was 40. Even now I make about what a software engineer II makes at my same BigTech company at 49 in cloud consulting.
As I say, please play the worlds smallest fiddle for me.
Last year, my wife and I got rid of everything we owned that wouldn’t fit in 3 suitcases - including our cars - and bought a vacation/investment property in a resort area in Florida. We stay here half the year from October - mid March and we fly around the US the other half of the year “digital nomadding”. My wife in the meantime is retired at 47 and she is involved with her passion in the fitness/wellness industry and meets up with people everywhere we go and flies to conferences from wherever we are staying in a given week. I also fly for work a few times a week.
We take Uber everywhere.
This is what I meant by I can make different life choices at 49 than I would recommend a college grad making. I can be okay barely making L5 compensation.
Our fixed expenses are really low.
By cultural I meant American. Just to be clear I am not American.
I'm not quite sure what your point is. You seem to be supporting what I'm saying which is you can get by perfectly well with various choices. You made choices (poor or not) to not go after the biggest paying jobs and you're doing great. Why does a new grad today have to go work for Google?
In the US there are lots of really good paying software jobs that aren't with the biggest tech companies. I've worked remotely for a small US startup many years ago and got paid really well. The current market is tight but over the last decade if you had a pulse and could code you could get pretty decent job.
I can make different choices at 49, after having a 25 year career, a house that doubled in value in four years that I could leverage, no family responsibilities, parents that have been retired for 20 years who still have four sources of income (2 pensions and social security) that are healthy and independent at 80 that don’t need me (for now).
I also had no college debt (scholarships and going to a cheap school) and I was able to get into BigTech by being old, with industry experience that allowed me to bypass the leetcode grind. But only by pivoting into consulting.
These are life choices I could make that a new college grad couldn’t. Also when I graduated, $BigTech wasn’t a thing. Apple was barely hanging on for dear life and MS wasn’t paying more than your average company adjusted for cost of living.
If you're a parent, you probably are long enough out of college that you don't know what most new juniors are getting out of fresh from college. There's very few to no ways to pay your way through university with no debt, these days. Cheap in-state tuition & fees is over 5,000 per semester, not including books, equipment, travel, housing, food, healthcare...
When I was still in college, most kids were graduating with debt. Tuition and fees have only gone up since I left. Lots of people I knew who dropped out to try and save for college ended up in dead-end despair jobs.
Well, for one thing not everyone is in the USA. I was not. My older daughter graduated from university. So I do know. But also not in the USA.
Does nobody in the US live with their parents while going to university?
It's not uncommon where I live for parents to save to pay for their children's education. Believe it or not, the government even gives you money if you do that.
EDIT: That said, the US has plenty of good paying software jobs. You can pay back your loans even if you don't work for Google. I'm sure this isn't easy for everyone (especially these days).
> Does nobody in the US live with their parents while going to university?
The tuition fees are probably a bigger expense than your living expenses if you go to college in USA.
Here in North Carolina, the flagship state universities are still only about $10k/year or less for tuition. If the student lives at home with their parents, it really wouldn’t be difficult to pay most of their way through school. College is expensive in general but it doesn’t have to be as bad as a lot of people make it out to be.
> If the student lives at home with their parents, it really wouldn’t be difficult to pay most of their way through school.
Please tell me how a full-time student can fairly easily earn 10 grand a year (plus books, clothing, phone bill, entertainment, laptop, etc)?
If you paid your way through uni with no debt you most likely didn't graduate recently, which makes your advice to current students (or ones that have recently graduated) not very well aligned with reality.
At least in a big part of the world, student loan isn't part of the problem when looking for a job straight out of university.
100% if you are in a hole, get out of the hole first
> Having worked at a couple big tech companies I’ll say money isn’t everything in life.
That’s because you have enough of it now.
Hah exactly. I've noticed that people who say "money doesn't matter" either come from a childhood of privilege and have no real concept of value, have earned and spent enough of it to lose sense of its value, or have subconsciously de-valued money as a defense mechanism to cope with the anxiety of not having it.
That's not to say that one should work in oppressive conditions or make other people suffer for a little more money; there is nuance. But, seriously. You can't pretend that money doesn't matter.
Of course it matters but to a point, I would rather feel happy with the short life I live than miserable so I can buy a fancy car
>so I can buy a fancy car
You're assuming that everyone's goal in life is to be happy as defined by material comfort or hedonism but that's just not the case. There are many non-hedonistic things that are enabled by having money (philanthropy and venture capitalism of various forms, freedom of various forms, sense of security, etc.)
For such people, because hedonistic happiness is a worthless commodity, trading it for money, and by extension all the non-hedonistic things that money can buy, is a very favorable transaction.
It's easy to see why earning more money by sacrificing comfort doesn't make sense to someone who considers happiness as their goal in life -- it's a longer route to reach the same point.
I think the point is I would have enough either way
Do you have anything constructive to say? I worked at faang for 3 years, donated 90% of what I made to charity before changing my life to work at a place with half the salary.
Don't assume things and don't comment here with bland reddit one liners. Only say something if it's constructive.
IMO this is 100% right and not just for new grads. The compensation for software engineers is on such a broad spectrum. Even if I think leetcode style questions are completely inane and reveal horrendous values, I'll put up with it for the prospect of double the compensation (or more).
I wouldn’t go that far. I did say college grads. I know a number of people who are 40+ (including myself) who would never want to give up their lives in the burbs with the 2.1 kids in the good school system to relocate and work for BigTech.
My story is that in 2020, I was 46 years old living in the 3200 square foot house I had built in 2016 making $150K (and getting offers locally for $170k) my wife was also working part time making around $25K in the school system.
We had more than “enough”. We went on two or three trips a year, had “date nights”, saving “enough” for retirement. I ignored every message from recruiters at BigTech. I actively didn’t want to work at any large company as a software developer nor did I want to move.
The only reason that the recruiter from Amazon Retail even piqued my interest was that when she suggested I do a slight pivot to “enterprise application modernization” cloud consulting.
The extra money is nice. But it really just ended up going into my bank account and didn’t make an appreciable difference in our lives besides “retiring my wife”.
I would have no problem going back to my (inflation adjusted) prior compensation.
Your case is a strong argument for remote work. Tech companies are kind of weird in that they all focus on the same cities, SF or Seattle, then NYC, Austin, Boston and then maybe they branch out from there. As if great engineers don't live in Kansas City. There is a lot of talent being left on the table imo when companies aren't willing to hire remote, or create more small satellite offices if they can't accept remote. There's a lot of great engineers working at say Lockheed Martin because that is the highest paying job around them, that would be a strong asset to any tech company.
I hinted at it but I didn’t make it clear. The software development job was “remote until Covid cleared” in mid 2020. The role in Professional Services assigned me to a “virtual office” permanently with travel when “things got back to normal”.
I think a lot of companies are okay with leaving that talent on the table.
Sure, but that means there is opportunity for other companies to pick up that talent.
You're phrasing this as disagreeing with me but I don't see your anecdote as contradicting me at all. A lot of people would strongly prefer to make $400k rather than $150k, and not just college grads. Not everyone, and maybe not you, but many people. Are we actually disagreeing here?
I’m saying that people who already have path dependencies and are making “enough” prioritize other things over the pursuit of money.
https://bemorewithless.com/the-story-of-the-mexican-fisherma...
I know plenty of developers 40+ who would never give up their lives in the burbs of Atlanta (where I lived until this year) to move to the west coast.
Even now, I’m almost sure that the new college grads who I work with (and one that I mentored as an intern) that came in after I did at an L4 will be promoted to an L6 long before I will (if I ever get promoted). I actually told my manager and my skip manager that I don’t want to be an L6 or the responsibilities it entails. I’m already saving/investing every penny of my RSUs. My base salary is about the same as it was before I left “enterprise development”. My fixed expenses are actually lower
> ... didn’t make an appreciable difference in our lives besides “retiring my wife”.
Heh Heh Heh
Wonder if your wife would look at her being retired as an appreciable difference? :D
Well, the longer version of the story is that when we got married in 2012, we had both been laid off from the same company. Her job increasingly became “I need you to work for the benefits while I change jobs 5 times between contract and perm building my resume and getting my career back on track”.
By 2020 she was working for the school system part time on the school schedule.
I used the two year prorated signing bonus to pay off all of our debts, increase savings and reduce our expenses.
We also moved from the big house in the burbs of Atlanta to a smaller condo in Florida and we don’t pay state income taxes.
So yeah over the past three years I both increased my compensation and reduced our fixed expenses.
Big tech is not hiring now. Those absurd total comp levels may not be coming back.
Big tech includes more people, and pays more, now than it did in 2019. 2020 and 2021 hiring and offers were unprecedented. Things have slowed down, but this is no where near a dead, or even cold, industry.
Yea, I was just told by an AWS employee that they're desperately trying to hire as many people as they can still. Those companies are massive. The layoffs and divisions with hiring freezes don't seem to even make up a tenth of their capacity to hire.
Sigh. No one has stopped hiring SWEs. Only 5% of those laid off were SWEs[1].
[1] https://interviewing.io/blog/2022-layoffs-engineers-vs-other...
People who work for BigTech are still making those “absurd “ total comp levels. I see no indication that the offers are much lower or that when reviews come, they aren’t attempting to offer more stock to keep them at somewhat decent levels.
I’ll know myself in a couple of months…
Good luck putting that particular genie back in a bottle now that people know how much money is just floating around.
What do you mean? It's a market. Knowing the money is there does not magically make it get paid out to employees.
Maybe not. Probably so. Do you have any indication that tech is dead?
The amount that Big Tech can afford to pay is correlated to their revenue.
The amount that Big Tech is willing to pay is correlated to the market supply of employees.
But I'm unconvinced that a very profitable company will decide to pay less than it can afford, in order to acquire the best engineering talent it can, when their revenue is directly linked to their engineering products.
The only thing that would make FAANG pay nosedive is if they made an organizational decision that they didn't want to prioritize software development.
So basically, if one of them decided to hire an Eddie Lampert type.
> I'm unconvinced that a very profitable company will decide to pay less than it can afford
They are not o paying maximum of what they cam afford, they are making huge profits.
If they are not paying maximum that they can afford what are they paying? Provavly the minimum they think they can get away with.
Its all mind games.
I agree that they could indeed be paying their engineers far more and still make a profit, but that’s just supply versus demand. They found the pay bands that provide the sweet spot:
Relative maximums for profit and talent willing to accept those salaries.
do they? You seem to generalize a bit no? I have 2 grads that are pretty happy with their life and their salary right now, and are starting to look into buying a house. So I'm doubtful it generalize well.
You can have pretty good salary in non-FAANGs, make a difference, and match your own software values with what you're doing.
I would even argue that by doing so, you're elevating those company, allowing them to then compete a bit more closer to FAANGs comps, and "stealing your talent" away from those less-matching software values, creating what I believe (from my own values) to be a virtuous circle.
Or you can grind at FAANGs, accept stack ranking and elevated salary, play the yearly promotion game and end up with a very nice pile of money. Then clearly do not expect them to change, and I would argue: you do not get to complain unless you are actively trying to change them from the inside
I am not complaining. I work for one reason only - to exchange my labor for money to support ny addiction to food and shelter.
And let’s not pretend that startups are trying to compete with BigTech. They are just trying to survive long enough to get acquired. Out of all of the companies that YC has founded, how many have gone public?
If you are working for a company that has accepted VC money, your “values” don’t amount to much at all. The only values that matter are those of your investor and all they care about is an exit - statistically by getting acquired.
The opposite choice is accepting Monopoly money - “equity”.
True my unvested RSUs are half what they were at their highs. But at least I can sell them and trade them for real money once they vest every six months.
Don’t get me wrong, I spent my entire career from 1996-2020 working for mostly unknown companies working as an “enterprise developer”.
The only reason I fell into my current BigTech job is because I both know how to talk to customers and I can create pretty diagrams, PowerPoint slides and a shit ton of yaml and HCL and can develop (cloud consulting department)
Yup. It’s FAANG or almost don't become an engineer.
I mean, not really. There are plenty of us making US FAANG salaries outside the US and not at any major tech company.
My sample is myself and all my peers.
Industry is BIG and the demand for smart people with a track record for getting things done is even bigger.
Most years I barely do anything more than basic eng management and my teams build glorified CRUD apps.
I'm paid much more than others in the UK because I'm a sure bet. FAANG not required.
Can you share some samples? The only ones I can think of are investment banks, but you need to be in the top 5% to be paid as well as FAANG and live in one of the Big Six global banking cities: New York, London, Tokyo, Singapore, Hongkong, Sydney.There are plenty of us making US FAANG salaries outside the US and not at any major tech company.Amsterdam has trading firms
Had recently a message from LinkedIn: some trading company is hiring to Amsterdam office, offering 85k plus annual bonus up to 40%. Peanuts comparing to NYC offerings.
That is exactly my point!
I guess FAANG-level comp is less than 500 people in the whole city.
>making US FAANG salaries outside the US and not at any major tech company
Out of curiosity, what kind of non-FAANG domain are you in that pays soo good in the UK? IB? HFT? ML?
I'm asking since every single UK person on HN complains that UK tech wages are shit yet you claim otherwise.
What makes you the exception or why do you think the others would be wrong?
Thanks in advance.
Defence/gov £250k base. CRUD apps, nothing fancy, just lots of data.
The salary curve in the UK has a long tail of rubbish pay because that's what people accept. Budgets are often much higher, and supply of skills is low.
You just have to follow the money and negotiate well. Firms with deep pockets don't care if you cost 90k or 300k when the project is in the hundreds of millions and the fallout of failure would cost much more.
The important think is to show you represent 5x less risk for 5x more pay.
How do you show that? A large portfolio of successful projects/many years of experience?
Yes. Also be good at interviewing. 90% of getting what you want is being liked without appearing like you need to be liked.
Wondering the same. Maybe they meant "US FAANG salaries" very literally, i.e. just the salary, not the bonus and equity...
If they really mean US FAANG total compensation, my guess is quant trading firms.
I did mean salary, I'm not sure about total comp because UK firms don't seem to operate similarly for the most part.
even UK quant firms don't really pay similar to US quant firms unless for the very top roles or firms that have multiple locations
Do they pay similar to US FAANG, though?
That's the comparison sirsinsalot was making.
"FAANG" is long outdated for the reasons you say -- I think it's shorthand now for "tech companies that actually pay really well" compared to, say, the public sector, or startups, or small companies that can't afford to, or places that view software development as an unfortunate cost to minimize, etc.
In any case, even in that light, the claim of FAANG or don't be an engineer is absurd. Even if someone doesn't end up with a really well paying job it is still a solid career option "especially in this economy"
In the UK at least the compensation curve is weird. You can be on a team and be paid more than the senior devs above you.
You just have to play the game right. Excellent developers are rare in the UK in my experience or hiring. Mediocre ones who think they're excellent are everywhere.
Yeah with the opaqueness of comp, the same thing can happen in north america too. Someone beside you will be getting massively more RSU grants or something because privately management likes them more.
> Mediocre ones who think they're excellent are everywhere.
Absolutely
I think you are underestimating FAANG comp. Which non-FAANG companies outside the US pay >$300k USD?
I work at a FAANG and took a paycut to get here (from a startup).
What other occupation are you going to pick that you can a) get hired fairly easily right out of college, and b) make $150k+ in a few years with only an undergraduate degree?
Do I understand you correctly? After all that's happened in the last 12 months in Big Tech you tell them to pursue cheap Fed money infused talent hoarding jobs at FAANG companies?
As opposed to working for non profitable startups?
The second part is that I tell them to save aggressively and diversify themselves by selling their RSUs as soon as they vest. They wouldn’t use 25% of their cash salary to buy their company stock, why keep your shares once they vest?
I sell all of my RSUs within the six months after they vest and diversify
I would much rather to have loved and to have lost than to never have loved at all. I used every penny of the after tax difference between my “enterprise job” I had in 2020 and my BigTech job that I got then to “increase my net worth”. If I lost my job , it would have been a good three years.
The next time any recruiter asks me to do a "homework" assignment, I will ask them to write me a 15-page essay explaining how that homework assignment will actually be used, to what extent it will be reviewed, and what criteria it will be judged on.
If the comeback remark is something like, "if you really want this job," I will reply, "if you really want to hire me."
My current job, I told them that I was too busy to do such a thing (and I was), and got hired anyway.
Nobody w/ actual responsibilities in their life should be coerced into doing something for free, for someone they do not know.
Will an attorney give you free legal advice until you decide they are fit to represent you, or will you get free surgery until someone proves they won't completely butcher you? Can you drive a car for free (for 5 - 8 hours) until you decide that's the car you want to buy?
Why is it any different in the software industry? Because we just clack on our keyboards all day and do nothing?
>Can you drive a car for free (for 5 - 8 hours) until you decide that's the car you want to buy?
This is a perfect analogy.
I went to a Chevy dealer once, asked if I could test drive a car and they wouldn't even let me take it off the lot. I was allowed to trundle around the rows of cars at walking speed with a salesman in the passenger seat.
I went to a Honda dealer, and they let me take a test drive with a chaperone, but only around a short designated loop of streets "for insurance reasons".
I went to a Mazda dealer, and the salesman said he was busy and tossed me the keys and said have fun.
An acquaintance went to a Subaru dealer, and took the car they were considering home overnight.
You a Chevy.
Going further off topic, it amazes me how cagey some dealers can be about test drives. Same as you, I've had some just toss me the keys, without even wanting to see my drivers license, and that makes an infinitely better impression than wanting to come with me or wanting to talk about my needs first. I don't understand why it's not standard.
I think it's more up to the individual dealer than a policy of a given brand
It's sometimes useful if the salesperson can tell you things about features of the car while you're driving, but usually I know more about the car than they do.
I'm sure that some brands/locations have essentially zero risk of theft while for others it's a bigger concern. It's similar to how some places you can just pay in full with a personal check and they are fine with it (even not doing a credit check), while others won't accept that.
Insane. You've got to at least test drive on a highway.
Some years ago, a Nissan dealer offered to let me take a high-end crossover home for the night without any prompting. Ended up buying it. Would buy again from that dealer if I were still in that area.
If it's insured, it's no skin off their nose.
Gestures of goodwill go a long way in sales, that's why people love to pay for coffee or lunch when out with potential customers.
I've bought three cars in my life and never drove any of them before buying. I know it's unusual but they've all been great. I did do a lot of research ahead of time though.
> Will an attorney give you free legal advice until you decide they are fit to represent you
I agree with your point overall, but I do have to say that it is very common for lawyers to give free consultations with potential clients, often offering very useful advice. This is something I've benefited from more than once, actually.
Yes and developers often give free advice in interviews. However attorneys are not writing contracts for free to get jobs.
And in fact, if as an employer all you require is a copy-pasta employee contract, you probably still want to go through a lawyer to have a professional opinion and work. But they on the other hand often just copy-pasta, and write an expensive bill.
"Can you drive a car for free (for 5 - 8 hours) until you decide that's the car you want to buy?" Yes. It's called a test drive. I think the longest I've seen is a free 48 hour test drive with an online e dealership, but free all day test drives are the norm in the UK. Similarly most lawyers will give you a free consultation. I think you have a highly distorted view of reality, how other professions operate, and of your own worth.
>Will an attorney give you free legal advice until you decide they are fit to represent you, or will you get free surgery until someone proves they won't completely butcher you?
These are bad analogies because both have extremely extensive tests that are not only unpaid but the tested pays a small fortune for.
If development had the same no one would be asking you.
>I will reply, "if you really want to hire me."
It’s not about hiring you it’s about trying to prevent hiring the wrong person which is extremely expensive in time and money and takes weeks to figure out.
I understand what you're saying, however, I think that a lot of the criteria used in these homework assignments can instead be worked out via some conversation to judge the depth of knowledge and culture fit.
I've passed tech interview challenges only to fail the culture fit because I thought it would "be so easy" after the tech challenges.
And I've passed culture fits only to fail the homework assignments.
In one circumstance, not realizing that a separate recruiter was sending me to a company I had previously interviewed with, I've also seen previous work that I did in a homework assignment, given to me in a different homework assignment with a "what improvements and features would you add to this solution?," when the original homework assignment was the same task. That was a major blow, and gave me feelings they were using some of that work internally.
I get that it's extremely expensive in time and money to hire the wrong person.
On the other end, it can also be extremely expensive in time and money to not be extremely selective of whom you want to work for.
lmao do I understand correctly that you were given code that by chance you yourself happened to have written and were asked "how can this piece of crap be improved?"
It had work in it that I put in before, but it wasn't entirely all mine, nor did it have all the original work I put in the first time.
> Will an attorney give you free legal advice until you decide they are fit to represent you
All but one new legal engagement I’ve entered into started with a free consultation. (The only one that didn’t was a straightforward real estate transaction where I knew the lawyer for years beforehand.)
I don't pay $200,000 a year for a car. If I were going to then I'd like to test drive it for 5-8 hours. An ideal test drive should help both sides understand what the future relationship will be like. That saves time and disappointment on both sides. You'd like to date someone before you marry them wouldn't you. I would always be happy to pay for a test drive though. If you're not willing to offer a test drive then you'll also probably miss out on some great relationships in your life.
Fair point, though nobody is obligated to pay a year's salary to a candidate that didn't work out.
Clearly you've never employed someone in the UK/EU.
The proceedings to jettison a bad hire can take 12 months alone, meanwhile they're still a bad hire being paid.
In the UK you can sack someone for being a poor fit without any of that within the first 2 years.
Good luck at the tribunal
Employees in the UK don't have access to an employment tribunal unless they've been at the company for longer than 2 years.
BRB, speaking with HR about some firings.
It is entirely normal to have a test period of 1-3 months here (inside the EU, which is not homogenous and doesn't look like the UK at all) where you can be fired immediately.
It isn't a magic bullet. I've had people perform well then tail off and while it may be a failure of my management that I couldn't help them perform better again, it isn't the norm and replacing those people isn't easy.
Rightly so, it should be hard to fire people and should require first showing evidence of trying to work together and help in good faith.
Thats just wrong, you can have 3 day, 3 week or 3 month probation and fire them immediately if you dont like them.
And if you cant tell whether they are good developer after 3 months, how is an intervoew going to help?
Have you ever run a business or paid employees from your budget?
Maybe I'm weird but I kind of like the take-home assignments. I find them pretty fun and not at all stressful like a whiteboard interview. And I've never not gotten an offer this way because I think there's a lot less variance (whiteboard style interviews you might get asked some random BS that you don't figure out).
IME you also often get paid (albeit a relatively small amount) for doing them. And FWIW once they want to hire you many employers will actually spend hours trying to sell you on joining. Usually not in the form of an essay, but doing several hours of sell calls or lunch meetings with different people at the company is not that strange (especially if you are higher level).
You’re not weird, just lucky. I too have gotten my longest job of my career from a successful take home exercise, but I have never been paid for my work and am rapidly reconsidering just how willing I will be to do take home exercises for free in the future.
> Because we just clack on our keyboards all day and do nothing?
No, it is because some people have worked at very impressive jobs and probably just clack on keyboards all day and actually did nothing while there, so when you hire them you learn that they actually can't do the job you hired them for. You just can not rely on the CV alone.
In the hiring process there has to be some kind of skill test. If it is not a "homework" then it has to be a whiteboard/live coding/system design type of interview and there are a ton of problems with this type of skill test as well.
We usually give people the choice which route the candidate would like to do. Take a 1 hour interview or a "homework" assignment which took me about 1 hour to solve. Which is probably best because some people really prefer the homework because they get nervous in interviews.
> Will an attorney give you free legal advice until you decide they are fit to represent you, or will you get free surgery until someone proves they won't completely butcher you?
I do not know how attorneys or surgeons are hired, but my guess is that the education side of these jobs lines up much closer what is actually needed to do the job (which isn't really the case with Computer Science) and that when they claim to have done X or Y then it was actually them in court or at the operation table and not someone else from their team.
Clients select their attorney based on reputation of the attorney or the law firm they work for. For Surgeons it is probably similar. If you have the choice you want to go to the hospital with the best reputation. If the company you apply to already knew your name and reputation beforehand then the skill test is probably also not necessary, but that is not usual in my experience.
It also probably helps that both of those jobs require a Professional License to practice these jobs. Maybe if the software industry introduced a Bar Exam then these "homework" assignments would not be needed anymore.
> Can you drive a car for free (for 5 - 8 hours) until you decide that's the car you want to buy?
Usually you can. At least in my experience. Not fo 5-8 hours of course, but long enough to know.
If you know of a better way to hire people: I would be happy to listen.
An interactive whiteboard / live coding session is by far greater than any homework assignment, in my opinion.
W/ an interactive session, you get instant feedback (either verbally or via emotional cues) into what they are expecting.
With a homework assignment, it's hard to determine which path to optimize for.
If a homework assignment is necessary, it would be better if that were more of a "probationary employment" type scenario, perhaps at a very reduced amount of pay, to only imply that both parties actually have skin in the game.
Even better, for helping me judge if it's a decent fit? Show me some code you're using in production so that I can code review it on the call. Surely it's not all hyper sensitive.
I work on the matching engine of a stock exchange. So it kind of is fairly sensitive, but I get what you mean.
> interactive session, you get instant feedback (either verbally or via emotional cues) into what they are expecting.
For some people, like myself, that is a nightmare scenario. I do not do well in these type of sessions. Which are also not even close to reflecting the real work we do. There are absolutely no meetings I go into that I can't prepare for [1] and there are absolutely no meetings where I have to solve a problem on the spot.
Being on the autism spectrum also means "emotional cues" are pretty lost on me.
There is also the fact that interviews have to be during normal working hours. Some people prefer to do a "homework" assignment which they can do in an evening or weekend.
This is again why we provide the choice. Not everyone is the same and prefers different interview and assessment routes. Both types are useful for different people.
> If a homework assignment is necessary, it would be better if that were more of a "probationary employment" type scenario, perhaps at a very reduced amount of pay, to only imply that both parties actually have skin in the game.
Really? You would refuse a "homework" assignment, but would agree to "probationary employment"? The later just sounds like WAY more work on both sides.
We have discussed something like this in the company I work for, but came to the conclusion that it just is way too much work. Getting the contracts in place and working out the insurance and tax implications and all that. It just is way too much work to do legally, because it would be the same amount of work to just hire them. However: we can't just hire everyone.
Maybe if you are already a freelancer beforehand then we could work something out that way, but in the jurisdiction I am in not every software engineer is ready to accept freelance contracts. It is a simpleish process to do, but not everyone does and those that do don't apply for full time positions.
[1] You can "prepare" for interviews, but more in a scatter shot approach studying all the interview questions that could possibly be asked. That is not what I mean. There is no meeting I am going into where I do not know the precise topic that the meeting is about.
I have been caught off-guard by coding tasks in the middle of an interview that I thought was more geared for casual conversation, and it's not always a comfortable feeling, but sometimes it can prevent either party from wasting a lot of time.
However, I've also been a few hours into a homework assignment thinking that I could probably go down some rabbit-hole to try to perfect something that I may have been struggling with, and sometimes can't determine the appropriate stopping point.
The "probationary employment" would be more like, I don't know, a gift card, or something, vs. something formal.
That way, if I totally bombed out in some assessment, no big deal, here's something for taking the time to apply, and maybe I could use the card to buy a book.
Now, I get that companies aren't giving gift cards away to all of their interviewees, so this type of thing would only come after at least the first round of interviews, etc.
More often than not, a non-interested company will often not even tell why they didn't pass the assessment, and it generally feels like a waste of time.
The downside to 'probationary period' is that as a candidate you're limited to a single company. If you have to spend a week/month at X, you can't really consider Y, you won't be able to give them the time. Then at the end of the week/month X gives you an offer and you can't even compare it to any other offer, because you didn't have time to get any.
Another thing is, if they let you go after the probationary period, you pretty much start the search anew. If you applied to another company while you were 'employed' by X and you managed to schedule another probationary period with Y right after the end of the trial period with X, you have to refuse them last second in case X wants to hire you. If you didn't search for anything while at X, you're off work for another week/month while you search for another trial.
If X ends up giving you an offer and you want to consider Y, what are you gonna tell them? Please wait a month for me while I work for this other company and see if it's any better than you? This simply doesn't work, for both companies and candidates.
Your proposed solution works, if X is your dream company and you will accept their offer no matter what. That's not how most job searches go though.
Yeah, that is what I mean when I say that either way has problems with them.
I had someone try to implement a whole relational database when the interview task was just to read from a CSV file and provide a REST API to the contents of said file using any tech stack. Impressive for sure, but unnecessary time wasted.
> The "probationary employment" would be more like, I don't know, a gift card, or something, vs. something formal.
We discussed something like that in the company. Mainly because someone asked to be paid for the time spent on the "homework" assignment. We came to the conclusion that there is no real legal way for us to do so. We probably spent more money on discussing the possibility of paying the candidate then what 1 day of work would have cost us, but the cost wasn't even the issue.
With an in person interview there was maybe a chance. Inviting the candidate out to the exchange, giving a tour of the trading floor and then paying for transportation, lunch, dinner and hotel would be no problem.
Though interviews are online now as we are a "remote first" company anyways.
We just can't pay for work without a contract, insurance, tax and background checks in place. We can however ask them to complete a test. Which is what the "homework" assignment is.
> More often than not, a non-interested company will often not even tell why they didn't pass the assessment, and it generally feels like a waste of time.
It sucks. It just is that nothing positive can come from providing feedback and you open yourself up for a lawsuit.
Whiteboard coding almost always devolves into leetcode which also requires at home study. You're going to be spending evenings and weekends coding something in either case.
That part is understandable. Perfecting your craft requires many hours of hard work and dedication, and you can never know everything.
For me, I work on a lot of open-source as side projects, so there's always the coding "something" factor.
> it is because some people have worked at very impressive jobs... they actually can't do the job
> In the hiring process there has to be some kind of skill test
Thats a contradiction:
1 - 90% of companies do skill tests
2 - you cannot trust the CV of a candidate even if the company they worked on previously also did a skill test
3 - you think your skill test will enable you to hire the right candidate
So everyone is doing tests and everyone is hiring shit candidates anyway?
That's literally proof that these tests are worthless.
Maybe if the recruiters actually knew what they need from a candidate, companies were clearer about what the job involves, they would stop hiring the wrong people for the wrong job.
1 - I don’t know the hiring standards of every company and that 90% sounds made up to be honest.
2 - We can not trust the CV. Maybe the company did a test, maybe they didn’t. Maybe their test are to our standard, maybe they aren’t. Maybe the candidate barely passed. Maybe the candidate actually failed but made up in other ways that might be relevant to the other company, but not ours. I doubt anyone would give us this information about their current or former employees. Maybe the candidate really did work on the projects he claimed or maybe he was only tangentially involved. That is a lot of unknowns. A test clears up what the person actually can do.
3- Yes. It works for us. Do we miss out on some good candidates because of this? Yes, but it definitely prevented us from hiring bad candidates.
Again: if there is a better way then I would like to know. I have not found one yet.
There are some studies that claim a large percentage of people (40%+) lie on their CVs. As someone who hasn't done that and struggles to lie even in games, I have a hard time believing it. But even if 5-10% of people lie, hiring is a lemon market, and you definitely don't want to get stuck with the lemons.
Most lying is apparently to change start and end dates of previous employment to cover up short stints where they may have been fired for poor performance, but even if work samples/skill tests/structured interviews were 95% effective, you would still regularly hire duds.
I believe it. Even with the take home assignment people lie and cheat.
We had one person who literally hired a team in India to do it for him. Presumably planning to give his work to them later if he was hired.
How could you possibly have found out about this?
You sound so jaded, I don't blame you, but I hope you're not ever my hiring manager.
We asked him some questions about the code when he came in for the last interview and contract signatures. He didn't have a single clue about any of what he supposedly wrote. When we talked further he admitted to using a Team in India that has been working for him for a while.
Needless to say we didn't proceed with signing the contract.
The better way is to treat people's time as important as you treat your own and your resources at your company.
There is a large company based in NYC that interviews thousands of developers a year, but only hires a few hundred. Each of those devs do a take home project that takes about 16 hours to complete. The project requires nothing to ask, so this company asks everyone. 30,000 hours is a lot of time - it's about 3.5 years of someone's life - and at least this time is spent each year by this company (and they pay nothing for it), not counting the rest of the interview process, because it is cheap for them to do this.
And we wonder by productivity is down :)
Respect people's time. If it is hard, figure it out. That is your job if you want to hire people and feel good about it.
I've tried to push my employer(s) to offer the dual paths. Like you I prefer the HW-style process (and think they are a truer judgement) but I understand why some don't like it. I wish more companies would support either path.
> In the hiring process there has to be some kind of skill test.
No there doesn’t. There isn’t one for the CEO is there?
How many CEOs have you interviewed to run your company? How did you conduct the interview? Do you think that same interview is appropriate for a developer?
The fact of the matter is that without the skill test there are way too many people that are incompetent and you have to have some way to filter those out.
I do not know how it works with CEO positions, but I am not hiring CEOs. I do hire software engineers and would like to work with competent people.
If you know of a better way then I would like to hear it.
Then fire those people. It's an easy solution.
We would like to know who is competent before we hire them. The point is that we do these types of tests so that we don't hire them in the first place.
We can't hire everybody. I can't hire 20 people (then fire 19 of them) for the 1 position in the 8 Person team I want to fill. That can't work.
Just hiring someone and then firing them 2 weeks later is expensive as fuck. We can't just do that until we find someone actually competent.
Your "easy solution" is only easy if you don't think about it at all.
You're just hiring people who care to do these types of tests or are good at them, they don't mean they will be good at the job.
I'm not saying you don't do ANYTHING to hire someone. Obviously there needs to be some judgement of skill. But take home projects where you're judged essentially against how much time you put into them (I've personally had projects where they asked me to build an entire web app, it's insane) because ... it's a race to the top in a way here, if you spend more time it's going to look good, because the best projects are from people that spend a lot of time on them...
It's just ridiculous. You end up doing 8-16 or more hours of work for a job you likely won't get (it's free for you to ask me to do this test, so you ask everyone to do it).
It's not expensive to hire and fire someone. Not that expensive. It's probably worth 20hrs of 100 candidate's time, that's for sure. If you don't think so, I have a bad opinion of you.
I'd like to ask you, how many candidates a year do you have turning in take home projects?
> You're just hiring people who care to do these types of tests or are good at them, they don't mean they will be good at the job.
We align the Take home assignment very close with what the position actually entails. If you struggle with this assignment you would not succeed with actual tasks we do daily.
> Obviously there needs to be some judgement of skill. But take home projects where you're judged essentially against how much time you put into them
As I said elsewhere as well: People are free to choose to do a Technical Interview instead. If you fear that you would need 2 whole days you could take the 1 hour interview instead.
> if you spend more time it's going to look good, because the best projects are from people that spend a lot of time on them...
That is not my experience. There are people that just can't make it look good.
> It's not expensive to hire and fire someone. Not that expensive.
It absolutely is. Maybe not where you are, but where we are it is expensive. For one there is a big amount of paperwork involved, contracts, NDA's and background checks. We can't do this unless we know we actually really want to hire you.
I also can't expect a new employee to be productive right away. There is going to be a onboarding period of at least 3 months during which the employee probably needs a little bit more mentoring of a (more) senior engineer. If we would just hire 20 people for every position that we need to fill we would not do anything else then just onboard new people. So, YES, it is significantly more then 20h of 100 candidates time. Not to mention that the take home we give out takes maybe at a maximum 4h (if the candidate is fairly junior and has to look up stuff constantly, many people just do it in 1h).
Also: I think it would be HIGHLY unfair for the people to just hire 20 candidates if you only plan to keep 1. They probably quit their jobs or said no to other opportunities. We only hire people we see a long term future at the company.
> I'd like to ask you, how many candidates a year do you have turning in take home projects?
It is fairly late in our hiring process and it takes a significant amount of time from at least 2 senior engineers each time to review. I don't have the exact numbers how many times per Year, but we try to keep it very low and only if there is an actual interest on our side.
> People are free to choose to do a Technical Interview instead.
Ah, I missed this, and I'm sorry. My experience has not been this over 100+ technical interviews over the last 10yrs.
Almost all of them have asked for crazy take-home projects. I'm pretty bitter about the fucked process.
It's extremely intensive to onboard a new employee; and extremely disruptive to onboard a new employee and then immediately fire them. It's also very demotivating to the existing team.
I would say it's a lot more disruptive to leave someone incompetent on the team and let everyone else make up their work instead of building a solid team.
IDK, schools do it all the time.
Because take-home exercises are both the best way to evaluate a candidate (I have never gotten a nasty "surprise" while hiring in companies that did take-home exercises: I mean it, absolute 0% "bad hire" rate, the only bad hire I got was a candidate that was allowed to skip the take home exercise) and they allow you to find absolute jewels amongst the candidate pool who are great, reliable engineers but regularly bomb regular interviews.
While there are people who are naturally great at interviewing in person or who don't mind grinding leetcode for free for months on end, there is a whole "base of the iceberg" population who a) can't spend months grinding leetcode on end but they can definitely spare a couple of afternoons for one job they are interviewing for – these are normal people with normal jobs and a normal family who don't interview every month just for kicks, they do this a couple of times every 2-3 years, and, b) just cannot code while under stress. You may scoff at b) if you've never experienced it, and while I found it really hard to understand for a while as I also have no issues with getting into deep focus while people are looking and/or trying to talk to me and there's a time limit, it is absolutely real.
Unfortunately, there is an extremely vocal minority (you) who go absolutely ballistic when asked to do a take-home exercise, who absolutely ruin it for everyone else and make hiring managers shit their pants whenever take-home exercises are suggested. I honestly don't understand why there's such an outrage, take-home exercises are the minority already, because there's a strange huge backlash.
Sure, if you're a great communicator (usually native English speaker) and grind leetcode for fun so you can shove 5 interviews in one week, you hate take-home exercises. And that's fine, really, just please apply to the leetcode grinding contests and stop poisoning the well for the rest of us who would like to cater to the large amount of extremely competent engineers that don't fit that persona.
> Nobody w/ actual responsibilities in their life should be coerced into doing something for free, for someone they do not know.
I take it you haven't interviewed for a SE in a while? The status quo requires you to perform months of unpaid labor, not days, just in the form of memorizing the solutions to every single Leetcode Hard problem. How is that better?
> I take it you haven't interviewed for a SE in a while? The status quo requires you to perform months of unpaid labor, not days, just in the form of memorizing the solutions to every single Leetcode Hard problem. How is that better?
I guess because it kinds "scales". Once you are familiar with Leetcode you can use this skill to take interviews with different companies. Take-home exercises dont' "scale".
That being said, I believe people are against take-home exercises exactly for the reason you support it: it's usable code. They worry the companies will exploit candidates by using their code for free. Leetcode is "useless" so it's safe.
For clarity, these take-home exercises that I'm referring to are completely toy problems. They're usually one or two REST APIs on top of a toy data structure (sometimes in-memory, without a DB at all), just to see how the candidate models, writes and tests code. An API that does some pricing math based on some in-memory state or the likes of it.
If we're talking about "make me a website" kind of exercises that one could throw into production after some tweaks, then that is free labor and I would absolutely refuse those.
Also, the fact that "leetcode scales" is part of the problem I was talking about. I really dislike the fact that if you train for months to develop a complete set of skills parallel and irrelevant to your actual role you can now efficiently interview for a number of companies that pay top dollar. So not only people with actual skills and experience are on equal (or worse!) footing than someone straight out of college, it also incentivises those who mastered leetcode to interview everywhere, since "it's all the same", while normal people who are good at their job get their torn to shreds after one or two interviews. So not only it's judging the wrong thing at the interview, but it's also causing a starvation situation for the candidates that are not playing that game.
Why are we creating incentives to people becoming "professional interviewers"? This would be like Google encouraging SEO spam instead of fighting against it!
I agree with all what you said. I'm just speculating on why some people prefer leetcode over take-home exercises.
I've never had a take home assignment that was "usable code" - they've all been toy problems.
That's certainly your prerogative, and we all have to draw the line somewhere in terms of interviewer demands. Personally a reasonable homework assignment (like 2-3 hours tops) is less annoying than multiple 6+ hour interview panels stretched out over a period of months, which is also very common. At the end of the day though, the company decides the hoops you have to jump through if you want the job, and you can either take it or leave it. FAANGs can get away with onerous processes because of the technical brand and outsized comp, but startups and lesser known companies who try this are shooting themselves in the foot.
In my experience you still have a 3-4 interview loop on too of your take home project.
While I do agree with your general point your analogies are bad because yes, you can test drive a car, yes attorneys often have a first free session with potential clients.
That being said the come back makes perfect sense to me. When I was still on the IC track and interviewing, when I was submitted to some "dumb" questions I would ask the same kind to the interviewer. So you just asked me to write this stupid algorithm? How about you whiteboard something for me now?
It was almost always met with dead silence to which I would say "well you're trying to figure out if I'm good, I want to do the same and make sure your team has good engineers".
Of course it never went anywhere since, while they had 55 minutes to grill me I was only given the last 5 for my questions.
There's a big difference between both of these industries and the software business - if you show up claiming to be a lawyer or a doctor and are bullshitting, you're in a LOT of trouble.Will an attorney give you free legal advice until you decide they are fit to represent you, or will you get free surgery until someone proves they won't completely butcher you?If you do that in software, you're probably just back on the job market...
> I will ask them to write me a 15-page essay
I have my own list of questions. If they answer all of them I have a pretty good idea if I'm the guy for the job. The most wonderful part is figuring out if it is an employer is looking for initiative or obedience. If you are running a sheep farm it can be very exciting to see initiative.
Agree, lately I’ve been telling people I’m too busy interviewing, they try to force it then call me back in a week and let me skip
i got a really good job off of a homework assignment so this is a hard disagree for me. of course mileage might wary
As someone who's been around the tech industry for a while, I know firsthand all the sausage-making that goes into building a great technical hiring funnel. On the flip side, as a job seeker, I also know how demoralizing it can be to go through a broken hiring process that doesn't accurately reflect your abilities.
With recent layoffs and many talented professionals on the job market, I was compelled to write a blog post about how to build an inclusive hiring culture and find exceptional engineering talent.
If you're involved in your organization's technical hiring process at any stage, I encourage you to give this a read. I share some best practices for conducting effective interviews and improving your own hiring process.
Let me know what you think!
I think structured interviews can be just as easily biased as unbiased. I'm not disagreeing necessarily about being structured with your interview practices, but I find this to be a grey area with promotions also.
As soon as people figure out the check boxes or the structured pointing system they start to check all the boxes, but it doesn't necessarily speak to the nuances between individuals that make them diverse and both valuable. In fact it can lead to a certain type of person people hired or promote, which can be on good characteristics, but I find many times turns into a "certain type" of person.
I guess what I'm saying is structure can take you so far, but you have to be willing to explore a little bit about what makes a person special, and that many times means not controlling the whole interview, and be willing to have your bias challenged through the candidate directing some of it.
Agreed. It's very demoralizing to know that you are extremely capable of doing the job, but since you don't satisfy the narrowly defined fitness function you're being passed into, you fail. But more than that, companies are missing out on great people that may simply have a different way of thinking that isn't accounted for in their structure.
My take-away from interviewing is that as a candidate, getting a "you did not get the job" doesn't change anything. Since I've had the privilege of mostly being employed while interviewing for new jobs, this is a "meh, status quo" thing. Nothing changes, I have no decisions to make. If I get an offer, I have a decision to make.
Does it sting when I get a "No"? Yes, a little, but I did my best and (presumably) someone else did better. So, I take solace in that I did not have to make a (relatively large) decision.
I've also been kinda curious about the research in favor of structured interviews, because as far as I've seen they're generally drawing their conclusions based on "interviewer-predicted performance" vs "actual performance".
How the heck do you measure actual, repeatable performance? Or skill? Income / promotions / etc is very frequently a horrifically biased metric, for similar reasons to interviews, and we have much larger mountains of evidence showing that to be the case. It seems like there's a pretty good chance these studies are just measuring relative bias between interviewers and the interviewee's management, and concluding interviews are done poorly when they disagree with management. i.e. "structured interviews force people to think more like managers" rather than "structured interviews more accurately measure skill".
Using one bad measuring tool to conclude another tool is bad seems... problematic at best. I will grant that "interviews should measure what managers measure" is often what businesses want in bulk, but that does not seem like a particularly good thing to me.
In another comment I was also poking some holes into the “structured” comments, what you say above was in my head but I couldn’t quite articulate it. Well said.
I think checklist interviews miss the mark as you say. You may not have a perfect rubric, but I’m not grading students on a history exam; I am evaluating them for a role in a given position. In the limited time we have to speak, I want to use my intuition and experience as an interviewer and engineer to rapidly get to where the candidate is strong, and where they may have issues.
Project based interviews, even paid, are so frustrating because there is no ROI if you don’t get the job. If I study leetcode, I can apply this skill to interview at a bunch of companies.
I once spent 2 hours coding Tetris for an interview. I lost to another candidate that completed 2 more features than I did in the same time period.
Super minor language nitpick, but I thought you might want to know. There are a couple places where you seem to have mixed up empathetic and emphatic. One section is titled “Emphethatic”. I’m not sure if you were trying to make a portmanteau or just misspelled it, but I was confused by it. Other than that, I appreciate you taking the time to write up something like this. Wish it existed when I interviewed in my younger days. (MS also asked me the manhole cover question.)
Appreciate the feedback! That was a typo. Just fixed it. Glad it resonated with you.
Still there as a heading.
> inclusive hiring culture and find exceptional engineering talent.
Thats a contradiction in terms. Building something exceptional always involves excluding mediocrity
I'm sure that sounded smart when you wrote it.
An inclusive interviewing process does not mean that you hire everyone. It means you reduce the weight of people's biases as part of identifying who you hire (because people turn out to be quite bad at prediction in hiring).
Employment is, be definition, exclusive. More so for well-paying companies.
I hate when companies lie about that, it massively lowers my respect for them.
They don't lie, they just call it "culture fit"
People in general yes, but some particular people become reasonably skilled at it if they put effort into it over a long time
Sure, but if you are lucky enough to have an entire team of interviewers who have this much experience, you're probably not having the same hiring conversation that's happening in this thread.
That sounds nice and all but the distribution of engineers doesn’t match population distributions.
This leads to organizational pressure to hire based on population distribution. Doing so inevitably means hiring based on attributes other than skill.
Fair, but:
- The companies that make an effort get a lot closer to population baselines than the ones that just give up.
- Organizational pressures are something leadership and management should be steering. I'd rather have hiring practices be an explicit choice than something that "just happens"
If the % of woman developers is 10% and management creates hiring goal of 50% it detorts reality. It means hr has to work harder at filling those female roles which often reduce their checkboxes while increasing the checkboxes for everyone else. Now you have to leaving positions unfilled longer in hopes of finding a candidate who matches a gender. You've turned the hiring process into a broken mess and require 1000 times more candidates.
Let's say you are successful. Let's say a class of companies are successful at this strategy. Lets use the example of faangs which are desirable places in terms of salary/brand. If faangs were successful at this that would reduce the % of female developers in other industry and assuming faangs are taking the best candidates that leaves the worst ones. Which then creates this reality where male programmers outclass female developers in these other industries. That makes it harder for women in general and makes this false impression that females are not as good as males.
To help women you really need to treat them equally. Trying to reach a goal of unhealthy unnatural % industry wide means women will left holding the bag when the music stops.
That does sound frustrating, but I think you are conflating naive management practices with inclusive hiring.
ipaddr is right that he described the actual situation with FAANGs scooping up all of high caliber underrepresented minority candidates (think black/women ivy league comp sci grads with high GPA).
But this also creates a positive feedback loop when more and more women decide to switch industries and pursue IT/Engineering jobs via bootcamps, college degrees, etc. I noticed the number of female candidates in UX/UI, fullstack, QA, Data Analytics - has increased in last several years.
Partly because the demand is still high for these professionals, partly because there is entire cottage industry of bootcamps churning out IT specialists en masse, partly these diversity hiring practices that opened up doors for women
Yes and anyone with an ounce of technical hiring ability avoids those bootcampers like wildfire.
They churn out ego inflated beginners who think they're experts.
The mentality peddled by bootcamps to sell their wares produces dangers "engineers" in my opinion.
I have no formal education, I'm not coming from educational elitism here.
Regardless of whether you're coming from educational elitism, you're still making sweeping claims about big groups of people based on extremely limited evidence.
I'm a bootcamp grad, and would not have gotten into the field if bootcamps did not exist. I'm about four years into my career now, currently working at a major well-reputed tech company, and haven't gotten an average-or-below annual performance review yet. (And one reason for that is that I tend to be cautious, critical, and thoughtful in my technical decisions.) There are a number of other people from my bootcamp class with similar results.
I am making a generalisation, based on having interviewed over 200 developers in the past 2 years as part of technical screening.
80%+ of the bootcampers were rubbish and shocked to be told their knowledge was way below where they thought it was.
A classic is a 6 week JavaScript bootcamp grad claiming to be an "expert in JavaScript" (their words) and couldn't explain the JS type system or basics of variable scope. That was the norm. That kind of rubbish.
I'm happy you're an exception and everyone gets a fair chance with me, regardless of background, but I am never shocked when I have to bin yet another bootcampers CV
"Everyone gets a fair chance with me, regardless of background" is an _extremely_ different statement than "anyone with an ounce of technical hiring ability avoids those bootcampers like wildfire".
If you mean "in my experience, bootcampers fail technical screens at much higher rates", then say that, instead of implying that you're stupid if you even consider hiring someone who went to a bootcamp.
If I have a stack of 20 CVs and I can only interview 5, the bootcampers are most likely to get cut.
If I could interview everyone I would, and I'd happily hire a bootcamper that seemed excellent.
The reality is that statistical likelihood of passing screening means those CVs often hit the bottom of the pile.
The two statements I made aren't contradictory
I don’t think you really understood what I said.
If your goal is population distribution then you are inevitably hiring based on attributes other than skill.
For example, women don’t make up 50% of the engineering talent pool so if your goal is 50% women then you have to lower standards to achieve that.
if your goal is 50% women then you have to lower standards to achieve that.
You don't have to. But you'd remove them from the job market, making the pool smaller for other companies. IOW, a few companies can target 50% women, but that'll make it that much harder for other companies.
If you think we have a perfect way to measure (or even to define) pure skill I have a bridge to sell you ...
We're always hiring based on attributes other than skill. If we're lucky and purposeful, skill becomes a part of the hiring process.
Well, felt smart, anyhow.
No it is not.
You can exclude mediocrity while also being exclusionary on other axes.
They are unrelated issues. In fact, I’ve even heard of exceptional people being abused/bullied for belonging to the wrong group to the point of being told “you couldn’t have done that” which itself is an assertion of their supposed mediocrity for exclusionary reasons.
Don’t assume you need to be bigoted to exclude mediocrity. Discriminating, yes, but not discriminatory against groups that inclusive hiring policies attempt to protect.
This take assumes a priori that inclusive hiring results in mediocrity. Sounds more like a reflection of biases, to be honest. Inclusive hiring means expanding your search criteria beyond "hire those that look like me, speak like me, have awesome education like me, and are basically smart like me". It turns out there are plenty of exceptional people outside of that narrow band.
> have awesome education like me, and are basically smart like me
At least this is meritocracy, the kind of thing that people (e.g. eugenicists) can make a serious argument for.
> look like me, speak like me
...is something that can't be justified except by terrible people. Even worse is "likes the same music and movies that I do" or "we coincidentally have mutual friends."
> is something that can't be justified except by terrible people.
I have bad news for you about the people commenting in this thread.
> Building something exceptional always involves excluding mediocrity
This is wrong by definition
US army was exceptional in 1970's, did you need to graduate from harvard to join? No, they drafted everyone, even your sorry ass didn't want to join.
British industrialisation was exceptional, they didn't exclude anyone, even got children something to do by sending them into coal mines!
Amazon is exceptional, is it hard to become a worker in Amazon warehouse?
Organisation can be exceptional without any individual being exceptional.
Also you could be exceptionally bad!
> Project 100,000 soldiers included those unable to speak English, those who had low mental aptitude or minor physical impairments, and those who were slightly over- or underweight.
You are moving the goalpost from mediocrity to disability.
Subsequently the US set a legal floor of IQ 80 to enlist.
It's illegal to enlist anyone with an IQ below 80 to the military.
They don't have any use for them.
Approximately 1 in 10 people have an IQ below 80.
This misses 90% of what I, a startup CTO, find valuable in technical hires. What I want to know is: what type of projects have you worked on, how did you develop expertise in those systems, what level of ownership over your work did you display, how well were you able to plan and design the solution to a problem, and how did you handle the execution over the X months of work to make it go live. Demonstrate expertise, curiosity, and ownership. “System design” is like 5% of the work we do, and it’s important, but putting the designs in motion and driving value from them is 95% of our time and that’s something we do not screen well for. The way that I do this now is a process with a soft skills interview, a coding interview, then a “case study/system design” interview where I have candidates write a system design doc at home for a project they have worked on IRL and use that as a starting point for a 45 minute panel convo where we review the doc and ask questions about their choices and how execution went.
I suspect if I asked ten people to explain how they'd evaluate candidates for "curiosity" or "ownership", I'd get ten answers with little in common.
Is that 3 separate interviews for your startup? Or is it done in one session? If it is the former you're missing out on lots of potential candidates
Why would you say no to a job because they want you to interview for a few hours? As a candidate I always respect the employers more if they can put together a coherent 4+ hour interview. Why would I want to work somewhere where my co-workers were only briefly vetted?
It's not "no" to the job. It's "no" to the interview(s) process.
They're tied together. Once you join you'll be in a place that uses that interview process.
This is 3 interviews, done remote. 45 minutes per interview. Can be same day, can be over a week, candidates choice. Haven’t had any issues with people thinking this is a blocker, if anything it’s a lower load and faster process than people are going through at peer companies.
This is the right way to interview, I agree.
You want to see repeatable behavior and a general interest in going through the process. If someone takes the time to apply with homework and is able to articulate well, it gives you so much valuable signal.
Pressure cooker style interviews only reveal someone can remain focused under stress and that they studied their leetcodes.
I'd love a process like that, however it's much more common for companies I've interviewed with to not only not ask that type of info, but actively avoid and discourage talking about these things.
Been treated in past like I'm avoiding the "important part" of solving their quiz and that it's some softball topic.
It's not a bad post per se but we've been reading similar, anecdotal blogs like this about making the interview process kinder for decades. Yet the only companies in a position to do a rigorous statistical test - large tech cos - stick with the traditional, somewhat adversarial whiteboarding process. I would even suggest that a strictly technical whiteboarding process can be less biased than what the author describes, because you can so regularly grade everyone on the same exact rubric. That's tougher when "pair programming" or doing a take home.
Also, stop giving take home projects. Bad candidates will cheat them and good candidates will not even do them. If one of the random startup names listed on the author's site sent me a 12 hour take home project I would delete the email. Do you think they pay twice as much as the bigger company that only makes you waste 6 hours doing a whiteboard? I doubt it.
Larger tech cos with very high TC know that they will always have a pipeline of more qualified candidates, so a false negative has basically no cost to them, whereas a false positive has a significant cost. I think the reason that they run these processes is because they have a very low false positive rate, and so long as that is true, it doesn't matter to them how high the false negative rate is.
And I think that smaller companies copy this as a part of the tendency to copy large companies without thinking about whether the thing they are copying actually makes sense at their scale. In this case it can be very damaging, because false negative for a startup with a limited pipeline can be very bad.
Also, as far as I know, nobody measures false negatives in hiring. How would you even do it? Keep track of everyone you rejected, and then 5, 10, and 20 years later check on their career? I'd be fascinated to see the results of such a study. I'm sure it would be super valuable if you could find some kind of pattern where some filter is falsely excluding candidates that are actually great.
Hire a random sample of the candidates you were going to reject, and compare performance against the ones your process said to hire
I unironically believe that large tech co's would do this.
I think the goal of avoiding false positives is actually more important for smaller companies. A bad hire at a startup can significantly shorten the company's runway, while a bad hire at a bigger company tends to get isolated and managed out without doing much harm (except to morale of course).
You're right about the copycat behavior. This goes all the way to top of funnel: these small, even trivial-scale web application startups just don't have hard engineering problems. Many imagine they do, or imagine they will once they take off, but the work they're offering these high powered candidates they claim to want to hire is like, wiring up CRUD apps and making javascript buttons. It's not technically deep work, it's product work. A little humility about whether or not your tech startup is truly doing "tech" problems would, I think, fix some of the expectation/reality mismatch people are having when they complain about how hard it is to hire engineers.
And sure, lots of people join Google to work on world-scale problems and end up wiring CRUD stuff anyway. But they can at least plausibly offer some technical depth (or could anyway, perhaps Google's reputation as a great place to develop an engineering career has been fading).
(this is not to knock on "CRUD" but to highlight that a technical problem solver is an overlapping but not identical skillset to someone who can work with a fast moving team to quickly and reliably develop product changes)
I agree about the false positives. What I'm saying is that false negatives are more costly at a startup because you have less of a good candidate pipeline. Passing up on good candidates extends your hiring process and may even push people to hire someone you otherwise wouldn't have later because you need engineers. Google doesn't care because they will have 100 more candidates the next day.
A bubble sort at startup co is annoying but often less likely to cost the company millions, at Google or Amazon it very well could. Extreme example, but I think the point stands.
> In fact, you may be doing a disservice to yourself by filtering out slow thinkers or neurodivergent candidates that are likely to not shine thinking on their feet.
Yes, yes, 1000x yes. I was recently rejected after two technical interviews from a company that my former principal engineer that I worked directly under had referred me to. The position was to work under them again, which is why they referred me. The feedback I got from the recruiter was that it wasn’t the result they had expected, and I hadn’t achieved a specific number on their technical assessment. My understanding of this after some discussion was that the lower score was due to my speed in answering the leetcode style questions in the live interview with another engineer.
Here’s the secret I never brought up with the company while interviewing: In high, school, college, and for the CPA exam, I received accommodations for extended time and testing in isolation to reduce distractions from my ADHD. With those accommodations bringing me up to an even playing field with a neurotypical test taker, I was able to get into a good university, graduate with a bachelors and masters at the top of my class, and pass all four sections of the CPA exam on my first attempt. In the real working world, I have never needed extended time. I always deliver what is asked of me on time while I have witnessed neurotypicals show up to meetings with their work majorly behind.
I have always hesitated to bring this up with companies because I fear they will make the incorrect assumption that extended time on testing implicates that I will be a slower worker, which I have not found to be the case. I don’t want to introduce any biases for the interviewer to pick up. For whatever reason, testing with pressures absolutely slows down my thinking. In the real world, I have found when I face particularly tough problems, I find solutions after going on a 15 minute walk outside or while taking a shower in the morning. You cannot test for that style of problem solving in these high intensity algorithm technical interviews.
I certainly miss having a CPA license as evidence that I was a competent individual from my previous accounting career, which allowed all parties to skip technical questions in the interview and instead focus on fit for both sides. The software engineering industry suffers from too great of an emphasis on absolute performance levels in my opinion. To pass a section of the CPA exam, one needs to score a minimum of 75. What do you call an accountant that passed every section of the CPA exam with 75s? A CPA.
I have ADHD too and it is immensely frustrating how interviewers expect you to solve the leetcode questions very quickly.
Meta recently sent me their interview prep material and it says:
> In your tech screen, you'll be asked to solve one or two problems in under 35 minutes. Practice coding solutions to medium and hard problems in less than 15 minutes each to help you be ready for the constraints during the interview.
Less than 15 minutes?! The only way I can solve them in less than 15 minutes is if I've seen them before.
> Less than 15 minutes?! The only way I can solve them in less than 15 minutes is if I've seen them before.
I think that's the point...
It’s crazy that references don’t mean anything in most companies nowadays vs few years ago. That’s another ‘success’ of the overall hiring process with the various sites and ways to be referenced to faang, etc..
Now even if you have a living and respectable proof that you have worked with this person and was good, it means almost nothing. Let’s five rounds of interviews. I have seen the same pattern even for internal transfers! Crazy!
Agreed. I've also seen this a few times on both sides of the fence.
As a hiring manager, when I get a reference from a good report I tell HR to CT the BS and trim down the process. I talk to the candidate, have tem meet a few folks from the team and we're good to go.
As a candidate, when I come in as a referred candidate and you throw your 2 hour take home in my face I'm walking away.
At this point, being referred just means that you have a good chance that your resume is at least passing the first review, but that's pretty much it.
It's also frustrating even if it does matter. I was applying for a job where they informed me I may need to provide references, and I was having a hard time trying to come up with people I wasn't currently working with, who could speak to my skills, who I also thought were good enough writers or speakers to convey that, and were also someone I was willing to impose upon to do that for me.
I've been in the industry for ten years, but if I had to narrow down to the folks I'm not currently working with who worked closely with me, it's not that many people, unless we're going to people from 5+ years ago, and at that point, if I was on the receiving end of that, I don't know if I'd trust a reference that was that out-of-date.
References are worthless because they're used to pawn off bad employees to competitors. No HR department wants to risk a defamation lawsuit from an honest reference on a bad employee.
Why not just apply at companies that focus on take-home problems?
That’s my plan going forward. I’m in my first role as a self-taught engineer, so my network is still small. A coworker I worked closely with and had a great experience referred me to this company, so I went through with the process. Initially, the recruiter sent me a 2 hour take home leetcode challenge, which they waived after I responded I was surprised that they required an initial coding challenge for a referral.
It’s hard for me to tell which companies require those style of interviews, but I’m not working in the Bay Area and have found when applying to positions remotely there, they seem to have a much greater emphasis on leetcode than companies in my area.
Maybe you'll find this useful, I think I saw it elsewhere in this thread and bookmarked it https://github.com/poteto/hiring-without-whiteboards
What bothers me about tech hiring is that tech companies overthink it. To use a housing analogy, they act like they're signing a 30 year mortgage when they're only signing a 1 year lease. Engineers come and go all the time. At present, tech companies are laying off engineers by the thousands. Think of how much time, effort, and money was spent hiring those thousands of engineers! It's a giant waste. Premature optimization is the root of all evil, and that applies not just go writing programs but also to hiring programmers.
It's funny how they claim that a bad hire is devastating, and they can't rid of them easily, but somehow they can do mass layoffs and get rid of a bunch of engineers easily.
I wholeheartedly disagree! It takes months, sometimes even half that year for engineers to fully ramp-up on teams and integrate into the culture of a company.
Yes, you are expected to hit-the-ground running on day one, but no one will immediately operate at their full potential. Even with all the shared best practices in the world, the secret sauce is the part you have to learn.
As an employer it's very hard to know if the reason for someone's uneven performance is due to ramp-up or if they are just not a good fit. Without a rigorous interview process, so many months would be wasted waiting to get a clear signal on that person.
That also doesn't account for complete cultural mismatches that cause instability in teams and hurt the impact of your other employees.
Another implied reason, good engineers want to surround themselves with other good engineers. So knowing its hard to get into a company signals to each applicant that the other employees there made it through that process.
> It takes months, sometimes even half that year for engineers to fully ramp-up on teams and integrate into the culture of a company.
Maybe that's because companies tend to hire whiteboard-master generalists rather than subject-matter specialists who may not be great at standardized technical interviews. ;-)
Also, if the company culture is ultra-bureaucratic, maybe the company should fix that instead of wasting months on every new hire.
Seriously, if a new engineer can't commit code within the first week, that's a company problem, not an engineer problem. Of course their code shouldn't go directly into production, but that's true of any new code. Give them something small to start, like some bugs to fix.
> That also doesn't account for complete cultural mismatches that cause instability in teams and hurt the impact of your other employees.
Technical interviews can't determine this.
> knowing its hard to get into a company signals to each applicant that the other employees there made it through that process.
I realize that's a signal, but it's not necessarily a good or accurate signal. I think it's mostly PR and hype. Reminds me a lot of fraternity hazing. Google engineers believe they're the best, and some of them may be, but some of them don't impress me at all. And as I mentioned, engineers tend to move from company to company anyway, so if Google engineers are "the best", they're constantly losing the best too.
I cannot ensure laptop + software + permissions are established within a week. Let alone expect someone to be contributing code.
The best engineers I've seen tend to ramp up pretty quickly (1 month or less). Sure there are probably exceptions, but the ones I've seen that take 6 months to 'ramp up' end up with very low performance. I think there's just a strong correlation with learning quickly and doing a good job. It's probably less true the more senior you get, but the relative effect is still present.
> It's funny how they claim that a bad hire is devastating, and they can't rid of them easily, but somehow they can do mass layoffs and get rid of a bunch of engineers easily.
I hope you realized that this should answer your own questions. Layoffs may be (relatively) easy, but firing someone for "you're just not cutting it" is much, much, much more difficult.
First off, most companies are loath to do large scale layoffs unless there are strong economic reasons to do so - many of the FAANGs have never had layoffs as big as the recent ones. So if your only chance to get rid of bad hires is every 5-10 years or so when there's an economic downturn, that's a problem.
But more importantly, while it's generally straightforward to fire someone who's flat out bad (as there is usually plenty of data to emphasize why they're bad), firing someone for cause who is just kinda mediocre is nearly impossible in the tech world in my experience. For example, if someone can do the job, but say is 50% slower than your average programmer (I've definitely seen this), it can be extremely difficult to gather enough evidence to fire that person. And it usually sucks for everyone involved, because often times these people who are slow are hard workers, but they're just not as capable as their peers.
One of the reasons you see the behaviors you see in technical interviews is precisely because hiring a kinda-OK-but-at-or-slightly-below-par is basically the worst kind of hire you can make.
> firing someone for "you're just not cutting it" is much, much, much more difficult.
It's actually not. When upper management is motivated to fire people, they get fired fast. Whether that's an individual person or a large group of people. We've seen this happen over and over. Self-imposed bureaucracy is the only thing that prevents fast firing.
> it can be extremely difficult to gather enough evidence to fire that person.
You don't need evidence. There's no such legal requirement. It's at-will employment.
And I don't want to hear about potential lawsuits. These are ghost stories, designed to scare, but ghosts don't exist. Show me the lawsuits. Incompetent people who are suddenly out of a job don't have the time or money to file frivolous lawsuits (which could get them blacklisted from the entire industry). The ratio of lawsuits to firings is close enough to zero to be negligible, and certainly big tech companies can afford to defend themselves.
Yeah. In my experience mediocre hires never get fired and just kind of coast along forever. But they can be pretty harmful to team dynamics.
I've worked with 200+ engineers and I know of exactly four that were fired for performance. But probably another 40 were quite bad and we would have been better off without them, they just didn't exactly meet the bar for 'so bad we have to fire them immediately'.
Layoffs aren't that easy, and they are very expensive. A lot of the big tech companies paid the equivalent of 6-12 months salary (if you include the various stock etc). There's also the cost in organizing the layoff, which can be millions of dollars in consulting fees, the costs of decreased productivity and morale, etc.
> they are very expensive
> the costs of decreased productivity and morale
I don't dispute any of that. I just mean that they can legally do it, and they don't have to justify it, they don't have to put employees on PIP, they don't have to give reasons why every employee was included. I mean, Elon Musk can basically walk into Twitter and haphazardly fire a ton of people. The consequences may be bad, but it's "easy" in the sense that he can just do it whenever he wants. Even more so for individual firings as opposed to mass layoffs.
A lot of the commenters in this thread (and elsewhere on HN) flat out refuse to do take-home assignments, live algorithms coding, 4+ hours of onsite rounds, etc. Yet every interview I've ever done in the last decade+ with FAANG and FAANG-adjacent companies have always been like this. So where are all you interviewing that pays competetively without this "traditional" interview loop?
I will do live or take-home algorithms (~1h) and such for any company since they are fun and help me practice anyway if I'm in interview mode.
I will do take-home assignments (assuming 4~8h of work) or 4h+ onsite only if I am (quite-to-very) interested in the company. This is either the company is famous so I know them well, or there was a good interview process and they passed all of my questions/no red flags. If I'm on the verge of rejecting a company and they ask me for a sudden 4h+ process, sorry but not.
I live in Japan so it's been interesting as there's vastly different thinking companies, you have from the most modern flexible silicon-valley-like company (few, but there are) to very traditional ones that might even be confused when you reject them (again few, but some). Last time I interviewed I told a company I wasn't interested in their offer, only to receive an email later telling me they were not interested in hiring me. I could guess HR marking me as a no-hire was a lot better for that interviewer than marking me as rejecting them, but still made me laugh a bit of how much "no, I am breaking up with you" it sounded like.
> that pays competetively
I think this is the broken assumption—there are a lot of us who simply are willing to accept a sub-FAANG wage in exchange for a work environment/interview process that we feel respects us.
A lot of us aren't gonna leave 5x the wage on the table.
Uhh.. what? I've never had an interview process with a live coding session and whiteboarding that lasted longer than an hour, and my salary as a senior engineer has annually been anywhere from 150 to 200k. I might be leaving a little bit of money on the table but I highly doubt it's anywhere near 5x. Most FAANG level jobs aren't paying above 300-400k except at the most prestigious.
Was the whole process <= 1 hour, the total amount of live coding was <= 1 hour, or any session of live coding was <= 1 hour?
All the interviews at Google are 45 minutes, and when I interviewed only 2 had coding, so there was realistically 70-80 minutes of coding that day. I did maybe 15 minutes in a phone screen on an earlier date. Even if you did only 60 minutes for the whole process, you really aren't that far off from a typical FAANG.
Level 7+ at FAANG can reach $1m+/yr. Obviously I can't say that you would be able to reach that (most don't), but it's a possibility at least. Part of the difference is that non-FAANG companies often don't have as much of a path to these high levels for ICs so you are forced into management if you want to keep raising your compensation.
For anyone senior, it absolutely is.
Yup, it's all tradeoffs, and my preferred tradeoffs won't match yours.
That's a pretty wild tradeoff though!
Is it? At my current wage I make enough to pay for all my family's needs and retire at 50.
Could I use more money? Probably. But would I be happier with more? The research suggests I wouldn't.
Daniel Kahneman, the famous psychologist who authored the paper you're probably thinking of (the one popularized as "no increase in happiness above $75k/yr") actually just released a new paper a few days ago, finding that for the large majority of people, happiness continues to rise with income far past that point.
A big issue with any analysis of happiness is that it's rated on a "capped" scale. For example they ask you "how happy are you 1-10". However income is "uncapped". That means there will always inherently be a sublinear relationship between the two in any analysis, because you are compressing the top end of happiness.
I think for most people amounts up to something like 200-500k/yr (depending on COL) would provide increased happiness. Basically, if you ever have to worry about not having enough money, you could stand to make more.
Of course that doesn't mean it's necessarily worthwhile to work more to achieve that, that would be a personal decision you have to make yourself.
That's a quite an interesting paper! That said, the final results still line up with my intuition, which is that there are diminishing returns, as evidenced by the log(income) relationship that that new study identifies.
I already make a 90th percentile income for my area, and I don't feel that investing additional mental and emotional resources in maximizing salary is the best route forward for pursuing happiness. I think that at this point there are other axes to optimize on that provide greater marginal gains to happiness.
Presumably the tradeoff is more along the lines of retiring at 40, if you don't care about the money.
Fair, but I actually enjoy my work (one of the things I feel like I have gained by avoiding FAANG), so retiring at 40 doesn't have that much appeal.
It's true, I could afford to retire now (I am <40), but I expect to keep working until I'm completely unemployable (or physically incapable). It's just more fun than anything I've found in my free time. I do work at FAANG btw so it can't be that bad ;).
Most people on hacker news and most people in tech do not work for faangs. These salaries of $250,000 or $200,000 or even $150,000 seem unreachable. But these faang salaries have *.
You signup for a new Amazon job. You are a senior developer you expect to make $400,000 with the stocks/salary. Your base outside of California is 139,000 or 129,000. After year 1 only 5% vests.. after year two 15%.. the average employment length is 1.5 years. So you end up with $140,000/150,000 for working 16 hour days. If you manage to stay 10 years you could retire..(you have to because at this point you hate life) but they don't want people staying at the same level so you need to get a promotion when the 4 year vest up or you will be at your base. Getting one takes the right project and is hard and requires a breakthrough project.
Most people 95% of developers never worked at a faang and those who have, on average worked for 1.5 years. Very few are still employed or seeking faang employment. Faangs make popular entry level position but very difficult to keep for life but if you can survive many years you usually leave the field or create your own startup because of burnout. Faang adjacent companies can be the worst of all worlds same issues worse pay/upside.
I would guess <50% of commenters here are even professional devs.
We don't want to do it but sometimes we do if it means getting to work on interesting things/paid competitively.
I've had interviews with heavy LC and ones where I got plain old fizzbuzz and there wasn't much difference in staff competence or how quickly we delivered.
If anything, the place with the low bar had more well rounded peers I wanted to spend time with after work.
I'm in non-FAANG tech (biotech, DoD) and no one does take homes or live coding, even for code heavy jobs. Our comp isn't in the 200k+ range, mostly due to location, but it's pretty good all the same (120k+).
Where I'm from FAANG pays the average for the market salary. They might be more attractive as a bullet point on the resume or because of various other perks (both related to the job directly or not at all).
In the world outside HN I very rarely encountered people who'd turn away from any kind of hiring process. Maybe one in fifty candidates?.. I don't have the numbers, but I think I only met such people twice in my life.
I bailed from interviews for different reasons, but I think that homework is a legit way to test someone's skills, so I wouldn't mind that.
The reasons I cut the hiring process short in my job hunts were most commonly:
1. Employer is an MS Windows shop. Sometimes it's hard to figure this out from the job posting.
2. Employer requires employees to use company-provided tools s.a. code editor, or antivirus etc. In other words, an over-reaching IT.
3. Crazy / not very smart / borderline criminal employer. Examples include a guy who had "scrum cards" deck on his desk and essentially showed me to the door when I asked if they used this stuff for real. Another one who couldn't get my homework to run, asked for a Docker image, couldn't run that either, asked for a VM image, couldn't run that either...
----
There's one litmus test I have when interviewing that turned out to be surprisingly precise, and I don't know why. I ask potential employer if they ever use git-merge. If the answer is "no", the company turns out to be intelligent people who are nice to work with, and if the answer is anything else, it turns out to be dysfunctional in more ways than just infra. They will have toxic culture, under-the-carpet skirmishes where each department undermines another department, while at the same time trying to do as little work as possible.
As you can imagine, unfortunately, I had to take jobs where the employer answered "yes" or "sometimes" etc. That's how I know :(
I'm confused... What's wrong with merge?
For all I know, I once joined a company where the policy was to only do squash-merges. I left from there at the brink of mental breakdown.
What's wrong with squash-merges? As long as your merge requests aren't enormous, squash merges keep your commit history clean, instead of being polluted with tons of commits with the comment "fixed typo" when most of the real work is in a single commit or two. When I'm reviewing someone's MR, I do not want to wade through dozens of very minor commits that are probably already fixing the things I would have complained about.
And what's wrong with merge? How do you even use git without merging?
> what's wrong with squash-merges?
In theory I don't mind them. In practice I've found that I more often encounter a scenario where I wish a change had been split to smaller commits rather than the scenario where there were too many commits to go through.
Anyway... I intended my example as an explicitly anecdotal evidence to counter the seemingly absurd suggestion of using Git without ever using merge. Feels like going back to subversion or CVS.
All companies I worked in for the last 10 years used squash-merges exclusively, combined with trunk-based development and small commits. I don't see anything wrong with that, it's nice to have a history where every other commit is not a merge commit.
Merges make it prohibitively difficult to go back in time because they create alternative versions of history. There are many negative aspects to this, but I will use git-bisect to illustrate the problem.
For more reliable code-bases you want to do the following:
* Run tests on each commit when accepting PRs.
* Be able to remove or edit intermediate commits, if you find that they've created problems afterwards.
* Only have one path from past to the future.
This is so because if want to use git-bisect, and instead of deleting faulty code you reverted it, the command will keep failing on the code that you've already fixed, and there's nothing you can do about it. git-bisect also has to follow one and only path from the past to the future because if you don't, then, at best, you get an combinatorial explosion of possible paths git-bisect may take, and at worst, some of these paths will fail, but others will not.
So, what ends up happening is this: people who use merges are like people who never clean up their apartment. For some it will take longer, than for others, but, inevitably, the apartment will become a filthy mess. But this is just a symptom of people being afraid of not understanding their code, being afraid of making big changes, undoing things committed to long ago.
This fear is usually an indication that people aren't good at the technology they are using. They would be too afraid to delete code because "what if it breaks something?" -- and nobody can tell authoritatively "no it doesn't". In a situation like this any change in technology s.a. using a different version of the same tool, or replacing the tool altogether will be almost impossible to implement because of the fear.
It's also usually very characteristic of places like this to be afraid of knowing / learning the underlying technology, the one that supports the entire company's stack. Eg. if it's a Python shop, then they'd be opposed to writing Python modules in C, even though this is how Python typically works, because they are afraid that they won't understand this code and one day will end up with a "magical" program that sometimes fails, but nobody knows why.
It's also usually the people who won't even try an unpopular technology, even if the benefits were huge, based on their fear of not having expertise to deal with it. Eg. XML schemas are hugely superior to JSON schemas, and if you want to validate your inputs, XML is just a better tool for this, but the company I'm describing will never consider using it because they are afraid of not being able to find people willing to work with DTD / XSL / RNG.
Such a company will never consider self-hosting, and will pay through the nose for the expertise of others, being mortally scared by a prospect of running their own infrastructure.
----
And... this is the majority profile. The problem is, this is not a winner's profile. It's a scrapping-by profile. It puts an individual programmer in the situation where there's no need and no reward for bettering themselves. Where management is antagonistic to programmers because they are in a conflicting situation, where on one hand they want to give customers more stuff, but on the other hand they are too afraid to make more stuff, since it may deprive them of the stuff they already have. So programmers are punished whether they do or whether they don't. It's where cargo cult flourishes. Basically, Dillbert comic before its author went into politics.
I refuse them. If I'm looking for a job I don't have half a day for some pointless thing. Interview me in the standard way, a few hour long interview, one at least with coding. When I interviewed at Microsoft and Google a few years ago, take home assignments were not part of the deal.
Last 10 years no take home. Today it's coding something in a vc meeting, maybe in a web browser or coding env. Maybe beginners do some coding.
Despite the HN bubble, most of the 2.7 million developers working in the US in banks, insurance companies or other “enterprise developers” will never see a total compensation amount (inflation adjusted) above around $170K in most of major cities in their life.
These developers may go their entire careers without ever reversing a binary tree on a whiteboard while juggling two bowling balls on a unicycle.
> A lot of the commenters in this thread (and elsewhere on HN) flat out refuse to do take-home assignments, live algorithms coding, 4+ hours of onsite rounds, etc.
Well, I personally have always refused to do take-home interviews but happily will do live coding and systems design interviews.
For me it's about respect and power imbalance. A company asking me to do work without them putting in equal effort sets a tone for a culture I personally don't ever want to be a part of.
Like I find take-home interviews disrespectful.
Time-bounded interviews with an interviewer also there (aka FAANG style onsites with 4 hours of interviews) is far and away my preferred process, especially if I can do them all at once. One problem I've seen in a remote friendly world is companies wanting to spread the interviews out over multiple days.
I won't do anything that requires more time from me than it does for them, simple as that. If they think it's worth it to waste 10-20 man hours for a day on sites, that's concerning but not disqualifying.
> So where are all you interviewing that pays competitively without this "traditional" interview loop?
You see people on HN balk at 500k+ engineering jobs even existing, so I think that's your answer.
Personally, either regular tech companies in Europe who tend to be more relaxed with hiring ... but also don't pay the same astronomical salaries FAANG adjacent companies pay.
Or startups. :)
Are there any other industries that do this? Do cabinet makers ask you to do like a full day of work for free?
Of course.
My buddy just got a job at a high-end cocktail bar as a bartender. Part of the interview process was asking him to mix a drink.
Gordon Ramsay has talked about how he'll interview chefs by asking them to make scrambled eggs.
Actors, even famous ones, generally have to 'read' for roles in order to land them.
Musicians interview for seats in symphonies by playing music.
MBAs have to do case studies to land jobs at high end consulting firms.
Hell I applied to Taco Bell as a kid and they made me take a short math test to prove I knew how to make change.
I could give similar examples for dozens of other jobs.
The cases where you don't need to demonstrate some skill in order to get the job generally fall into a few categories:
- There's some outside certifying body like the Bar, CPA, PE, various tradesmen unions, or all the licenses like a CDL.
- The jobs are undifferentiated so the workers are fungible (no special skills required).
- Job skill is immediately apparent (less than two weeks to know for certain if someone can do the job or not).
- The cost of a bad hire is low so you're willing to eat the cost and just cut the workers how don't work out.
The difference is it doesn't take 8 hours to make a drink or scramble eggs, it takes as long as an interview usually takes. So the equiv. there is not a take home project, it is a in-person coding question. Or getting them behind an IDE and having them type some stuff.
Lately it seems take home projects are more and more common and these do take 8+ hours. If you want the job.
I don't think many of those are 4+ hours. Famous actors are not made to wait 4 hours to read lines, for example.
Engineering and medicine require a lot more investment to get the job. Especially medicine. It's a lot more regimented, but to put things in perspective: to be hired by a hospital, a doctor needs to do stage (few months in rotations between different departments) and then residency (36-hour shifts with 12-hour sleep intervals for a few years). And then they become attending, maybe will try for a fellow, but, basically, they become a "proper" doctor, and get a permanent contract from a hospital. Sure, the procedure varies between countries and hospitals, but the process is a lot lengthier and exhausting than anything you can possibly face in programming. Hell, even karate kid punching trees with bare hands for a year fades in comparison.
You described a process required to became a doctor, not to get a job at a hospital. You don't need to do any of this when you want to move to a different place.
Not true.
Here's what happens if you want to move to a "different place".
Say, you go to a different country: you have to spend upwards from a few month, but likely few years to confirm your degree. You won't need a stage, but you will not become an attending right away. In many cases there aren't even analogous positions if you move countries, unless medical systems are very similar, so, in most likelihood you will have to do at least a good chunk of residency training all over again. This will also be usually compounded by studying a new language to a very high degree as doctors are expected to produce a lot of written reports / engage in written communication, and, unlike programmers who almost universally use English regardless of the country they work in, doctors absolutely have to have good command of the local language(s).
Similarly, if you move between different medical organizations which manage hospitals. Sans the language requirements. However, within the same country hospitals will usually be more similar than between countries. Anyways, most hospitals will have fixed dates when job applications are processed, and even if you are extremely lucky and you don't need to redo any of your previous residency (both systems use the same PACS system, same or very similar internal organization etc.) you will still have to wait until the "draft" date. Typically, and due to competition, doctors will go to the hospital they intend to work at anyways before the "draft" date.
Even within the same hospital, if you want to move to a different department, you will still do residency, at least in part. I.e. say, you were already an attending in internal medicine, and you want to move to radiology: then maybe instead of 4 years, you'll do 3 years residency.
----
The above has a lot of compounding factors. Huge waiting times to get a position lead to doctors holding on to their positions with a lot more devotion than programmers. In many cases it's a job for life.
Because hospitals have to be in geographically diverse areas, they cannot, like programmers, all bunch together in one or two cities in a country and jump jobs w/o moving to a different apartment / house. A lot of hospitals thus include accommodation programs, which make it even harder to switch jobs.
It's very common for doctors to marry doctors. This makes some things easier, but it also means that if you need to switch jobs, then you have to do it in lockstep with your spouse.
Not in the least, if you move from a "less prestigious" country to a "more prestigious" country, you are almost automatically downgraded in your rank, and if you want the equivalent job, you'll have to jump through the same hoops the second time.
> Here's what happens if you want to move to a "different place".
All of this is still not true in a most simple case, so getting job at a different hospital in the same specialization. Ex. in Poland most doctors are hired in multiple hospitals at the same time.
> Ex. in Poland most doctors are hired in multiple hospitals at the same time.
Genuinely curious how does this work? Do they get paid per the number of hospitals who hired them? How do they go to work? How do they know what hospital to go to?
PS. My wife is a doctor, and I had to live through what I described. So, none of that is invented, it's just what I see happen to her and to her colleagues. To make this more concrete, she was an attending in emergency department and wanted to switch to radiology. In her case this resulted in the full 4 years of study on top of about half a year of just showing up in the hospital and tagging along with the radiology team. (This was in Israel, one of the central hospitals). One of her colleagues was a transfer from internal medicine (also an attending), and he was doing 3 years of study to get into radiology. Another was a Russian emigrant doctor with about 10 years of practice from a hospital in St. Petersburg. He was also doing a 3 year of residency.
They also had two people drop out of the residency just during the year my wife was there (before she gave it up), and that's out of a group of six residents. One was a Brazilian emigrant, who eventually decided to go back to Brazil and another one was a guy who was an Israeli, but received his degree in Romania, which was cheaper, I guess. He just couldn't pull it up, and eventually was let go from the program.
The Russian guy was also on the verge of leaving due to some bad blood between him and the head of the department. The head was actively trying to sabotage him and make him leave for god knows what reason. The Russian guy though, despite having some sort of a chronic illness was spending multiple days in a row w/o leaving the hospital.
I mean, back to my original point: I saw nothing that could come close in the programming world. And the fuss people here make about home exercises is just a sign of being way, way overly privileged compared to the majority of the workforce. By which I don't mean to say programmers should suffer like everyone else, rather everyone else has to get better conditions. It's just of all people, presently, programmer should probably show more comradery with other paid workers instead of complaining about their own issues.
> Genuinely curious how does this work? Do they get paid per the number of hospitals who hired them? How do they go to work? How do they know what hospital to go to?
They have duty schedules, so they know where they should be at a given time. They have contracts signed with each hospital, like any employee. They can also work in private healthcare at the same time. It's just the case of setting up a schedules so they won't collide.
> she was an attending in emergency department and wanted to switch to radiology. In her case this resulted in the full 4 years of study on top of about half a year of just showing up in the hospital and tagging along with the radiology team.
That's normal because this is a specialization change. Not many doctors change specs or have more than one in most cases, at least in Poland.
I imagine you get paid for all that BS ...
I can totally see this: can you whip up a quick breadbox while we stand here and comment on your every move? You've got to use our toolbox, work with unfamiliar materials and no measurements are allowed. You've got one hour. Oh, and we'll keep throwing in new requirements along the way.
I'm not even trying to touch on the "in person interview" stuff, more of the take home and spend like 8-16hrs on the project type stuff. Which is just crazy. That's more like "build me this table to get the job" type of thing.
That would be reasonable if you were hiring an employee, since they would have to work in your workshop with your tools.
Of course many carpenters work as contractors so that's why it seems a bit silly.
Academia will easily ask you to do a lecture.
To real students? How long of a lecture? Do they pay you if to real students?
A real full lenght lecture, not to students but to peers. Not paid.
Academia also has the concept of tenure, which is basically "you're evaluated each year and easily fired" - if they don't like you you don't come back. This makes sense to me!
How long is the lecture? 1hr? That doesn't sound bad compared to a 20hr programming assignment!!
You should mention, what kind of teams you've worked with and what kind of stuff you've built. Every team has different requirements and hiring bar. In my previous company the low bar caused not so good (able to understand stuff, knowledge and connect dots) people be a burden to rest of team. Heck in 2 years, 4 important people have left the team due to hiring a bad manager (has neither tech nor soft skill(s)).
There wasn't growth in that team due to mediocre hiring and eventually all the good ones - left to other companies.
My current team is an infra platform and has lot of growth as IC. Everyone is learning something in-depth and are explorers - rather than blind sheep. The bar here is higher than the one for my previous team.
Our team requires you to know about whatever you talk on, not just usage but it's internals - why ? That's what we do daily. It can be about scheduler, checkpointing, auto scaling, concurrency, different data structures & algos, integrating with ecosystem, etc.
Even soft skills - like helping others, taking feedback, communicating clearly, etc.
Yeah so, mediocre will always be a burden to team.
> Google: 90% of our engineers use the software you wrote (Homebrew), but you can’t invert a binary tree on a whiteboard so fuck off. — Max Howell (@mxcl)
If inverting a binary tree means swapping the left and right subtrees of every node, I wouldn't want to work with someone who can't do that either and Google is definitely right to reject him.
The actual work for which you are hiring an engineer is building a software product/service, and the Homebrew developer has a track record of delivering great results.
Rejecting the guy because he cannot do a whiteboard brain teaser is like rejecting LeBron James because he did not make a shot at the arcade basketball game.
I'm not saying the guy would be perfect. Comparing him to LeBron James might not be a great example. Google might have other reasons to reject him.
What I'm trying to say is the current coding interview is a really poor mechanism to gauge a software engineer, especially when it comes to hiring one with real-world engineering experience.
Maybe he would have been a poor fit at Google. Just because you wrote some popular open source software doesn't mean you will succeed in a corporate environment. The dynamics and skills required are quite different.
Now does being able to reverse a binary tree mean that you would fail at Google? I have no idea. But we don't really know if that was the reason he was rejected, it's just his own guess. There could have been other reasons.
> What I'm trying to say is the current coding interview is a really poor mechanism to gauge a software engineer.
People like to say this, but in my experience this is not true. It's just that people misunderstand the goal of technical interviews and they often are poor at evaluating their own skills.
First off, these giant tech companies have enormous economic incentives to improve their interview processes as much as possible. They also do a pretty rigorous assessment of the effectiveness of their interview process (Google, for example, has publicized some of their data). I'm not saying these tech companies interview processes are perfect, but I also have a problem believing they're so fundamentally flawed that these companies can't figure out how to fix them given the giant economic returns they get for optimizing their hiring processes.
Moreover, as some other comments mentioned, many companies (and individuals, myself included) believe it is much worse to hire someone who ends up not cutting it, than missing out on a potentially good hire. I can list out all the reasons why, but Joel Spoelsky has a pretty famous essay from a couple decades ago on the topic that explains it well [1].
Thus, it's not surprising hearing a lot people complain that they can do the job, but they aren't good at interviews. Because, from Google's/Microsoft's/etc. perspective, they're fine with a bit higher false negative rate if they can greatly reduce their false positive rate. And my experience matches that: I have never seen a candidate who did awesome in "whiteboard-style programming questions" who couldn't cut it programming-wise (they may have had other issues, but "coding productivity" wasn't one of them). Now, I certainly believe and have seen that there are some people who aren't good at these questions who can do a job well, but there are also a ton more people who can't do the job if they can't pass a technical screen, so hiring any of these folks means much more risk.
I also think that whiteboard-style coding questions help show a quality that is very important to businesses, even if those questions don't represent "real world" work. There are basically 2 types of people that do well at these questions: people who are just naturally smart and have a ton of experience to the point that they wouldn't even need to study to do well, and people who are of more "normal" intelligence/ability, but who can do well if they study a ton. Either of those two groups would likely do well in a programming role. So often I hear the complaint "I'm a busy person, I've got outside responsibilities, you can't expect me to spend all this time studying". And that may be true, but you'll be competing against people who are willing to study, so I don't think you can fault Google et al for favoring people who show a willingness to do more preparation.
1. https://www.joelonsoftware.com/2006/10/25/the-guerrilla-guid... "And in the middle, you have a large number of “maybes” who seem like they might just be able to contribute something. The trick is telling the difference between the superstars and the maybes, because the secret is that you don’t want to hire any of the maybes. Ever."
> First off, these giant tech companies have enormous economic incentives to improve their interview processes as much as possible.
But we have examples where the companies themselves have admitted that their past interview practices turned out not to work: https://business.time.com/2012/10/23/no-brainer-brainteaser-...
> They also do a pretty rigorous assessment of the effectiveness of their interview process
Companies do review their hiring processes, but actual experiments and data seem fairly rare. It's harder than you think. What experiment would you run? Hire a group of people entirely randomly, and compare their performance reviews after 2 years?
> But we have examples where the companies themselves have admitted that their past interview practices turned out not to work: https://business.time.com/2012/10/23/no-brainer-brainteaser-...
That's pretty much exactly my point. In the 90s, wide-scale hiring for software engineers was a relatively new thing - many companies were just figuring it out. And so they did some shit pretty early on that didn't make sense. But for all the times I hear folks pulling out the "Why are manhole covers round?" and "How many cars are there in Manhattan?" examples, I haven't heard these types of brainteaser questions being used for nearly 2 decades.
I'm not arguing that the FAANGs have some perfect, unassailable interview process that can never be improved, but I am arguing that so often I hear grumbling discontent from people who don't like the interview process, but rarely do I see much examination around why those particular hiring processes appear to work fairly well for the likes of Google, Apple, etc.
Those puzzle questions weren't just used to hire software engineers. Hiring fads sweep corporate America, despite no evidence of effectiveness, and despite economic incentives to hire well.
Yes, they did move away from it, but that doesn't mean we aren't now in the grip of equally bad fads.
> but rarely do I see much examination around why those particular hiring processes appear to work fairly well for the likes of Google, Apple, etc.
You assume they work well, but you don't have any data to support that. That's sort of assumption is basically where these hiring fads come from.
Not knowing how to do something from the top of your head is not the same thing as not being able to do it.
… but that's an interview. If you cannot, within the time, demonstrate any ability, why should you be hired?
The question above, as clarified, is not complicated, nor does it rely on memorization or some "trick": anyone purporting to be a SWE should be able to write an essentially de novo solution to it.
(And in my own technical interviews, there are multiple questions, to specifically hedge against any one being "that one question a good candidate is going to miss because it's just not their day". It doesn't happen: it's either all or nothing.)
> If you cannot, within the time, demonstrate any ability, why should you be hired?
It is more likely that the interview process is broken and missing the right candidates, than it is that the interviewees are all mediocre. Most interviews are very non-inclusive the same way that the main track of school is becoming less and less inclusive. Different people need different methods to bring out the best in them.
> It is more likely that the interview process is broken and missing the right candidates, than it is that the interviewees are all mediocre.
Here's a thought experiment for you: if the interview process is so broken, why hasn't some tech company succeeded and become famous for an improved interview process, e.g. "Moneyball style"? My guess is because the process is not actually that broken, at least from the employer's perspective. I'm sure the interview process could be changed to be less regimented and more "inclusive", but that's also likely to reduce it's predictive power (i.e. you're more likely to make bad hires, and from a company's perspective that's almost always worse than missing out on a great hire).
> if the interview process is so broken
Hiring is guessing. Firing is knowing. If the hiring process worked, we wouldn't have layoffs like we do.
The layoffs that I have seen reported have been reported as being random. I've been involved now in 3 layoffs directly in my career, and 100% of them, the laid off individuals were laid off without regards to skill. The reporting in the media on layoffs happening elsewhere largely matches my experience.
Sure, an argument exists around "you shouldn't've hired that many people", but that is different from an argument of "the hiring process can't discern good hires". The former is a management & long-term planning issue, the latter is how interviews are conducted.
I'm on the same page.
Sure, your day-to-day work may not involve manipulating binary trees. But presumably it does involve working with variables, objects, references, manipulating data of some kind... And if you're comfortable with the fundamentals of those, then this is something you should be able to figure out even if you've never heard of a "binary tree" before, once somebody has sketched it or shown you the definition of their TreeNode class, right?
It honestly baffles me how people consider this something which needs to be drilled or memorized.
There are absolutely algorithmic questions which would fall into that category. But if somebody considers this to be one of them - or something like "find the smallest number in an array" - then I have to question whether they have an understanding of the most fundamental concepts in programming...
Or if they get through each day solely using things they've memorized by rote, or looked up, and they don't really have any idea how any of the foundations they're building on actually operate.
So hire a baby ? That'll eventually do in 20 years ..... works, right ?
That is a fine and common opinion, but just one question: how often have you inverted a binary tree at your job? Because after nearly 20 years it hasn't come up once for me. I am sure for some roles it is a necessary skill but my issue is that most of these questions are more or less toy problems that come from academia and not business. They are a great test of your retention of a data structure class but not super relevant beyond that.
I would rather hire an engineer with a strong business or user sense - reading between the lines of requests and anticipating future issues or uses adds so much more value in a real sense.
To me, these are great entry level questions because it is a good baseline for new grads when you have little work experience to judge. Past that, it is like making a lawyer take a mini bar exam for every new job - a waste of effort if you want to hire for specific skills and experience.
Just because you haven't worked in a team that requires those skills doesn't mean they aren't valuable.
In my old team, I had to come up with a coupon distribution logic based on count, percentage, time, then generating reproducable random values that required to deep dive (algorithm) into library code & explicitly storing state in redis, then an application of dynamic programming in building as custom platform, atomic token validation, custom rate limiter algo, state machine, scheduler, distributed circuit breaker, etc.
In my current team, I had to read raft paper, zab paper, look into their implementation, make a poc with raft protocol, then autoscaling algorithms, scheduler algo's, different data structures, heck even the oss engine itself is DAG, heavy threads + concurrency stuff. Even now I come across new data structures and algorithms.
Clearly you don't know the entire industry, just because you haven't worked in such teams, doesn't mean these aren't important.
You are experienced in a bubble. The hiring bar for our team is higher than other teams and heck even for SDE3 - the requirement is higher. You would be very much surprised to know that even the senior members have research publications and deal with complex stuff.
Core teams like in AWS or GCP or Azure solve these sort of problems.
Who do you think will solve autoscaling (that's what I'm doing now) or managed scaling or network or security or any infra problems in these cloud platforms ?
As experience increases, we expect more knowledge & insights - doesn't mean to ignore basic coding stuff like arrays or linked lists or trees or graphs or simple message queues or etc.
If companies are paying competitive TC and there are multiple candidates, why not hire a smart person ? What's so special about doing regular normal stuff ? That's just a normal dev right ?
None, but I have written code that requires more data structure and algorithm knowledge than inverting a binary tree (at my job).
Sure, but job isn't to write Homebrew either, so it just seems like a bad example all around.
(I'm not sure Homebrew is all that well engineered, actually. Hard to tell, but I've had trouble with it and avoid it.)
I think what it comes down to is that nobody really knows what to interview for.
Just last week I had to implement a common graph algorithm from scratch to solve a business problem (basically topological sort, with a minor variation). It's certainly much harder than reversing a binary tree (which is one of the easiest possible interview questions you could imagine).
Generally it wouldn't really make sense to reverse a tree in practice (why not just build it the other way initially?) but it has a similar structure to other tree traversal things that could actually come up so it's reasonable to ask.
For every one of you, do you think there are more engineers doing things just like you or do you think there are more engineers who never have to do a binary sort and can still lead a perfectly satisfying, fulfilling career?
So if you are just doing normal stuff, then why should anyone pay competitive TC ? That's just a normal dev, right ? What's so special or talented about you ?
Companies that pay huge TC want to hire smart people not just an average joe. Sure, you can live satisfying career and that's your pov.
What about a company's pov ? Did you ever think about it ?
Wouldn't that be a "mirror" operation, while inversion would be (I dunno) swapping the direction of the edges?
I went out of my way to avoid homebrew (still do) when I worked at google because it would reliably fail to complete some key operations in a dag, hence the interest in ensuring developers know how to do CS things.
Yeah it's not clear what he was asked. Swapping the direction of every edge of a binary tree would result in a DAG that is likely no longer a binary tree though.
Are you confusing a binary tree with something else?
Here's an example:
https://leetcode.com/problems/invert-binary-tree/
It's an 'easy' question. The solution is <10 lines.
Yeah it's really easy. I wrote a correct solution in about 30 seconds on my first try. And I haven't actually practiced these problems for 10+ years.
"Mirror" is what this question is generally understood to be asking. I don't think swapping the direction of the edges produces a tree in general although it would produce a DAG. I think "invert" could mean "mirror" or turn upside-down depending on the context.
Personally, if I were asked this, I would just say "convert the graph to a matrix, invert the matrix, and then convert the resulting inverted matrix back to a graph", and let them try to figure out if that would work for a bit before joking "oh come on, preorder traversal with a temp var, do you have a more interesting question?"
It would be more leetcode to be given an ordered binary tree and asked to reverse it O(N). It's a lot more fair to the interviewee to be given the explicit task without knowing 'the trick' unless one considers knowing recursive functions to be a trick.
There you go ! This comment should be highlighted.
It doesn't matter, the candidate should ask for clarification if the question is ambiguous.
I was referring to second para or statement. (-_-)
Most of us havent touched a tree structure since college, because there are other, real, problems out there. Trying to remember, or rederive it from scratch is slower and error-prone and bad for interviews
You are just ignorant and naive.
Tree structure ? json, xml, protobuf, classes, functional programming, databases with foreign key, database internals, etc ?
Oh I forgot you also serialize and deserialize data - did you forget how that works ? Tree traversal again.
Do you know how organizationl hierarchy is structure ? It's a tree.
Do you know various maps and their usages ? We use them daily - it's very very important to know their internals. Hashing vs Trees vs Linked hash vs etc.
Google maps ? n-d trees ? Comparing data - merkel trees ? etc.
Every dev out there has common work with mine. But you won't be able to solve the problems that I face on a daily basis without thinking hard & without this dsa + concurrency knowledge.
Now, is it reasonable to ask these questions ? Heck yes.
JSON is also a tree structure. Granted one rarely inverts it, but “no trees in the real world” is not true.
Then most of us aren't suitable for jobs where this stuff is important. Google does have 'real' problems involving trees.
Also, I don't believe that 90% of Google engineers use Homebrew. I'm not sure I'd believe 9%. Google is a Linux shop with its own internal package repo. Even if you're using a Macbook to work remotely, you're using it as a fancy terminal wrapper to connect to a Debian-based system to do your real work.
Yeah it's such a basic structure we use in-directly - in classes & sub-classes, intellij dependency list on left side, using google maps, etc.
How complex is homebrew ? Can no one else replicate it ? Why should a company hire for something you did that's simple ?
What are the skills he posses that no one else has ?
Learn your basica dsa stuff for gods sake people.
Judging by Google's track record, hiring some people who have demonstrated an ability to launch and maintain a piece of software would bring a skillset they desparately need. Google engineers are incapable of keeping much going for the long term.
It is managers who decides what project dies or lives, not engineers.
That's a problem in every company ? How do you know they don't need smart people ? Do you've any public stats or reasonable justification ?
Since it mentions the infamous interview challenge, I'll ask: has anyone ever "inverted" (i.e. swapped left and right recursively) a binary tree in production code?
I can't think of any reason why anyone would ever do this. Just navigate the tree in the reverse of your normal direction instead.
Why not ask the much more interesting and potentially useful question of balancing a binary tree? Or do something else recursive, if that's what you're after.
> Why not ask the much more interesting and potentially useful question of balancing a binary tree?
Or even just when would you use a binary tree? Figuring out which data structure is appropriate for the problem at hand is the hard part, how to implement operations on the data structure is easy in comparison, you can just Google it.
> Or even just when would you use a binary tree?
That seems to be the wrong question though. It seems to be jeopardy style "question". You are not asking which data structure is appropriate for a problem.
Here is the answer, but what is the problem it solves.
Never in my live have I sat down and said: I don't know what problem is I need to solve, but I know the solution is a binary tree.
You are just ignorant. You should read about distributed systems. Google around for Zookeeper and it's zab paper.
Every (recently with raft protocol) multi master distributed system out there interacts with Zookeeper for assigning leader and maintain configuration.
Do you know how the syntax or api calls look like ? They are node path in a tree. You want to store something ? That's a tree path again. You want to listen to some change ? It's tree path again.
As I said, most people are ignorant here and don't do "true" computer science engineering in daily life.
Most devs simply convert business logic into bunch of apis + adhoc implementation.
Did you ever work at a banking firm ? I've read their codebase - they are structured as trees, every damn thing is tree. It's a headache to navigate, code, heck even the objects are literal trees.
As I said, people are ignorant and think world revolves around them.
It's not really the point of the question. The LC interview is basically a standardized test to eliminate bias as much as possible. If you prepare, you should be able to solve it. "Invert a binary tree" is actually considered an easy problem to test your basic knowledge of how trees work and tree traversal.
a standardized test to eliminate bias as much as possible
Uhhhh... It eliminates competent, skilled people who don't have the time to memorize the latest cargo cult trends in hiring. Look in the article for a glaring example.
"Invert a binary tree" is actually considered an easy problem to test your basic knowledge of how trees work and tree traversal.
So would simply printing out a tree, and at least that's something a person might actually do.
Tree "inversion" doesn't even make any sense and at this point I'm convinced that the cargo cult is choosing it because it's the extent of their own understanding of trees and somehow sounds extra technical to them.
I think it's more that the "inversion" interview story became a bit of a hot topic, and so is something people tend to reference in these discussions. Not that it's a specific problem people think is great to ask in interviews.
It also seems to neatly split people into camps who think "this is trivial, and totally reasonable to expect somebody to answer, even if it's a little contrived", vs those who think "this is not practical, and you'd only know the answer if you'd already practiced it, so it's not fair to ask".
> It eliminates competent, skilled people who don't have the time to memorize the latest cargo cult trends in hiring
But given that you already acknowledged that it's pretty trivial, why would memorization be necessary for a competant person?
Why do you want to memorize it ? What happened to thinking logically ? People just complain for everything - they want it to be easy, say it's unfair.
To be clear, I didn't imply I was agreeing with it. Just saying what the justification for this kind of question is.
I agree completely it eliminates a huge swathe of people, mostly experienced and older people. FAANG employees, ime, are biased towards childless / single people with privileged backgrounds.
Tree inversion sounds weird when you hear it phrased like that, but in an interview it would be explained with an example (just swap the left and right children recursively).
LC interview is biased towards people who are into LC/Olympiad/competitive programming or who have enough free time and willingness/desperation to grind LC.
The point is just to ask a simple tree traversal question, to see if you know how to traverse a tree (or more generally, write recursive functions and operate on basic data structures). They could have asked some other operation instead, but this one is one of the easiest so it seems fine.
> Why not ask the much more interesting and potentially useful question of balancing a binary tree?
Honestly? Because that's harder.
The swapping question is basically a softball / FizzBuzz-style question to test the most basic familiarity with data structures, pointers/references, and recursion.
Easier? You usually aren't coding FizzBuzz in production code either (although probably more often than inverting binary trees.)
Easier?
What's easier?
You usually aren't coding FizzBuzz in production code either
I would say you're constantly coding FizzBuzz. Looping, modulus arithmetic, and conditionals are all over the code I've written. At least with FizzBuzz you have a test of a person's ability to understand a task, break it down, and make sure the logic is consistent. With tree "inversion" it's not even a sensical request, it's utterly useless, and there are countless more interesting and practical ways to test understanding of recursion and trees. Knowledge that, I would bet, isn't even relevant for 90% of programmers, and if tree traversal were relevant then you'd probably want to jump to way more difficult questions (I'm thinking of the Facebook graph, for example).
I agree with the other commenter that it would make far more sense to ask questions like "you need to process data of this type, what provided data structure (e.g. C++'s STL) would you choose?"
I've certainly done tree traversal far more than modular arithmetic (at least at work, for fun I do recreational math problems...)
Let me mention the elephant in the room:
You the candidate, did not get the job because they did not like you ( e.g. you had a voice similar to the kid in school who was bully to your interviewer etc.). For most software positions out there, a relatively mediocre level of skills is sufficient. No fucking need to hair split on a person's technical skill.
If you truly care about the candidate first judge him on the 'cultural' fit. If he has crossed that barrier then the following advice from the article is a great one:
>Give candidates a heads-up about the attributes or topics the interview will cover and any other information you can reasonably share upfront.
All other advise in the article like paid assignment etc. are also great.
Usually you have 3-6 interviewers. It seems unlikely they would all "not like you" as the reason for not getting hired. Now sometimes a single very strong no is enough, but IME a strong yes can easily override that. I myself have interviewed 400+ people and have definitely had cases where I got people hired over objections because I thought they were really good.
I would say it's more common to not get hired because you didn't have a strong yes. Maybe everyone thought you were ok but none of the interviews really wanted to argue for your case. There are also cases of people that are just obviously not qualified, like every single interviewer says 'no hire'.
>I myself have interviewed 400+ people and have definitely had cases where I got people hired over objections because I thought they were really good.
You brought up (perhaps inadvertently) a good point here , most interviewers in the field are young, and their callowness is obvious, often you need senior people with the vetoing power to override their shallow analysis of a candidate.
FWIW, we’ve passed on (what turned out to be) the same excellent candidate(s) multiple times because the other candidates were seemingly stronger.
If you have two almost identical candidates, but the other has better credentials, it’s difficult to turn them down - if you have some HR guidelines to follow
We need an industry agreed-upon certification exam.
When you apply for a job as a doctor, they don't make you demonstrate surgical techniques. They take your board certification and ask you questions about what you've done in the past, things that went wrong and what you learned from that experience, and so on. They just assume you have the technical skills if you have the license.
Now, I'll admit that with doctors you are legally required to have the license, and we certainly don't need to go that far. If a small startup wants to take a chance on "unlicensed software engineers" or even offer to pay for the exam as a job perk (like a lot of law firms do for their interns), then great! But I can see a lot of time and effort saved if all the big enterprises would get together and come up with a national certification exam that you take once. Or even better, a series of exams for junior/senior/staff/principle, so neither candidates nor hiring managers have to waste time on tech assessments.
One of the keys would be making the exam inclusive for neurodivergent candidates, people with disabilities, etc. But this can be solved.
Basically everything about that idea is bad. It will turn into a cash grab, encourage pointless gatekeeping, allow some group to exert outsized control over a diverse trade, and anyway would never get universally adopted. And if it did happen, instead of people selectively spending their time to interview at companies they want to, they'd be forced to waste their time proving whatever the governing body thinks they need to demonstrate. Bringing some artificial barrier to entry to software development is the last thing the field needs.
This system already exists for the vast majority of real engineering, it's called the PE exam. I see no reason why we couldn't implement something similar in software engineering. It's not like it would keep you from being able to get a job as a code monkey working on Wordpress sites, but it would help in guaranteeing minimum levels of competency in the fundamentals of computer science.
All of those bad things only happen when the licensing body lobbies to get laws requiring the license. As long as the license is required, it wouldn't have any of those effects, because then you could get a job with someone who doesn't require a license.
We have university degrees which consist of numerous exams and we know they don't work to guarantee technical skills. The industry changes too quickly for curriculums to keep up, the field is vastly too large to cover adequately in one package, and testing well doesn't translate effectively to programming well.
Why would this hypothetical certification not have the same problems?
Universities have an incentive to pass you, and never claim to teach you technical skills -- they teach you theory.
This would be different because they would have an incentive to be an accurate measure of ability so that companies would continue to require them. Presumably the big companies would constantly contribute to the curriculum, and the small companies would then benefit from that.
Medicine and the law are constantly changing, and somehow those licensing exams stay up to date. I assume the same methods could be used.
> When you apply for a job as a doctor, they don't make you demonstrate surgical techniques. They take your board certification and ask you questions about what you've done in the past, things that went wrong and what you learned from that experience, and so on. They just assume you have the technical skills if you have the license.
That's when you already worked as a surgeon in a reputable hospital. The same way, the technical screens (leetcode) is generally minimal for hires that have demonstrated a clear pattern of success at reputable companies.
> We need an industry agreed-upon certification exam.
We already mostly do: proper CS/Engineering degrees have a pretty good signal to noise ratio for hiring.
Companies won't share their stats, but they know which schools and programs to target. Real engineering departments typically have job placement rates near 100%.
We already have the problem of people who are good at grinding LeetCode and passing FAANG interviews, but terrible at actual software engineering. And the inverse.
Why would this not just transfer to the certification process?
Are you saying a higher percentage of good swe’s could not pass a LC interview with prep then can pass, but are terrible swe?
Usually you have to have an internship before taking a certification exam. I assume the same would be required here.
As someone who has seen my share of “paper tigers” - people that either memorize enough to pass a certification or find brain dumps online. Certifications are useless and easily gamed. This has been true sense at least 2000.
When I was working in the “real world” (I’m in consulting now), I could tell the people who just studied for a cert within the first 5 minutes of an interview.
Most industry cert exams require internships before taking the exam. It could work like that.
I agree with you that most certs suck and are meaningless, but there are a few that are actually hard to get and meaningful.
>We need an industry agreed-upon certification exam
I'd be OK with this if all the answers were just shown to various companies so they could see strengths and weaknesses. If some company doesn't care about DP or low level OO design they can throw out those scores, etc.
I certainly got annoyed in my last job hunt where I had to answer some basic whiteboard easy level LC questions over and over and over. Thank god I got to skip the Google screen. If one more person made me do basic BFS or something I was going to freak out. I understand why they asked it, but I had to keep coding it over and over and over...
Law and medical licensing both require showing basic proficiency in all areas. I don't see why we wouldn't do the same, and then let you get specialization afterwards.
Sure basic proficiency is fine but how do you decide who is worth 150k and who is worth 400k?
Experience. Ask them about previous situations and what they did in them, what they learned from them, what they built, etc.
But it lets you skip the basic coding exams about data structures and OOP.
Let's be honest, you're not choosing 150K vs 400K based on how well they can reverse a binary tree.
I wouldn’t hire anyone who couldn’t reverse a binary tree (with a bit of coaching is fine too). I can certainly understand why someone without white board experience would initially freak out and assume it’s difficult but I’m confident I could ease some panic and point them in a reasonable direction.
We’re talking like 4 loc here…
But there are absolutely people with the title of senior engineer who cannot understand the concept of swapping some pointers and recursion.
Of course you wouldn't. But with a certification exam that included such things, you wouldn't have to ask.
And more importantly, like I said, you aren't choosing their salary and job level based on those skills. You're choosing it based on their experience.
The few skills that are transferable across the whole industry are a tiny part of what it takes to do many jobs, especially more senior IC roles. There is no certification that is possible.
I think you vastly overestimate the skills that are necessary for most jobs.
Also, it should be noted that Amazon and Google don't interview for specific jobs -- they interview you for a job at the company and then you get placed. That would imply that at least at Google and Amazon, the skills are transferable across everything they do.
In other words, if I need those skills, the people doing most jobs are unsuitable.
Doctors and lawyers are legally required to have licenses. This is not the same as a certification (which is legally optional). The argument for doctors/lawyers requiring licensing is that they deal with the public who are not trained in evaluation. That doesn't really apply to companies hiring engineers, so who exactly are we protecting here?
>We need an industry agreed-upon certification exam.
I'm an actuary who had to take 10 of these for my profession (pass rates 20%-40% each sitting, takes 4 months to study for each one).
They come with the same complaints about false negatives, unrealistic/random questions, and fairness that technical interviews have.
Ok, but at least you only had to do each one once, and not repeat those tests for every job interview.
To do the ten exams is ~3,000 hours of studying. A typical job interview is, say, 6 hours. So that's equivalent to 500 job interviews. Now I suppose if you job hop A LOT then you could reach that number but I expect for most people it's not going to happen. I'm about halfway done with my career and have had <50 interviews.
How about LinkedIn certificates? Last I checked Triplebyte has a certification you can put on your LinkedIn if you do well enough on their process.
100% agree. I think we should have a specialization of the PE exam specifically for software engineering.
Its called fizzbuzz.
Anything more complex than that is probably job specific anyways.
My primary programming languages are not allowed in leetcode interview sessions so a lot of the challenge is remembering how to use Ruby or Python on the spot, and also to think imperatively.
Lot of my thinking is based on visuals and emotions -- It's challenging for me to transcribe to English on demand and it interrupts my process -- it's somewhat like painting.
I always shine on take-homes since I'm allowed to be my authentic self. I'm enabled and have the full capacity to do my rituals, routines, and quirks.
Admittedly, this means I won't succeed in cooperative environments like pair programming. I'm better off left to my own devices.
Tbh the way that hiring is done in tech is just lazy and reeks of mediocrity. Memorizing 'take home' assignments, banks of algorithms, pair coding, white board interviews, what a joke. Companies want to place all the burden on candidates and spend as little money as possible. It doesn't matter if they waste people's time.
Whatever happened to getting to know a candidates work? How about look into the work that a person has done and take the time to understand where a person's skills are. The problem with tech hiring is we have people trying to cut corners. So-called 'non-technical' recruiters doing interviews with a checklist, companies that treat people like hoop-jumping monkeys, and generally f*king idiots that won't do their job (they get paid for it, why again? They're not actually doing their job.)
Hiring is not a complex problem. The problem is literally incompetent people doing hiring.
On a smaller size (company) it's relatively easy: pay well, don't oversell the position, send a small assignment representative of work and give them ample time to solve on their own OR ask for references you can talk to from previous workplaces.
Game changes if you actively contacted someone.. if you're no BS, assumption is you know who you contacted and why, hence only thing to do, once contact established is not to oversell and pay well.
Pay-well can constitue compensation as well as time.
This is focused on finding technically skilled engineers, but I think you can get a more wholistic (holistic?) view of the person by asking them to walk you through their work history, project by project, and call a subset of the people they’ve worked with.
It’s more conversational, and you don’t have to live in hypotheticals.
We all know that skilled engineers will learn whatever skills they need to on the job, so less and less am I interested in what they can do in the interview pressure cooker.
If you really want to hire engineering talent, paying above "competitive salary" is very important.
It's a bit orthogonal to the concerns in this article, but in some ways it's much more important.
What I wonder about is given an org that is able and willing to compensate at market clearing rates, how do they get the word out well enough to get engineers interested. Because the other big BS in hiring is the whole recruiting side of things.
> the best experiences were when the interviewer wanted me to succeed, was emphatic
I assume you mean empathetic. Same word is spelled “Emphethatic” later. (I tried finding a way to reach you privately, but your site “about” says you have contact methods on the left but, on mobile, there is no left…so here will have to do.)
In addition to the other comments, the errors you noted is a signal that the author didn’t have editors/peers who cared enough to spot basic mistakes.
Thanks for pointing it out! It was a typo, fixed now.
With all the overt and hidden biases that us flawed humans live with daily, the idea that you will be able to pick really skilled devs consistently is laughable unless your doing some sort of blind interviews like symphony orchestras do nowadays.
It's so blatantly obvious that interviewers are basically trying to hire themselves, and will almost always select candidates whom they share the most personality traits with.
Also, I see the hiring process as similar to wanting to be a politician ie anyone who really wants to be one and is just really good at it should never be given the job.
The people who impress you the most almost surely have simply put a ton more effort into gaming the process with long leetcode sessions, live interview practices, and other bullshit tricks to convince you that they are the best person to hire.
With so much as stake, why wouldn't young devs spend tons of time working on interviewing skills and not really giving a damn about developing the real skills needed to be a goto resource at Big Software?
It's very much like taking steroids in professional sports...well no shit your taking PEDs when your career paths are either making generational wealth fucking with a ball or working selling Jordans at the local shoe store.
Mostly good article: and totally agree that leetcode style interviews under intense time pressure are a wasteful plague on the industry. A good web developer could go through a bunch of interviews without ever once discussing designing APIs, HTTP requests, web security, or any one of a 100 practical topics which they deal with nearly daily, but might be asked to solve a pile of different leetcode questions (99% of which are close to irrelevant for the vast majority of typical day to day web-development).
If I could give one minor persuasive writing critique to the author of this article though, I'd suggest not emphasizing inclusivity (which is basically bog standard, meaningless corporate drivel by now), but emphasizing that changing the typical interview pattern ensures that you're casting the widest possible net for talent. There's business people out there that couldn't give half an ounce of care for doing a solid for whatever the heck they might think a neurodivergent is, but if you emphatically frame this a bit different (solely as the company missing out on talent) I think the argument instantly becomes a lot more appealing to a pretty large group in the business world.
I have found success by doing the following (when hiring for an engineering role):
1) Doing a relatively shallow but wide survey of the technology I'll expect them to be responsible for. Because we use k8s, I steal the old "type google.com into your browser" question and make it, "I type 'kubectl get pods' and hit enter. What happens to make the list of pods show up on my screen?". From there, you can dive into basically any part of the stack you want.
I'll often ask them to explain to me what the difference between a container and a VM is, as though I were an intern, and then I'll ask probing questions about things they get wrong or things they leave out that I think are relevant.
This isn't to pass/fail them, necessarily (though some people have done so badly that they essentially failed themselves), but it's to see where their familiarity and comfort level is with the tech at hand.
2) talking through their resume with them, and doing a deep-dive into a couple aspects of their recent history - why did they do a thing? What alternatives did they explore? What was the reason they went with what they did? How did they implement it? What problems happened? Who did they collaborate with and what was the precise scope of their involvement? How did they measure success, and what was the follow-up?
I don't expect anyone coming in to have a deep knowledge of the tech stack we have. I do expect someone to have deep knowledge of the technology that they put on their resume, though.
My hit / miss rate is pretty decent. There have been a couple of times that I said no when I should have said yes, but I'm okay with that ratio.
Generally good tips, but I have noticed a few other things too.
1. If there are challenges, particularly if they are take home tests, it is important to make these reflect the sort of work someone will do without raising concerns that the work will be used by the company without pay. Candidates will spend time on relevant challenges and be happy. They will not be happy about irrelevant challenges. And interviews go both ways.
2. Dispense with "good questions" and go instead with "what do I want to know about a candidate.
3. Ask yourself before you start hiring, "What makes those who are successful at this company successful?" And from there, start building your interview structure.
Not every company will be the same, or will be good matches for the same candidates. The key should be to figure out what you need and use the interview to determine if the candidate actually is a good fit.
Unfortunately this cannot have data because it relies on a bunch of human judgment calls.
A lot of people are in the "if I could just hire right I'd be fine" camp when they should be in the "why is the way I'm trying to build this stupid" camp. Address the root problem, go in with the assumption that it will be easier to start over from scratch and rebuild with 3x the productivity based on what you've learned about what _doesn't_ work, and it will be easy to find people who want to work on your app/platform/etc. I've advised far too many mid to late stage startups where what they really need is to just take what they've learned, rm -rf, and quickly build something that works now that they've gone through the growing pains, and far too many have taken this option later than they should have.
What does rewriting your codebase have to do with hiring?
Companies that have trouble hiring and retaining right now have at least one of three things: a stinky codebase, a stinky culture/benefits/work-life-balance (i.e. not WFH, micro-managing, etc), or uncompetitive salaries. The ones that have none of the latter problems sit around scratching their heads and writing posts like OP
Hiring...my partner had an interview booked for a Friday at 5pm. The interviewer didn't show up. My partner end up emailing the person, whom apologised and then jumped on a 10m call, 10m before 6pm. My partner, as I do, spend a lot of time researching the company, preparing for the interview, etc.
How many countless stories like that exist?
For example, people messaging for a chat "found your work on project X and saw your github account and found about your past projects, we are looking for some one like you", then on the day "oh sorry to let you know last minute but X and Y happened".
Bunch of time wasters! This is the reality!
Once in the job, it's funny to see who actually does the work. Zero contributions for days, etc.
There are a lot of people out there handling these processes and they are bad, really bad!
And then interviewers and recruiters go on LinkedIn lamenting candidates ghosting them on a regular basis. I constantly have to fight the urge to post screenshots under their posts where they fail to get back to me after reaching out first.
While this seemed to start strong, I don’t buy into the “structured” portion of this blog post. The referenced research does not seem relevant to hiring engineers. In fact, the opener for the first reference says they researched “ 19 male applicants for life insurance sales”positions”. This is a “mountain” of evidence in hiring engineers?
My own interviews have a list of topics I want to cover (non functional requirements, data experience, app design, infrastructure, etc), so I guess there is some structure. But I mostly run the interview based on their own experiences and projects they have worked on. So we will focus on applications and systems they have worked with in the past. And then I see how deep down those rabbit holes of their own system they can go.
Dev for 16 years. The people who do the interviewing are themselves different now.
They have different values. Different expectations of what is normal or important. "Culture", "team fit" and other bs.
Everything changes. New people come who don't know the past.
I wish a company would run an experiment where they just higher at random from a stack of team vetted resumes. I think the results would almost be the same as being super picky.
Why can’t anybody trust verifiable info these days - that you worked for $COMPANY doing some $JOB for $YEARS. Why are resumes completely thrown out the window and you start from ground zero on a whiteboard when you have over a decade of experience. I wasn’t practicing leet for the last decade, I was doing actual engineering.
I work in aerospace which often does a STAR behavioral interview (very light to zero technical interview) and I can honestly say that I work with some of the brightest people I’ve ever met.
Unfortunately the STAR behavioral questions are even easier than Leetcode to study up for, and they're a pretty common interview set for managers or to set leveling, so a prepared candidate will ace those questions.
My point was more that you can get capable engineers without any technical interview process.
I've interviewed people from $BigTechCompany (with 5+ years experience) that couldn't do some basic whiteboard coding question. I wasn't asking any dynamic programming-type obscure thing that no one actually uses, but rather a realistic question that they might actually have to code (indeed it was based on a problem I had to solve myself at my job).
So it's not that I don't trust that they didn't work there, but that doesn't mean that they can do a good job here. Every company has low performers, perhaps their previous employer was just bad at detecting them.
I think a lot of people are just not as performative as interviews demand. I think these people would be perfectly capable at the job if you hired them, but the interview experience (emotionally, collaboratively) is nothing like how you work day to day.
Care to share the example problem?
For the small startups, who don't have time for BS like leetcode, I advise asking the candidate's code first of all. The senior-level developer almost definitely should have GitHub with his portfolio- or pet-projects. After carefully looking through his code, it becomes 100% clear do we want to talk to him or not. After that we usually had one interview, something like system design + soft-skills. Subjectively, I recall the hiring experience as very successful compared to what I experienced interviewing people for major tech companies according to their guidelines.
>his
There's the rub. Women are disproportionately likely to spend time caring for children and have less free time available for hobbies. Personal projects might be a positive indicator that someone really cares about computing but the projects I've been praised for in interviews make up ~1/500th of the time I've spent developing my skills. It can objectively display the quality of code that a candidate can write but its absence isn't a guarantee that the skills don't exist.
My biggest recommendation is that those working directly with the new hire, peers, direct reports, subordinates, counterparts, have a vote or veto power.
HR and recruiting relationships are sparse at best.
Have to also have a challenge process. I built an infosec talent pipeline for a fintech; stakeholders get a veto but the hiring mgr can go back to the veto voter and ask them to dive deeper (and possibly perform an additional candidate call) to confirm the veto. >1 veto = no hire.
It’s working well, and has avoided at least two false negatives since implementation within the last six months.
Have the “1 veto but hire anyway” cases gone poorly?
A scenario in which I can imagine this backfiring is if the new hire and the vetoer end up working together. With an open mind that could be overcome, as in getting positively surprised, but let's be honest, how many people do you know who really have such an open mind?
No, both going great, which means (it appears) I’m balancing the org’s health and need to succeed with giving candidates an opportunity they might not otherwise have had.
The purpose of the system is what it does. If desired state is not emerging, we must adjust and observe accordingly.
It sounds good but do we have any evidence that this actually works? There’s so many of these speculative “how to interview” posts but it’s all just cargo culting.
This is great, the approach I’ve seen work best in a couple orgs now is giving the candidate options. You can either walk us through and open source project you wrote, do a take home test, or do a live coding challenge.
There is a large diversity in how developers are effective. When you force people into one funnel you lose the rest of the ecosystem. Meet people where they are, the only metric that should matter is effectiveness
Doesn't that make it harder to do an apples-to-apples comparison between candidates though?
This is the criticism I hear but I think it depends on how you look at engineering and bias in general. I just see different people with different skillsets and pick the best one for the team. I have yet to run into a situation where the ability to code was the determining factor between two candidates. I imagine it could happen but so far so good
Interesting take.
We all want "inclusive hiring culture", "exceptional engineering talent" and a frictionless hiring process, but IMO you can't have all three. Like them or not, leetcode* interviews actually give a chance to people who can't do a home assignment, or come from a background that didn't let them have a bunch of code on Github. In that sense, it's the more fair way to test people's aptitude, and will find exceptional talent from all kinds of different backgrounds.
If you still want "exceptional talent", but not algorithmic interviews, then you end up biasing towards white guys who have a ton of projects to show you.
Actually, I think this should be verifiable. Select some companies that we think have exceptionally high bar (you could use compensation as a proxy, acknowledging it's imperfect). Then classify them based on whether they do "leetcode" interviews or not, and check their diversity reports. My bet would be that the "leetcode" companies do significantly better.
* Caveat is that companies people think do "leetcode" actually usually ban questions that appear on leetcode.
> Like them or not, "leetcode interviews" actually give a chance to people who can't do a home assignment
Not really—if I don't have time to do an hour-long take-home assignment, what makes you think I have time to practice leetcode-style questions? The take-home assignment is usually testing skills that I actually use in my job on a regular basis, so I don't need extra preparation, I just need a block of time to sit down and do it.
I agree that expecting people to be able to show side projects is a mistake, though I'm not sure why you think that the bias there would be racial—I would imagine it would be much more a filter that excludes people with families and/or non-computer hobbies.
I'm not sure why you think that the bias there would be racial
This seems off to me as well. Wealth, and schooling, are not relvant here. Just grab an old computer, install Linux for free, and off you go.
Loads of free tool stacks, github is free, etc.
I guess my assumption is that measuring cognitive ability is better than specific skills. The algo interviews, when done well, do that better than interviews focused on experience.
There are of course a lot of caveats - experienced candidates should be treated differently from entry level. It’s also true that, e.g. Google has a weird fetish about dynamic programming questions and other problems that people are unlikely to figure out without having taken a class in them.
On the other hand, take home assignments take up time and room you might not have. It’s easier to do those when you’re a man in a developed country than, e.g. a single mother in Alabama.
So its basic, I think algo interviews are a good way to hire junior level engineers.
I have an interview process that's both easy and hard.
It's kind of a "choose your own adventure" style interview.
I explain upfront what the process will be, that during the interview "I don't know" is a preferred answer than BS.
Then I start with this question:
"Your task is to take some data in from a user, store it, and then present it back to the user. How do you do it?"
Based on their answer and follow up questions, I follow them down the path of their preferred stack.
This is rare, but I love it when prior to answering, a candidate asks me clarifing questions, like "how many concurrent users will the system need to support? What sort of performance is necessary?" Etc...
Many just assume that I'm asking about web development, so if they go down that path I challenge their assumptions, and follow up asking about their reasoning.
If they pick a framework, I ask what the advantages and disadvantages there are to that framework and how it compares to others.
Same with databases, etc..
It becomes pretty clear quickly what sort of level they're at. Many times the answer on a framework question is "that's what I used at my last job". That's not necessarily a poor answer, but it is informative.
I try hard to make it a casual conversation, like the sort you'd have with a technical stranger at a bar or something, though I understand that's impossible given the power dynamics and stakes involved. Still, I try.
So far, it seems to work pretty well for me. I have the luxury of keeping my team small, so I don't have to come up with a scalable "one size fits all" process, I'm able to keep it personal and relevant to the candidate and their experience.
I agree with the conclusions, though I've seen the structured part go wrong, e.g. the interviewer is so dedicated to following the structure of the process that they forget about the empathetic part. These interviews look more like scripts than exploration of a candidate.
So I'd add another criteria: interviewers need to be trained!
Singular: Criterion
Plural: Criteria
SCNR
I think something like this works well:
0. 45 minute homework/prescreen. Provide an (optional) pre-setup environment so it's mostly about coding and not about building/installing deps.
1. on-site where you chat about your solution, mostly an ice breaker/introduction to the team.
2. pair-programming to extend the homework or work on a simplified but real problem encountered day to day, open book
3. design review
4. code review
5. behavioral / case study
All of these can be pretty objective and don't rely on any memorization. All this should be pre-canned so individual proctors don't come up with their own questions and you're comparing candidates around the same prompts. It's amazing how few companies even manage these basic steps. I think most importantly the hiring should be done by a committee of actual practicing engineers - that means if you have checked in code in six months you aren't a vote on the committee.
Imagine the other side. You're more than likely doing this for multiple companies. Plus working your current job. Plus taking care of your kids! You have to devote minimum 5 hours per interested company. IMO that's ridiculous. I also think its why startups skew so young. I don't have the energy to put up with it anymore.
Sorry, but take home test as the first step to filter them is a big no for me. Why should i waste time on a test before even meeting anyone?
1, 2 and 5 should be more than enough. You get to talk to them about their past expertise and even combine it with some design discussion, you get a pair programming session and a final casual discussion. Why do you need everything else?
They call it a funnel for a reason - most candidates are not worth bringing onsite - which is expensive for both sides. So you need a screen anyway and homework should, if it's well designed, take similar amount of time to a screen. In my experience it takes most teams at least six months to get an engineer productive and a year before they're really hitting on all cylinders. If you're asking a company to spend six months to a year training you, giving up a day isn't (IMO) a huge ask. As a candidate you really shouldn't need to go onsite at more than a handful of companies or you're really wasting everyone's time.
It's too bad it's not a culture of interview questions to learn how well someone will work with you and your team, how much of a creative they are, a coach, a mentor, etc.
I'd pick someone who really clicks with the team and is smart any day over someone who's brilliant but hard to work with.
I agree with others that these posts only have limited value perhaps to people who don't have a clue. For those of us who have done this lots, it doesn't really say much but it does allude to a common fallacy, that the interview itself is the most important part.
Other things that are actually useful to consider (others have mentioned these too).
1) Your company culture is important for you to know; for you to codify and for you to communicate to your candidate.
2) In some companies, you will have tonnes of unsuitable applicants because of your brand, you should not optimise for people that aren't suitable - filter them asap
3) Your whole onboarding is a lot wider than just an interview. For many of us, we should be asking where/how we would expect to find our candidates and optimise those places. Do we visit hackathons? Is our recruitment page(s) clear and does it articulate what we want?
4) Your entire process needs to be like an interative development. Did you hire a bad person? What could you change to catch that in an interview? Wrong technical questions? Not enough about comms or culture?
5) In many cases, a company will hire someone that comes across well even if they don't necessarily tick the boxes so don't assume that you didn't get the job because you failed the whiteboard test, maybe you weren't as good as you thought?
6) Candidates need to think carefully about their approach to interviews, I would say more candidates than not seem almost entirely unprepared for normal interview questions and perhaps expect their ambience will get them the job! Study, swot up, never used owasp for web apps? Don't say that in an interview, spend 10 minutes learning about the top 10 and answer confidently.
7) Recruiters are in it for large commissions. They kind of care only in-as-much as getting a good reputation might help them but everything is second to money so don't expect them to place you well, don't expect them to sell you properly and if you get rejected, don't expect them to call you!
Nice article talking about the common pitfalls popularized by likes of Google and Palantir a decade or so ago. Some of these methods work if you are hiring enmasse and are ok to tradeoff identifying some outliers (like in homebrew creator's example) to efficiency. This is one of the reasons I built nitrohire.co to be able to get more relevant information about a candidate in a different way! The basic question is why try to guess something about a candidate that their peers might already know and help you make more informed decisions
Make a sign that reads "Free Starrett and Mitutoyo gifts if you like Maxwell's equations"...
Yes they know its a 100% a trap, but good engineers won't be able to resist responding on the off chance it is real.
Don't click this... it is clearly a trick... you already knew they never have overstock sales. yet had to click this anyways... =)
For a while my company did aptitude tests which were basically a kind of IQ test. Between good results on this kind of test and having a good conversation in an interview I felt there was a very good correlation for doing well on the job. I value this higher than specific knowledge. I feel smart people that can communicate and get along with others will usually deliver high performance.
Both are needed in a healthy team. Unless you're building trivial things then just being smart and communicative doesn't cut it. I would say you need a mix of 50% really smart people, that get along with others, and have the specific knowledge required, and 50% smart people that get along with others, and have some basic skill. That mix +/- is a good team. A team of smart people that have good people skills (EDIT: but lack specific/domain knowledge/experience) will consistently produce terrible software.
Question for the greater HN community? Would it help if software engineers demonstrate their skills on an ongoing basis via a third party which accesses them periodically without the pressure cooker environment of technical interviews and companies just do cultural fit type of interviews based on the previous assessments of their technical skills?
It'll be borderline impossible to persuade experienced engineers to do that.
I think so too. But the incentive of not doing tech interviews the traditional way will be a bit tempting too.
And the adventure to fix the issue of ENG interview saga continues...
"Process is broken" continues to be the theme where no parties agree whether a basic algo or leetcode or takehome is sufficient yet continuously reject professional designation.
Folks, keep in mind that at the end of the day, you are hiring a person, not an object with a bunch of methods.
"invert a binary tree"
This didn't make any sense to me, so I looked it up, and it seems exactly as senseless as I had thought. Is the 'inversion' not the same topology as the input?
my solution: "auto invert(regular_tree x){return static-cast<inverted_tree>(x);}".
When I tried to find what was meant by "invert a binary tree", the only thing that made sense was the operation of reversing the tree: ((1 2) (3 4)) becomes ((4 3) (2 1))
It means swap the left and right pointers for each node.
The sad fact is usually your assumed nationality, your assumed gender, your skin color (not necessarily being white is always good), your assumed origin and assumed religion matters more.
Depending on these, you end up with different attitude and questions.
I interviewed at a startup with ex Amazon employees as founders, and they had some crazy systems design and a leet code style challenge. I ended up doing entry level, basically data entry, work for 6 months before leaving.
Good jobs have friendly practices, bad jobs have awful practices.
I quite like how things work currently. It's easy to tell the difference.
IMO interviews are designed solely to maximize team size. This requires filtering for individuals for fit well into a hierarchy without disrupting the structures above and below them.
Being "good at the job" is actually bad because it reduces the team's required size.
Almost all the incentives revolve around this. Not only compensation, but entire hierarchies that revolve around EB1C dangling. (managers and executives at multinationals can get a green card faster)
I think there's a fundamental misunderstanding about the purpose of the whiteboard interview. The point is to eliminate, as fast as possible, candidates who simply cannot code [0] [1].
You can't do that with a take-home (and I'm against take home as the signal to noise ratio is too low) because people will cheat and have them done by someone else.
I've heard horror story of a "senior" engineer from "his country's top school" being interviewed for a technical position by several non-technical managers and HR reps. They only included an engineer in the final round, which was basically supposed to be rubberstamped anyways. He was then asked to implement something trivial like fizzbuzz or wordcount on the whiteboard. The candidate then became extremely defensive and tried to argue that such task was "beneath him", arguing for a good 15 minutes why he shouldn't have to do it.
Then the dev just left the room and said that he used this question as a warmup with new hires and it typically takes them less than 10 minutes.
Now, a lot of folks do whiteboard interviews wrong. They often expect to get the exact implementation of an algorithm they found in a textbook and for code on the board to compile. This isn't the point of whiteboarding. Doing this only promotes rote memorization. A good whiteboard interview should be a toy problem that can be solved in several different ways by using different strategies or data-structures. The idea is to see how the candidate will break down the problem. Is the candidate able to formulate test cases, write a simple implementation, verify his code and correct the implementation should it fail a test? On the more meta side of things is the candidate able to take feedback and explain why a certain strategy was chosen? Of course it's not representative of real world engineering but it's a good way to peek at someone's ability to debug and reason about programs; these abilities translate well into debugging and design. Especially at the college level, I really can't make any assumptions on what the candidates know. I'm not judging their knowledge of the standard library of X programming language or the framework-du-jour but their ability to learn it fast.
Now the hard part isn't so much to create an interview process that works well, but to create a pipeline that feeds into this interview process that has a high signal to noise ratio. In my experience, the best predictors of a good signal to noise ratio was to select for CS fundamentals, good references and offer above market comp. The latter is especially crucial now since there's no more "local market" to speak of now that remote work is a lot prevalent. The "local market's" best devs are working for SV firms at SV salaries mentoring SV employees.
[0] https://blog.codinghorror.com/why-cant-programmers-program/
[1] https://economictimes.indiatimes.com/tech/ites/95-engineers-...
embedded tweets, comics, memes, and cartoons ... dumbing down a complex subject. Next.
deleted too negative, upon refection.
Well, what did you expect?
When interviewing someone, I would still want someone to work through a problem live with me to see how they solve some problem. I don't care about the end result, I want to see how you're getting there.
deleted too negative.
If someone gave me a ChatGPT answer and nothing else, I don't have any way of telling if they're using it to boost their productivity or if they're just a complete charlatan copy-pasting off of ChatGPT who can't solve anything as soon as ChatGPT gets a little wonky. My bet is on the latter -- maybe it's my loss, but I would rather they have just worked through the problem without ChatGPT in the first place so I didn't have to guess which side of the spectrum they fell on.
"If you wanna hire great people and have them stay working for you, you have to be run by ideas, not hierarchy. The best ideas have to win, otherwise good people don't stay."
- Steve Jobs
I think the real disconnect with the 'inclusive culture' boom comes because humans are involved so heavily in the process. The _idea_ is great, we want to be fully aware of our internal biases and avoid having them color our perception as much as possible so we do not shoot ourselves in the foot.
In practice, I have yet to see inclusivity programs at corporations be anything more than virtue signaling, and an opportunity to exclude others under the guise of "inclusivity" wink wink.
Remember 'affirmative action'? It's palpably Orwellian that inclusivity is newspeak; what we call it now.