We compiled a library of realistic engineering take-home tests and ranked them
trytapioca.comStudies [1] show that a work sample test is the best predictor of candidate performance on the job, which is why many software engineering teams use take-home tests as one step in their hiring process. But designing an effective test is difficult and time-consuming. For example, candidates are reluctant to complete tests that are too long or not engaging enough. But make them too short and teams won’t get the signal they need for a proper evaluation.
To encourage more thoughtful test design (and hopefully save future candidates from the worst offenders), my team compiled the largest library of non-“whiteboard” take-home tests that real engineering teams have used. You’ll find the challenges that Stripe and Microsoft gave to their full-stack candidates, front-end tests from Tailwind and Rivian, and back-end ones from Basecamp and Revolut. Whether you’re looking to evaluate an Android, DevOps, or Data Science candidate, a bootcamp grad, or senior engineer, we found a few options for each.
Having built 20+ tests ourselves, we also rated the design of each test. The criteria for a 5-star rating:
1. Tests for skills highly relevant to those required for the position
2. Includes a well-written description of the prompt and even motivation for using a take-home test
3. Sets clear expectations for candidates (e.g. time requirements, evaluation criteria, submission details)
4. Asks for a reasonable time commitment from candidates (<4 hours)
A few notes: - We found most of these test prompts in public GitHub repos, usually owned by the hiring team but occasionally in the candidate-owned submission. We sifted through hundreds of tests and filtered out those overly focused on algorithms (aka LeetCode), leaving us with 142 tests in the library.
- The larger and more recognizable companies didn’t always have the best tests. Some of the most interesting prompts we found were from smaller teams (e.g. YC startups). This shouldn’t be surprising. Startups need to design candidate-friendly hiring experiences to compete for talent against more established players.
- There were common themes among the tests we found. For example, front-end candidates were often given a Figma design + content feed to implement, while back-end candidates had to implement an API given a set of requirements. Data scientists were usually given a data set to clean, analyze, and submit a Jupyter notebook with their findings.
- We’ll continue to update this library and add descriptions of each test so it’s easier to compare.
Have feedback, or another take-home test we should add? We’d love to hear from you!
[1] The Validity and Utility of Selection Methods in Personnel Psychology (https://www.researchgate.net/publication/232564809_The_Valid...)
While this certainly sounds like reasonable criteria for rating a take home, does this truly account for general candidate reluctance to do take homes due to the time commitment? Even 2-4 hours is more time than conventional Leetcode-style initial assessments, and many may rather use that time to interview with 2-4 companies instead of just one.
I recently interviewed with Ramp and enjoyed their model of practical problems, like a take home, but condensed into just one hour with less of the end-to-end expectations of a true take home.
I think most 2-4 hour take-homes can be condensed to 1-2 hours with some thoughtful choices. A few ideas I've used:
- Provide starter code and setup instructions so candidates don't waste time on boilerplate.
- Abbreviate requirements to what actually matters. E.g. do you really need 100% test coverage on a take-home? Ask candidates to write a few tests and then tell you what else they'd do given more time.
- Use an open-ended, time-boxed format instead of having end-to-end expectations. IMO a hybrid format where a short (1 hr) take-home is followed by a live discussion/pairing afterward can be the core component of a hiring process.
I'd love to hear more about the Ramp process. Do you mind sharing what sort of practical problems they used?
I interviewed for a backend role and the problems were greatly simplified versions of day-to-day backend SWE work. For example, use a server to complete a task. The focus was on how you make the requests and handle any edge cases that might come up, and the server was actually live so you could tinker and get immediate feedback. Hopefully that’s not too vague :)
Edit: to be clear, the interview was live with an interviewer. So it wasn’t a take home in the scheduling sense either.
I agree that take homes can be simplified with your suggestions above, and that certainly makes a better experience for the candidate. The hybrid format is also great - future interviews become an extension of your previous work, so it’s more comfortable than having to context switch for a new challenging problem each round.
I didn’t find this in your linked database, but I also enjoyed GitHub’s take home. I only recall spending 45-90 minutes on it, and the setup process was seamless. A recent blog post describes their approach here: https://github.blog/2022-03-31-how-github-does-take-home-tec...
Yes! I spoke to Andy (the author of the article you linked) when he posted this. I'm a big fan of this approach. And the software we're building is quite similar to the interview-bot one that GitHub uses internally. Why shouldn't every eng team be able to benefit from tools like theirs?
I'd love to add GitHub's take-home to the library, but I feel the article describes the exercise without sharing the actual prompt. If there's a public link to it, lmk and I'll add it.
You’re right, I don’t think their exercise is public. I just wasn’t sure if you knew about it and felt it was good enough to call out.
This is awesome to hear! I did check out your website but wasn’t sure how closely it tracked with the GitHub method.
Honestly, I had a thought to build something like this after my GitHub interview but wasn’t sure how much traction it would get. It’s cool that you’re working on it and that my random thought was validated; best of luck!
> Even 2-4 hours is more time than conventional Leetcode-style initial assessments,
Hard disagree on this one. Unless your day-to-day work includes many leetcode style problems, you need to put in significant time training on leetcode if you want to pass the interview.
You should be able to complete a take-home based on your current skill set. Yes, it may take 4-6 hours (or 8-10), and yes, that is a big ask of a candidate, but leetcode can take 5 to 10x prep time and you still can muff the interview.
Is there literally any other field where there's an expectation to essentially do fairly intense studying to pass interviews that apparently have very little to do with day to day jobs? Maybe they exist I've never encountered them. The limit of my "studying" for an interview is to just learn a bit about the company, their strategy, etc. either by online research or by talking to people I know connected with them.
ADDED: I suppose one could argue that the Bar exam is a bit like that but that's a credential as are degrees which are not necessarily all that overlapping with the real world.
> ADDED: I suppose one could argue that the Bar exam is a bit like that but that's a credential as are degrees which are not necessarily all that overlapping with the real world.
Yeah, I think most other professions where one might imagine them doing something like a stereotypical software interview, use that instead: a credential obtained by a (perhaps very difficult!) test, maybe with periodic re-tests or required refresher courses to keep the credential valid.
I have a suspicion that a big part of why top-comp software companies keep their interviews so incredibly unpleasant has more to do with discouraging job-hopping among them (so, suppressing wages) than with its being the best process for hiring good developers.
> I have a suspicion that a big part of why top-comp software companies keep their interviews so incredibly unpleasant has more to do with discouraging job-hopping among them (so, suppressing wages) than with its being the best process for hiring good developers.
You have to wonder if this has backfired. ;) Plenty of people job-hop every 1-2 years, and it's pretty common knowledge which companies ask which level of LC difficulty for which level.
It's probably pretty common within a certain cluster. A bunch of other people probably see the barriers and just pass. Or my experience is that others just find grass is distinctly not greener and move on.
Is there literally any other field where there's an expectation that you will work for free 4 hours in order to be considered for the job?
Tons of them. How much time do you think the typical actor or dancer spends auditioning and preparing for the audition? How much time do you think people in many creative fields spend putting together portfolios? And if I'm going to interview with just about anyone, I'm going to spend at least 4 hours prepping for an interview.
I'm not a big fan of take homes in general but a time-consuming job search and interviews is absolutely normal beyond low-skill jobs in a time of labor shortages.
I hard disagree with your disagree :)
1. Some candidates only need to brush up to be in interview shape.
2. To mitigate the need to spend time to prepare, simply use initial assessments to get back in shape. That is, apply to and interview at companies you don’t want to work at first to get more practice problems. If you pass, you get the added benefit of more leverage when negotiating your compensation!
3. As another commenter mentioned, as Leetcode style interviews are shorter, there should be a threshold where $prep_time + $leetcode_interview_time <= $take_home_interview_time. Of course, the threshold adjusts up and down based on your initial level of preparedness.
> That is, apply to and interview at companies you don’t want to work at first to get more practice problems.
If you have enormous amounts of time to waste you might as well do a take-home.
I honestly hate spending 4-6 hours of my time on a pointless take-home, but I suck at leetcode and I absolutely refuse to spend any time on learning it, so they’re the lesser of two evils.
The difference is that you are spending 4-10 hours of your time trying to pass the process for 1 company. If you instead invest 4-10 hours doing leetcode prep, you are studying to pass the process for LOTS of companies. Economies of scale is a huge factor here
Is your 4-6 hour estimate a total of all jobs a candidate applies for?
If you are interviewing for 10-20 jobs in a particular week, a 2-4 hour take home test is a full time job 40 hours worth of work.
I would accept a 2-4 hour time commitment but only much later in the interview process, when much of the interviewing was done and the list was whittled down to 2-3 serious companies.
Seriously, if a company wants me to spend more than an hour of my time on something for them, they better be including a check as well.
Less competition for everybody else looking at the position, then.
The test might be longer, but I don’t have to spend time studying and practicing something I’ll never use on the job. I use the same knowledge I build up while working
From the paper you link:
"This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63)"
1. The research is dated (1998). Long before many of the current best practices in SW Eng were established.
2. Says it is based on 85 years of research. Obviously not IT-related then.
3. Even if we get past that it gives 3 almost equally good methods of hiring where the highest one is GMA and integrity test - not work sample test.
4. Even if we get past all that work sample means work sample. It does not have to be produced under pressure in a weekend as unpaid work which as any professional knows is very hard to bring yourself to do right (being a professional means getting paid for my services as I live from selling them). It can very well be some past work on github etc.
So, unless there is a better/more focused on IT/up to date research proving that take home tests (which btw can be offshored/gamed very easily) lead to better hires I remain highly skeptical of all that and big fan of whiteboard/pair programming.
Re: 3 - if you read the full paper you'll see that on its own a work sample test has (barely) the highest predictive value (but a lower confidence in that than GMA which is more heavily studied). This quote I think does more to demonstrate that GMA and integrity are less correlated than GMA and work product testing or structured interviews, which is intuitive.
Re: 4 - I'm not sure why you would consider a take home test less valid as a work sample test and then prefer a whiteboard test. Certainly the latter is less representative of real working conditions. (I probably would not want to work in a place where whiteboarding is more representative, at least!)
Your other two points are reasonable threats to validity but I don't think especially strong ones. The research covers many different professions so I think the onus would be on you to explain why software engineering is so different.
Yeah I think #1 and #2 are valid criticisms. My issue with the parent comment's line of reasoning is that it doesn't apply the same level of rigor to evaluating whiteboarding as it does to work sample tests. Our goal should be to identify which evaluation approach is most predictive based on the evidence available.
One thing I wish I could see but probably isn't compilable is how much an impact the test has on the final decision or what stage it's at.
Lots of the comments are pointing out a 2-4h takehome is a poor value prop from the candidate's side compared to a 1h leetcode which transfers better across companies. (I'd agree with this.) But a 1h leetcode is usually just a first hurdle to what's often another 1-3 rounds, while for me the point of the 2-4h takehome is that it's the only thing. You do it, we grade/talk about, we can arrange a more formal team meeting if you want but that's the whole process.
Am I alone in wanting shorter interviews but longer tests?
I couldn't agree more.
While it's not compilable, I did reach out to a few of the 5-star test designers and asked them this question. How teams use their tests (both in weight and stage) varies a lot, but the well-designed tests usually featured as a central component of the hiring process. They were often given at the stage right before the final round of interviews, but occasionally earlier. The 5-star test designers I spoke with weighted the test heavily because it: 1. helped them reduce bias, identifying great candidates even when they didn't have the backgrounds they expected 2. gave them a foundation for the future interviews. It's less stressful on candidates when they're already familiar with the code being discussed live (because they wrote it!)
Anecdotally, on-the-job performance of the candidates they ultimately hired seemed to correlate with performance on the test. I realize this is very hard to compare against other evaluation approaches though!
After scanning the first page, I quickly noticed "Want something unique to your team? Tell us what you need."
I'm not a marketing guy (by any stretch) so maybe my opinion isn't worth much, but I think this is outstanding - provide something already useful, and then invite people to purchase your customized even more useful stuff.
It's always nice to see something well done — kudos!
(You seem to be focused on software development, but if you have anything along the lines of practical math used in a lab/shop/factory environment around machining, mixing, monitoring, testing, etc., It'd be great to hear about it)
Thank you! I appreciate this because I think I'm terrible at marketing.
I happen to be an Applied Math major and spent years teaching competition math classes, though this was a while ago. If I can be helpful, shoot me an email: alex@trytapioca.com.
Why are there no 1-star projects listed?
The reason why there aren't any 1-star projects isn't because they don't exist, but because we didn't think anyone would want to see them. Cataloguing these was a ton of work (we sifted through hundreds of tests). Including the 1-star ones seemed like it would only shame companies who used them.
I appreciate that you might not want to name-and-shame. I wonder if you could just scrub the identifying info and post them, though?
Personally I think it'd still be tremendously valuable to show clear examples of what "bad" looks like in your ranking system. Not only does it help you show off more of the hard work you did, but it also helps all your readers get a more complete understanding of your framework.
Beyond that, it's generally helpful -- in learning anything complex -- to see examples of "what not to do, and why."
You've given me a great idea for a future post - thank you! I'll see if I can do a more detailed teardown of a "great" take-home vs. a "bad" one.
These are great in theory. In practice, I've regretted the handful of times I've done them. It seems like some companies use this as a way to "waste" less of their interviewers' time. One particular dysfunctional company that everyone here would recognized gave me the feedback: you're one of the candidates out of nearly 200 that did this [7-10 hour take home test] and passed! Then they ghosted me for a month before asking me to come on for a full day onsite.
As a technical hiring manager, I was involved in some teams that did use homework assignments.
Their fatal issue on the hiring side is that they take too much time to properly score.
If you ask a good candidate to work for 4+ hours, they'll produce code that will cost 2+ hours of a great engineer to thoroughly check.
Lots of companies are very fond of giving candidates huge tasks and having them spend 4+ hours on these. But practically no company will have their best senior engineers (a very scarce resource) dedicate dozens of hours per month to check these tasks.
Hence, these tasks effectively never receive the resources required to properly check them. Especially at the more senior level. If you're hiring a senior engineer, and giving them a 4 hour homework task, a staff-level engineer would have to spend 2-3 hours to properly check it.
Forget about it, that's not happening.
Instead, these teams do ask the senior engineer to take 4 hours to work on the task. However, once the hard part arrives, the team will never check it properly. They'll assign some junior engineer for an hour to check it.
The team won't care, because it's the candidate's time being wasted, while theirs is efficiently preserved. Plus, our junior engineers are so amazing, they can certainly score the work of a senior in less than an hour.
That's awful! I'm sorry you had to go through that. The intent isn't usually malicious, but some teams (esp larger ones) are really disorganized and don't value candidates' time enough.
My team is doing our best to create the "perfect" take-home experience and reduce the frequency of these horror stories.
Sorry to break this to you. If you believe there's "perfect" take-home, you will be wasting your time. The perfect take-home is no take-home. Take-homes may make some sense for engineer with little experience.
I think a lot of people are getting hung up on the fact that these challenges take a long time. Yep, and that's a problem...
But the goal of having a candidate work on a real problem is a good once. IMO a far better approach is to do a live exercise with them. Not a leetcode problem... take an existing codebase and add a feature to it, fix an issue with it, etc. Heck, you don't need them to complete everything, but you just need to get signals on how they approach the task, how they pick up working in an unfamiliar codebase, attention to detail, communication, etc.
Anything to get rid of leetcode interviews that are a collosal waste of everyone's time and lead to awful hires. Over the years there has been a democratization of study material for "coding interviews" which has led to a huge influx of wildly inexperienced/unqualified candidates. It also means if you want to hop to another company, you have to waste months grinding leetcode problems completely irrelevant to anything you'll ever do.
Why not, you know, interview them? See if you like them, see if they are reasonably knowledgable about what you need them to do. Putting people on the spot doesn't mean they can or can't develop software or solutions. Sending homework is even worse though.
I'm a big fan of live exercises too, mainly because it's a great way to see how candidates think and how they collaborate with others. There are only a couple tradeoffs with this format (every approach has them): live exercises can be more stressful and teams with many applicants won't have the bandwidth to offer this to everyone who could be qualified.
My favorite is a combination: short (1 hr) take-home followed by live discussion/pairing with anyone who does a half-decent job. It reduces stress because candidates will already be familiar with the code (they wrote it!) while being efficient with time.
I actually don’t mind take home tests, but I want the company to also burn resources as to disincentivize ghosting.
Problem with take homes is that you can spend hours and then they can spend seconds or minutes reviewing.
At least with traditional interviews the time spent is equal for both parties.
Agreed. That's been exactly my experience on the few I did before I refused to them. The onus is all on you and your only interface is basically a flakey recruiter. There's no guarantee that any will every look at your submission.
As a candidate, it's an awful feeling to spend time only to get ghosted.
It's usually not malicious, just disorganization. From a hiring manager's perspective, it's too easy for things to slip through the cracks. When the volume of candidates increases, it's hard to keep track of all the zip file submissions + individual repos while making sure the eng team reviews them all.
This is one of the problems my team is hoping to solve through software. So far, we're seeing teams working with us getting back to 100% of candidates (often with personalized feedback), with median times as fast as 1 day. I hope we can reduce the frequency of terrible take-home experiences.
I didn't say or even imply that it was malicious. Regardless, it's disrespectful and unprofessional. I fail to see how adding yet another SaaS tool into the mix of the dozens that companies already use is somehow going to fix what is essentially a cultural problem. Applicant Tracking Systems have already been a thing for a long time now.
From talking to many hiring managers who use take-homes and using 3 ATSs myself, most engineering take-home processes don't run through the ATS.
I agree with you that it's disrespectful to candidates. My point is that it's not just a cultural problem - it's also a tooling problem.
As an interviewer, I believe take home tasks are unfair and also poor return on investment.
Take home tasks incentivise candidates to burn as much time as possible on the task. Consequently, they are a test of who is willing to put a week of work even when the instructions say it should not take more than 3 hours to complete.
I prefer pair programming with the candidate on the premise that if I ask you to spend 2 hours of your time it is only fair that I also spend this time with you. And also give you a chance to learn about me, your future boss.
Not to mention I can learn more about you working with you for half an hour than I would ever learn by studying the code you might or might not have written yourself.
As a candidate, I vastly prefer a take-home to live whiteboarding or pairing. That background sensation of being observed and evaluated greatly interferes with my ability to concentrate and perform at my best.
(I'll note that I have no problem pairing in real-world work scenarios! I love discussing problems with my team. It's the artificial pressure injected by the interview process that really gets to me)
I have been in situations were I had to pair program for an interview. The artificial pressure is true, I don't know how to react and even speak out when I'm trying to show my coding skills.
I don't have issues with whiteboarding and pair programming with my current teammates. I can speak up and have enough confident to do. But its not the same in an interview situation. I prefer take-home as well.
How would a take-home compare to a hybrid format? For example, if you were given a 1-hour take-home followed by a live session asking about your thought process and then pairing to extend what you previously worked on.
I think a hybrid format is the right way to do take-homes. The candidate is less likely to resent the time invested in the take-home if they have a chance to discuss or "show off" their work. It also gives candidates who have less free time to complete a polished solution the chance to say "if I had more time, I would [add tests, optimize this function, ...]>"
I still struggle with the "observer effect" during live extension exercises, but it's less pronounced than when starting with a problem from scratch. If I prepare a high quality solution in advance and do well in technical discussions, that's usually enough to offset any fumbling around during the live programming session due to interview jitters.
It is not about what candidates prefer. If that was the case, there would be no whiteboards or any kind of exercises at all.
It actually is, because the interview process influences the size, diversity, and caliber of the pool of candidates who apply to your organization, complete the interview loop and accept offers.
If you have an unlimited engineering budget or a big, attractive name then by all means, force candidates to jump through as many hoops as you'd like, you'll still be able to fill open reqs. But the majority of companies are not in that position and in fact struggle to attract and retain technical talent.
Perfectly articulated. Thank you!
+1 Vote
A two hour pairing session is hard on people who need to fit their interviewing time into odd hours. It’s also stressful, because it’s so much more consequential than a session of real work with two peers. Not to say you shouldn’t do it—just keep in mind it also has trade-offs. And I soundly agree with the positives you list, especially that it forces the interviewer to commit time like the interviewee.
I did a one hour take-home for my current job. I could see someone taking 2-3 hours on it if they were really rusty, and I think that is a good thing because rusty developers can be very good once back up to speed in a language or domain. Live programming wouldn’t give them a chance.
If you can't find 2h for a Zoom call I start questioning your commitment to get the job, especially when there is no time limit on when you can have the meeting (within reason). When I had an important interview in a place where I really wanted to work I would usually take entire day off and make sure I am well rested, prepared and in right state of mind.
Yes, it is stressful. I try to make interviews less stressful but I also believe ability to perform under stress is a desirable trait. I have been presented with many stressful situations in my past where I had to deal with high stakes, difficult problems on a short notice, with important people watching my every move. Like dealing with an outage that incapacitated an entire national bank and was causing a loss of about 50M USD per day.
Now, I am mostly interviewing senior devs, team leads, tech leads, etc. I would be much more lenient with junior roles but I just don't have time to interview everybody and I also like to give the chance for other people to work on their interviewing skills.
Have you ever considered that solutions to your perceived problem might exist?
I’ve done multiple different takehomes with some kind of time limit in place to avoid that sort of thing. Tools have this built in.
Also, there are definitely candidates who would prefer a takehome. If you’re worried that a takehome might be unfair, you can offer folks a choice to either do the exercise on their own async, or to do it live pairing with you. You might be surprised by how many will choose the former.
I spoke to a head of engineering yesterday who asked candidates pick from 3 options:
1. Bring some work that you did previously (>4 hrs) and we'll discuss it.
2. Complete a 1-hour take-home.
3. Complete the 1-hour take-home live with one of our engineers.
He said that 98% of candidates chose option #2. I was surprised at how high it was!
Counter-Point. If the assignment is properly sized: no more than an hour. Then the candidate doing it at home using their system and their tools gives you a more accurate picture of their actual capabilities than a whiteboard or doing it on your hardware in an unfamiliar setting with a stranger giving every appearance of critiquing your work. This is an interview after all. The power balance whether intended or not is very much different than an actual peer review
If a candidate chooses to burn a whole week working on it that can be safely left up to them I think. But a take home test is not unfair because they decided to burn that much time on it. It is an attempt to let them give the clearest most accurate signal of their skill level and approach to problems. I would argue that not even pair programming accomplishes that as well as a take home assignment.
> Then the candidate doing it at home using their system and their tools gives you a more accurate picture of their actual capabilities than a whiteboard or doing it on your hardware in an unfamiliar setting with a stranger giving every appearance of critiquing your work.
Or, they don't even do the task, but pay someone else to do it for them, so eventually all these take-home tests just go to the same set of expert test-takers who already solved all these tasks. Very efficient, and also worse than useless as a screening tool: the dishonest candidates most willing to cheat will score the highest.
> If a candidate chooses to burn a whole week working on it that can be safely left up to them I think. But a take home test is not unfair because they decided to burn that much time on it.
Of course it is. Worse: it creates adverse selection, from the interviewer perspective.
Top candidates who are looking for their next job while working a demanding full time job will not be able to dedicate more a minimum of time to each of the many take-home tests they are asked to do.
Meanwhile, desperate poor candidates will dedicate x10+ the time (or simply cheat) to pass.
So as an interviewer, you'll end up passing the random desperate candidate who somehow got through your earlier screening, and any good candidate with multiple other options and commitments will submit work that looks far inferior.
If I set a take-home test designed to take two hours, and then very clearly tell the candidate to take two hours, and then they clearly spent way longer than that, then I will consider that a signal that they don’t follow instructions well and don’t value their time appropriately.
How do you know they clearly took over two hours?
We love talking about productivity here. You're telling me employers will systematically fail the candidates who performed most productively in their process, because of a speculation they took too much time?
Well, it's pretty simple if you don't given them the task until they agree they're ready to start, and then record when you receive the submission.
Any candidate capable of actually inventing a time machine should certainly be hired.
Haha yeah we implement 'soft time-boxing' by tracking git commit timestamps. It's not as stressful as having a visible timer and we won't block a submission that exceeds the recommended time, but reviewers can clearly see who took longer and their last commit within the suggested time, which helps to create an apples-to-apples comparison.
No candidate wants to be entered in the Hunger Games for who has the most time to sink into a take-home.
Most companies that use take-home tests don't use such systems.
Also, such a system does not prevent cheating, which is really the worst issue here.
And a reasonable response in any case is "Yah, I know you said that but I don't like to do sloppy work and there were clearly a number of different edge cases that needed to be handled for this to work reliably."
I've seen a middle ground where a timebox is enforced, but there's a discussion question for what next steps would be
Yes, this is what we do at my company. It's the only way this sort of thing is valid IMO... it's not a race, but I want to see demonstration of reasonable output.
That would actually be a huge red flag to me for hiring.
I don't want an architecture astronaut who is never going to ship anything.
An 80% solution when I need it is infinitely more useful than a 100% solution that's months late.
And I'd rather somebody admit that they took longer, say 4 hours to get to the 80% because they were a bit rusty, their IDE on their private machine wasn't as expected (e.g. Intellij CE vs the Ultimate at work), they had to look up how to bootstrap a service from 0 because they hadn't done that in ages even at work etc. than to under deliver a 20% solution in the 2 hours asked.
Limits are fine if you ask me. Give me a couple days in which I can do this on my own time and if my kid needs me for something 10 minutes after I started I can deal with it and just come back to the test an hour later. Specific hours? Screw you, not even gonna start.
The limit is the limit. If you don't finish in the time given, I want to see what you did in that time.
I'm with the GP here, I'd much rather give them a few days and advise how much time they should spend on it than give them a strict timebox - it allows for a lot more flexibility with other responsibilities.
And this would be the opposite for me. Someone properly thinking out the edge cases and handling them (or at least pointing them out in comments) would be much more useful in the roles I'd be hiring for.
I need a stable system that can tolerate the real world and handle messy inputs without shitting the bed. A new hire that shows they consider their inputs and handle edge cases is a better fit to me than someone who focused on that 80% solution and didn't bother sanitizing inputs and crashes because of it.
> the candidate doing it at home using their system and their tools gives you a more accurate picture of their actual capabilities
We’re just gonna run with the assumption that anyone who’s applying for a job has their own development machine.
I like this concept - have a one time download link, and a timer to submit solution within an hour or two. Seems like a good balance of fair and realistic (assuming you can't Google for solutions and all).
"can't Google for solutions"?
I will never understand the desire to handicap interviewees like that. I recently passed an live-coding interview in which I was asked to use an unfamiliar library; bc it was open-book I was able to look up its docs and use it to solve the problem. At one point I also hopped onto MDN to confirm a browser API. The ability to search and effectively leverage available resources is fundamental to our profession. Closed-book "pop quiz"-style puzzles -- where the surface area of the relevant material is unbelievably vast, and the relevance of the problem to the job's requisite skills is tenuous at best -- are highly problematic.
I don't want to speak for the last commenter, but I think they meant google for literal solutions to this take-home (i.e. cheating to beat the timer).
I do strongly agree that people should be able to use whatever resources they normally have (google, mdn, autocompleting editor, etc.). This is how they would work on the job. Getting quizzed on easily referenced random trivia about language features is so dumb.
OP was not suggesting that search engines cannot be used as a tool in the process of taking the test. Rather, that your test should not be something which a candidate can Google a pre-existing solution to and submit as their own.
What if I have a lot going on and need to divide my time? I understand why you're choosing to do this, but ultimately it's assuming a lot on part of the interviewee, and doesn't offer any flexibility.
If you gave me such a code challenge, I'd probably take it as a sign that company culture is to put employees under pressure and I would immediately decline.
Compared to committing to a specific hour block of time for a normal interview, isn't it more flexible?
I guess it depends on how the company uses it - if they just stick the take-home in as an extra filter vs trying to replace live interview time
Years ago I interviewed somewhere that had a system like this. I was told ahead of time only to click the link when I was ready to sit down for two hours to do the work because the clock started then.
There are a number of choices.
1.) You can treat it like pretty much any other engineering job and give a "normal" interview, which will vary a lot by company. You probably rely more on credentials than is sometimes the case with software development.
2.) You treat it more like writing, graphic design, video, photography, etc. and expect to see a portfolio. For fields like that, you might talk about thought processes or how they approach their work but the portfolio isn't really negotiable.
3.) In the case of software, you can maybe have someone effectively create a portfolio for the interview but now you're back to someone putting a lot of time into creating something--albeit possibly something reusable.
If you give every person on the street 2 hours of your time, you wouldn't be able to do your day job.
I don't like take home tests either, but the point of them is removing the bad candidates from the pipeline so that people who interview them (often ICs, and these are some of the biggest expenses within the company) don't waste time on someone that is unlikely to be qualified.
Personally when I'm interviewing and I get take home tests I choose my battles. I only do them if I really want the job. If it is worth my time.
It's unfortunate, but take home tests are a necessary evil until something better comes along.
If only there were this magical piece of paper that everyone presented when they applied listing all their relevant skills and experience so that the company can determine if it's worth giving them 2 hours of their time.
All kidding aside, the efficient thing about resumes is that you write them once and use them everywhere. Coding samples are inefficient because you have to take 4h to write one for each company.
Edit: the other problem with takehomes is that it self-selects the people who have scarce-enough opportunities that they find it worth their time to do them.
Right, and the issue with that is that plenty of people lie, and even more exxagerate what they did.
The two things I like about them from a candidate perspective are that they're async, and that they generally have an actual problem to solve, rather than "solve N-Queens." Some companies (Dropbox comes to mind) have coding tests, but they're async, and they also aren't convoluted Leetcode problems.
I've also interviewed people using both live and a take-home review, and don't have much of a preference there. If the former, I can poke your thoughts in realtime, give hints, etc. If the latter, I get to deep-dive into some of your design decisions.
> they generally have an actual problem to solve, rather than "solve N-Queens." Some companies (Dropbox comes to mind) have coding tests, but they're async, and they also aren't convoluted Leetcode problems.
I applied to Dropbox some years ago, and the coding test was an NP-hard problem: https://en.wikipedia.org/wiki/Rectangle_packing
Oh my. Mine was SRE-specific to be fair, and was the kind of thing I'd do for fun to answer "what if..." It was timeboxed to two hours, and IIRC I had a working solution in about half that, then spent the rest iterating and improving.
> I can learn more about you working with you for half an hour than I would ever learn by studying the code
That's the problem. I think everyone is too focused on technical prowess and forgetting something just as important which is written documentation and explanation.
Pair programming has the downside of selecting people with good interpersonal skills and will tell you nothing about quality of code delivered.
A good middle ground I believe is simple takehome tasks with focus on documentation that could be done in no more than two to four afternoons.
>I think everyone is too focused on technical prowess
>and will tell you nothing about quality of code delivered.
These statements seem a little contradictory or I'm misunderstanding the context.
Not being too focused on technical prowess doesn't mean being oblivious to code quality, so no I don't think they are contradictory.
And you don't think that wording can easily be taken as "we focus too much on technical skills" while simultaneously saying "pair programming doesn't guarantee technical expertise"? Many people would think code quality is part technical prowess, at least.
If anything it at least showcases the importance of clear communication and how easily things can get lost in the details.
There is also the principle of the benefit of the doubt where, in doubt, assume the writer is not a complete idiot. Goes a long way to filling these communication details without forcing the writer to write a philosophical thesis.
One solution that seemed to be the best of both is, a take home test followed by pair programming session to add a small feature on top of that, or refactor something, or write tests etc.
Real world challenges needs time to think, often time to refactor after initial working prototype. A take home allows that.
Also the candidate is already relaxed in the pair programming session because it's a familiar codebase. You'll get signal if they didn't write it or wrote something they didn't quite understand.
Do you think that spending 2 hours of someones time, under the scrutiny of a stranger and already under the stress of looking for a new job, is fair?
Has anyone ever asked you to code for them, instead? So they could learn about you and how you deal with problems? Have they asked how you deal with situations under stress? Or maybe they never did so because they were never comfortable enough to do so, since they were in the spotlight?
After doing a few take home assignments, I've learned many won't give the courtesy of Q&A afterwards; many won't give any feedback by default.
I loved giving live code sessions, the more open-ended the better. Give an objective, explain there's no wrong answers, answer all questions, give answers when asked, keep it simple, and gauge response. I found it exposed attitude and knowledge.
I blame the legal teams at large companies who are worried that a well-meaning hiring manager will give feedback that could be used in a lawsuit.
Love that your approach focuses on understanding thought process! IMO many companies focus too much on raw technical skills, when softer skills like attitude may be more predictive of on-the-job performance. My team is working to elicit the same signal in a take-home format (since it's more scalable), but I think the best is a combination of the two: short (~1 hr) take-home + follow-up live session on the work that was already started.
These are great points.
How do you feel about take-homes that have an enforced cap at 60-120 minutes to try to remove the competition on time investment?
Do you learn more than it's possible to learn asynchronously? Most take-homes only have candidates write code, but it's possible to understand a candidate more deeply by asking questions about, for example, how they approached the decisions they made.
How would such a cap be enforced? If it's due 120 minutes from the time it's given, it's no longer a take home interview, it's a remote interview.
It doesn't need to be proctored. It would be easy to start a remote clock and the solution needs to be in by x minutes after the start. It can even be pretty generous. The idea is presumably not to see how fast they can sprint but just put some time-boxing in place so some people aren't taking days.
People can get help of course but that's going to be the case absent effectively a remote proctored assignment. And now you're back to effectively in-person.
> People can get help of course but that's going to be the case absent effectively a remote proctored assignment. And now you're back to effectively in-person.
I would vastly prefer an "in-person" interview in which I wasn't expected to speak to the interviewer to the in-person interviews we actually have. There is a world of difference.
You don't want to ask questions?
Why would I want that?
Because it's an opportunity to learn more about the company beyond what's on their website? Honestly, if I interviewed someone and they had no interest in asking any questions, that's probably an easy reject.
Presumably the difference is that the interviewee gets to choose when to start the clock.
That's what I had in mind. You would need a tool to enforce it.
At that point "take-home" may not be the best name anymore, but "remote interview" just sounds like it's an interview over video. Maybe "async interview"?
It is already a thing. I had a 2hr take-home before that was utilizing hackerrank.
But it wasn't a leetcode-style one (which are common), it was a skeleton of a React app, and I had to implement certain methods to tie it all up together.
The timer would start ticking from the moment I click "begin", and it will warn you that the timer will start after you attempt clicking (so that you don't trigger the timer by accident).
I have used https://takehome.io/. The timer starts when they check out the repo containing the project and instructions.
and however it was enforced, it would increase the stress a great deal
True. More stress than an untimed take-home, but I would think less stress than a normal interview (for most engineers, at least).
And the reality is that a lot of/most people interviewing in those type of situations arre very well familiar with time-boxed exams at school. (And even with take homes the reality of final exams as I recall is that you mostly didn't have the hours in the day to spend an outsized amount of time on one course.)
If the take home assessment is scoped to a small amount of time (1-2 hours is small IMO) with the expectation that the seniority level you're selecting for could complete it in less than that time, I don't see any problem as long as you don't put arbitrarily constrained time limits on how long you are willing wait for the submission.
I agree with you that if the candidate is going to spend the time to do the exercise, the hiring manager should be spending an equal amount of time as well in the process. But that effort can come from the detailed review of the code, the time commitment of which should be in the service of being able to have a meaningful conversation about the candidate's submission in later rounds (which is useful for your last point about discovering what its like to actually work with the candidate).
As a hiring manager I wholwheartedly agree with this. Take home tests introduce so many uncontrolled variables, I wouldn't be able to measure and compare two candidates.
A two hour coding session is a nope from me, and a lot of talent will feel the same, you're going to miss out on some good people.
You're going to learn little to nothing regardless of whether you're face to face or not. Modern interview processes are an exercise in feeding the other side things they want to see/hear. I'm a firm believer that good fit cannot be determined until the candidate has been in the seat for a month or so.
Unless you pair at your job, I disagree with this. Emulate your actual work environment as much as possible. That’s what you’re actually trying to get signal about.
Best of both worlds to use this compilation as inspiration for one’s own company pie programming sessions.
I don’t find it useful to evaluate whether or not a potential hire can map and reduce a list in JavaScript or fix some silly react bug thags so obvious it hurts. Seeing how they approach real problems, architecting solutions on the fly etc are things I enjoy doing but I’m not as much on the hiring side so YMMV
How do you structure the 2 hour session?
Would be interesting to hear which kinds of tasks you use in a short space of time.
One of my favorites is a code review + take an existing codebase and add some functionality to it or fix some design issue with it.
Have been doing SE for 10 years now. Got burned on three of these recently because they didn't used to be as common. Talking several hours each one. It leaves you with the feeling that the company is stealing your work. Will not repeat.
Also, one company had me do a SQL take home which I thought was amusing because then one of the employees nit-picked all the SQL code which I hastily wrote because it was already a risky time investment, even though it was a Python job and I'd likely just use an ORM.
I would argue that a truly qualified interviewer should be able to have a dialogue where they ask you very pointed questions and give you maybe a 20 minute coding problem. If they're not satisfied, they should invite you back for another interview. The major drawback of take-home tests is they are usually very subjective and huge time investments.
I ultimately got 3 job offers, got to the final round at 3 others which had take homes, and accepted one at a FAANG company who didn't use these poor interview tactics but to each their own.
What about a take home test that should take 15min or less? Something not as simple as fizzbuzz but still takes < 15 lines of code?
100%. The signal you get from pair programming is enormous. You not only get to see the end product but how they think. Also it's a good gauge if the person is effective at communicating/working with others.
Why would I do any of these? They are a huge time sink and I have plenty of job opportunities without jump through these hoops.
Depends on your target salary and luck. I do not earn a FAANG salary but my interview was a single 1 hour session where I showed them a few of my github projects and talked shop. My impression is most people going for high salaries need to do the leetcode grind and 8+ stages of interviews though.
i honestly prefer a take-home exercise to live-coding.
at least with a take-home exercise, i can take my time, do in a few hours, without the pressure of having someone looking at what i'm doing.
While hiring I've done a 15min test before an interview (always looking through their portfolio first) that took some candidates 30-45mins. If they fail or took longer than 45mins it ends there. I then do a 30min interview that sometimes take 45min if they talk a lot or ask me questions about the company
How does this sound? I have no idea if this is ideal but so far we didn't hire any more bad candidates than we did using a different process
Because they weed out people that just do a bunch of leetcode from people that actually know how to write quality software.
So there are two possibilities:
- There is no reason to take them and this process will slowly fail over time as companies who adopt these policies slowly push away talent options.
- You are not the target audience of a take home assignment hiring process.
I think it's easy to believe the first since it affirms your world view. Speaking as somebody who has a deep pocket of job opportunities and has taken one of these take home assignments, sometimes those opportunities become stale or you're in a tough market or need a job immediately. A number of companies who have bad hiring practices are still fairly acceptable to work at, even if it's not ideal.
TLDR - to pay the bills.
It's my turn today to complain about website building instead of content:
When macOS developers write in CSS "overflow: scroll;" they get the same behavior as they do with "overflow: auto;", but they don't realize that viewers on Linux and Windows now see ugly default operating scrollbar at all times, especially ugly when the dropdowns aren't supposed to have horizontal scrolling.
Test web software on two browser and two operating systems :)
Oh dang, thanks for flagging this. It's fixed now :)
Beautiful!!
Picked one at random, the Zero5 assessment, technologies: SQL. The first question was a scenario where I would build a backend endpoint using an API/Lambda and a web frontend. Nothing to do with SQL except for the 'database' icon in the architecture diagram. Rated 5 stars.
Sorry, looks like we didn't tag this one sufficiently! We're updating this now. There was a TON of content to filter through - we're doing our best :)
I also think there's a surprising number of edge cases that need to be considered for this challenge. Consider cars that were already parked at the start of the time range, ones that were entered during the range and never left, etc. So I think it does require familiarity with SQL.
Did you disagree with the tagging, the stars, or both?
It's painful to imagine how many teams miss out on candidates because their take-home or technical interviews are not extracting the right signals. Having access to a library like this will save so much time on getting new ideas and iterating. At the very least, I hope it saves the effort of running into useless leetcode libraries.
too many companies are using the take home as an initial filter. twice now I've spent a couple days really polishing the assignment only to discover later that there is absolutely no chance we will work together well.
at least make me _want_ the job before you start placing demands on my time and those of who I might provide as references.
Or it'll just cause the leetcode bar to be moved to fit the tests with "stronger signals".
Goodhart's Law will basically apply to any exam under capitalism: "When a measure becomes a target, it ceases to be a good measure."
If these jobs didn't offer more compensation and prestige than other positions, matching candidate competency to job requirements would be much more achievable, but as it is now, it's an adverseal system that demands gamification (because when any candidate plays the game, if you don't, you are actively harming your material conditions with a lower position and smaller salary)
Everything will be gamed when there are stakes.
Whatever is closest to the real skills you need will makes it harder to improve at the measure without also improving at the real job skills. At least that would be the dream.
All of ours[1] are given three stars but I would love to know why! I feel like they meet your criteria, but who knows.
Would love yours, or anybody else's feedback on our problems. We do care about them, and wrote about why we use homework here: https://adhoc.team/2018/02/26/why-we-use-homework-to-recruit...
Hey Bill! Rating aside, I'm a big fan of Ad Hoc's homework assignments. Here's what I love about them: 1. Your team has made them public for others to use. This is HUGE.
2. The library is organized and the prompts are generally clear and explicit about the requirements to complete.
3. You offer support for candidates with questions about the assignment.
4. Kathy Keating told me about the grading environment you all use with rubrics, blind grading, and rotating cohorts of graders. We actually created similar tooling for the teams we work with.
I also want to explain why they're rated as 3 stars (and why I think this undersells how good they are). When we designed the rating criteria, it was really important to recognize tests that could extract hiring signal without requiring a ton of time from candidates, since that's one of the biggest issues that candidates face. So one of our criteria was "setting clear expectations for candidates (e.g. time expectations)" and another was that the time requested was "reasonable...(<4 hours)". So a test that stated upfront that it would take 8-10 hours would meet only one of those criteria. You can see the full rubric if you hover over the "5-star scale" text in the sub-header.
Unfortunately, the Ad Hoc tests don't specify time anywhere (technically missing both of these criteria) while doing many other things well that aren't captured in our rubric (e.g. candidate chat tool, blind grading). I admit that our rubric isn't perfect and it feels like the Ad Hoc tests are being doubly penalized for something that's easy to fix. In fact, if you add this to the tests, I'd happily update these to 5-stars.
Finally, if you're up for a chat sometime, I'd love to meet you. I really appreciate the work that your team has done to improve the hiring experience for candidates beyond those at Ad Hoc! You can reach me at alex@trytapioca.com
Disclaimer: one of my take home assignments I designed is part of the list compiled by the author.
---
Every interview process will inherently be a flawed one. Having hired several hundred engineers in my career, I believe that it is essential to try to create an interview environment where candidates feel they are given the best possible chance at showing their potential.
There's plenty of bias in engineering already that puts women, people of color and minorities at a disadvantage - and this shows through how most companies approach engineering hiring.
In one of my previous roles, we looked at who made it though to subsequent interview rounds and found that we favored candidates with very similar backgrounds to our own. More than we had thought, as it turns out, than we believed going into the analysis. Our solution to reducing this bias up front was to design take home assignments that tried to resemble the day to day job as much as possible, and with measurable criteria for what we wanted candidates to cover to determine their experience and skill.
It took a tremendous amount of work to get this process right, with a number of iterations to figure out the right balance for how much time a candidate should spend on an assignment and how much of day to day work our engineers should allocate to evaluating candidates.
Over the course of two years, we were able to achieve an almost equal split between women and men being hired into our engineering organization, less employee churn and a generally more vibrant workplace.
Take home assessments aren't a silver bullet, and they don't address every piece of the hiring puzzle. I do, however, believe that they are much better positioned to help interviewees perform at their best in an environment they feel comfortable in. This is especially true for people who today are less represented in software engineering roles.
At the end of the day, take home assignments are one part of a larger toolbox of tools companies must make use of to hire great people. Just like any other approach, take homes can be abused or misused, but I think they deserve more attention than most hiring managers give them.
Use them, or don't. Just make sure you hold yourself accountable to your own bias and try to reduce it so you don't miss out on great talent that don't fit your predefined view of what a great candidate looks like.
As there are no tests rated 1 star, I wonder if the scoring is too lax.
I've taken some of these – shocked that Algolia's made it to a 3 star. Look at how much work this is: https://github.com/algolia/solutions-hiring-assignment
Yeah I agree that a test that's super long is awful for candidates. We did our best to be objective in grading, so we didn't want to dock more stars based on our subjective opinion on what's considered reasonable.
FWIW my team believes that test designers should strive to scope tests to no more than 1-2 hours (ideally 1).
The reason why there aren't any 1-star tests isn't because they don't exist, but because we didn't think anyone would want to see them. Cataloguing these was a ton of work (we sifted through hundreds of tests). Including the 1-star ones seemed like it would only shame companies who used them.
I guess what I’m saying is that if that is a 3 star, I’m not sure how valuable the rating is.
That's an absurd amount of work to do for free. Plus, it is useless: You won't learn anything new, it is going to impact no one, and it is going to sit stale at the inbox of 2 senior developers for 2 weeks until it is seen, as a chore.
This vicious cycle of wasting time must stop.
Holy shit, y'all weren't kidding. That Algolia task would be many thousands of dollars of work at agencies I've worked at, and I'd expect $1-2k even bidding as a freelancer (yes, just for a mockup of this size and complexity, not prod-ready). That's so far beyond reasonable, it can no longer even see its border.
Also, WTF is this about?:
> Important: Do not fork this repository to create your assignment. Doing so will wake a bot that prints out your code, immediately sends it to the shredder, and archives your application in our applicant tracking system. And anyway we’d rather give everyone an equal shot to show us what they can do.
They want you to host it on GH Pages later on in the instructions, so what's the point of this restriction?
FWIW mine was even worse.
It didn’t have a design or business example.
I had to come up with a fake business, explain how Algolia would provide value to them, and make the implementation and design.
Their wording on time:
> Note: We know how time constrained we all are, so the target is to spend no more than 6-8 hours on the assignment. If you do wish to spend more, please go ahead!
And their criteria:
> Evaluation criteria of the demo:
- Quality of relevance settings (eg. searchable attributes, custom ranking, index configuration settings)
- Richness of UI functionality. What kind of controls are provided to the end user to best capture their intent and ultimately retrieve the best possible content to put in front of the user to drive engagement and find what they are looking for? What features are leveraged in this endeavor (eg. query rules, geo, personalization, AI features etc.)?
- Quality of design. How does the layout, styling etc. contribute to the overall feel of the experience?
- Ability to drive a narrative with the demo, connecting how the user experience will drive ROI for a business
That time wording would be enough of a red flag for me, unless I really really wanted to work for that company.
I like the assignments where a starting project is provided, and you are given a few enhancements and bug fix tasks within the project. That is probably what your first few months will be like at the job anyway. You can match the coding standard of the existing project rather than agonizing about naming standards or other subjective decisions. I have an issue with the more open ended tasks where you create a project from scratch. I saw the directive "make the interface as polished as possible". That is how a 3 hour project turns into a week of work.
I agree! It's great when companies take the time to do this. Though it's tough when it's been a few months/years and the code hasn't been maintained. You'll see some repos in the library that are up to 8 years old.
One of the ideas my team brainstormed is to create and maintain a library like this that engineering teams can rely on.
Unless I'm desperate I'll never do an unpaid take home test again in my life. For me it has been a waste of time and hugely demoralizing. Best/worst story and the one that broke camel's back: at one uber-copycat company I had my test failing on one reviewer. Then after a few months another reviewer interviewing me for another company told me that "this test would be fine by me. Had I reviewed it when I was back at the uber-copycat company I'd pass it".
And my experience was more or less like that with all unpaid take home tests. So, bitter truth is I have more to my life than doing boring unpaid work with minimal chances of getting to an offer and TBH I doubt I'd enjoy working in a company that hires like that. So, my stance is give me as much whiteboard or pair programming or whatever _time_bounded_ test you want and let's go from there.
PS: I looked one of the tests. It said "follow SOLID". I'm not gonna say what I think of SOLID here but it's good that they say what they adhere to beforehand. I mean if you're going to have people pouring hours and days in your tests at least be transparent of what exactly you expect.
What’s wrong with SOLID?
The entry for this company Haraj states:
>"Just fyi, this submission make up to 50% from your overall hiring score. While your CV only make up to 5%. So do your best to create great submission!"
>"UPDATE (2022-02-05):
Due to many great candidates have applied to this vacancy, we decided to close this challenge on Friday (2022-02-11). So we will wait for the last submission until Thursday (2022-02-10) at 23:59 WIB." [1]
This is like a cattle call audition with a hard deadline. Who would do this? Also there is absolutely no indication of how much time one should spend this. I'm guessing this is intentional. If you want to give me a deadline there better be pay involved. I hope people are smart enough to call BS on these companies. Hopefully a company making demands one's time without compensation is a huge red flag for people.
People with a bad CV but good skills would do it. There are a lot of those people entering the market every year so it makes sense for some companies to look for gems among the people other companies considers unhireable.
That doesn't seem to make sense. If someone had good skills why wouldn't they just embellish their resume a bit? Also there's still a preliminary phone screen interview before even asking a candidate to complete this.
Once, when I started out, I did a take-home test and didn't get the job. The feedback was literally "In some of the files, you mixed tabs and spaces" - which was not true. The penny dropped a few years later: there never was a job. I did some kid's homework.
Give candidates real work & pay them for their time. https://blog.algora.io/post/2022-04-pay-when-hiring-part-2/
He calls it a "Ride Along": top candidates spend a day of coding with his team and receive $500.
How do you figure out if they're top candidates?
First we did screening & technical interviews to arrive at our 5-10 most promising candidates
Right, so we're back at square one doing "technical interviews" again.
That's just how Thomas at Cloudthread.io run his process, he used real paid work for his final hiring decision. It's not the only way. We personally (Algora.io founders) jumped straight into paid work without prior interviews.
We shared a hiring bounty with a dozen applicants purely based on their resumes, never spoke with anyone on the phone, received 5 PRs, awarded & hired 3 of them.
You awarded 5 of them, or 3 of them?
If you're only paying the people you hire, that's not paying people for their time.
I mean sorry, but if you did a good job, we're going to make you an offer such that you won't need to quibble about $500. The entire difficult part is what you're asking from people you don't hire!
Received 5 PRs, awarded the top 3. You can actually see them https://app.algora.io/algora/challenge/pulls/5
We clarified it in our bounty issue that only the best-winning submissions would get the bounty. Other folks have chosen to award all PR submissions regardless of hiring outcome. The format is up to the company hiring.
You're making a great point! It's exactly about what we ask from people who don't get hired.
I think real-world coding, getting reviewed by the CTO / hiring manager and getting compensated beats standardized coding tests & imaginary take-home assignments any time of the day.
I mean you do you, but it's a total farce to call this paying people for their time. You're paying people who win your little contest, which is what hiring already is before your startup tries to insert itself as a middleman.
I'd call it an early experiment and what we could afford at the time to evaluate candidates for a SWE summer internship. Maybe other companies will like this approach and offer challenges that pay much better, with different setups & time horizons. Ex a user turnt his challenge into a 3-month contract-to-hire arrangement that paid $15k to the person who did the work & got the job.
Ultimately it's about getting the right signals. Real work produces the best signals. The rest of the mechanics are a WIP :)
Really appreciate your input, thanks a lot!!
> what we could afford at the time to evaluate candidates for a SWE summer internship
You didn't even hire them as FT? Christ, what a submarine charade this entire thread has been.
submarine charade, that's pretty funny lol.
The principle of engineers evaluating engineers based on real work remains the same, whether it's for an internship, flexwork, junior or senior roles, no?
I don't see how it isn't fraud to claim that.
did you read the blog post though? :)
Kevin Hale at Wufoo (YC W06) did contract work for a month instead of interviews. Is that not "give candidates real work & pay them for their time"?
Thomas paid every engineer who did a 'Ride Along' (a day on the job with the Cloudthread team) regardless of his final hiring decision, and he did it with a bunch of folks.
Elliot paid every PR he received for his screening bounties on Algora regardless of his final hiring decision.
I think the title for the blog post checks out.
My attitude is that take-home tests should be given to candidates if they cannot share work they've already done.
This creates a win-win incentive because it would become standard in our industry to have a portfolio of GitHub contributions, a stack overflow profile, etc. At least those artefacts are helpful to a wider community as well as showing your abilities.
Couple this with an open ended technical conversation with the engineer, progressively drilling down on items they know about to see the depth of their skillset, you will understand the key skills they can present for the job opportunity.
I have never in my life done something you can find on Github and I doubt I ever will. I have worked on plenty of stuff that I can share generic 'war stories' on, stuff you might even have 'used' without knowing but that isn't actually anything I could really "show" to you. Other than said war stories, like 'did you ever ride a train and it was late? Yeah I worked on xyz backend thing which ultimately is involved in getting that info to that info screen at the train station'.
And no I don't think you can put any incentives anywhere to get that code onto a Github repo, nevermind a public one.
Hardware Reverse Engineer here. Would love to see some more tests geared towards my field. (Either reverse or hardware engineering)
I've never run into someone who that is their career - do you work for companies that are trying to take the competitions stuff apart, or legacy hardware, or is there some other stuff that's obvious that I'm missing?
Think more along the lines of "bad guys" vs "good guys" rather than corporate competition. Perfectly harmless systems are built everyday. But, the consumers of those systems may not always be harmless. Understanding - and eventually controlling - the inner workings of these custom systems [covertly in some circumstances] is often necessary to stop the bad guys.
Just a nitpick but when I see engineering without software in front of it I think EE or mech. This is a useful resource I almost skipped over. I realize certain industry segments are probably so steeped in the jargon that its obvious to people on the inside but its not to everyone.
Just a nitpick but when I see engineering without software in front of it I think EE or mech. This is a useful resource I almost skipped over. I realize certain industry segments are probably so steeped in the jargon that its obvious to people on the inside.
Software engineering has been turned into Star Search: https://m.youtube.com/watch?v=FfRihaJxozo
When is Musk finishing that rocket to Mars?
Take-home tests are incredibly disrespectful. Not only is it an implication my work and experience are illegitimate, but it's a request for me to do work for free. Not happening.
My frame is always "I'm a professional meeting another professional to see if what I'm selling is fit for what they're seeking to buy, and at a fair price." You can imagine my shock when the interviewer's frame is something like "Do this work for free to prove to me you're worthy." Sorry to burst your bubble, but I have nothing to prove, my friend.
I like to put things like this in mob terms because it distills things down to their simple essence. Mobsters are all about essence and simplicity. No Bullshit™. This humbles the nerd. Here:
>You like steak? I deliver steak. I could deliver you steak. At a good price too.
https://www.youtube.com/watch?v=ii5CiYxwuMo
That's it. That's the essence of the ideal interview. Two professionals having a conversation. No Bullshit™. Like literally every other industry. I'm a respectable steak delivery man and you are a respectable potential client. That's the frame. None of this "prove to me you're worthy" nonsense. When I deliver, you pay. If I don't deliver, you don't pay. Simple as.
>Before I hire you as a plumber, I just need you to install a section of piping free of charge, because I don't trust you or believe who you say you are.
Sounds ridiculous, doesn't it? It is. You'll be installing your own pipe with this attitude.
There's already a great deal of asymmetry in the hiring process. You're getting paid to interview me, but I'm not. Let's not make it any more asymmetric than it already is.
I understand interviewing is difficult. A lot of this difficulty stems from the fact that many of us in this business are pretty nerdy, i.e. socially awkward, both interviewers and candidates alike. This makes it difficult to probe someone via conversation to see if they'd be a good fit. But difficulty with socializing is no excuse. If you're a socially awkward hiring manager, you either need a different role or need to improve your social skills. If you're a socially awkward candidate, same thing goes for you - improve your social skills.
If a hiring manager tries to saddle you with unpaid work, you tell him "fuck you, pay me". Gabish? Have some self-respect people. The more candidates that set these managers straight, the more they'll be forced to start conducting normal human interviews.
Hugely helpful! Thank you for the work you did compiling these.
I have a better idea. Don't do take-home assignments as a candidate and don't give them as part of your interview process as an employer. Problem solved.
This is a phenomenal resource. I wish it also included the DevOps category -- k8s, aws, terraforms of the world.
You'll see DevOps as one of the options in the 'Roles' filter with 13 examples in the library.
Did anyone actually read the article? Just kidding, there isn't actually anything to read...
Thank you for this! Great work!
There’s no such thing as a ‘realistic engineering take-home test’. When will we admit tests are not only unrealistic, but an utterly useless predictor of…anything?
Very useful, thanks!
Now take that list, throw it in the rubbish bin and wash your hands thoroughly.
Take home challenges are simply an absurd waste of time and a huge disrespect towards your candidates personal time. You ask them to spend 4 or more hour of their free time on a useless, throwaway excersise that you "review" in 5 minutes and you conclude they're good enough because they laid out code the way you like it... You never see the really important things that you should be screening for: how they explain their approach, their though process, how well they communicate, the compromises they had to make to fit within the timeframe, and why they'd made those compromises, how they react and adapt to changing requirements and feedback...
It's just flawed on so many levels...
As a new Engineering Manager I've eliminated the take home challenge in my new company in favour of this process:
- You have open source code or closed code that you can show? Walk me through it. Our best engineers passed the interview this way.
- You don't have code to show, 1 hour live coding challenge, focus on communication, search want you want online, ask us any questions, and explain what you're doing.
We had about 40 such interviewes this year, one person refused the live challenge, 11 passed this stage, and we ended up with 7 hires accepting the offer.
If you lack inspiration, use this: https://github.com/guardian/coding-exercises
Your process seems equally flawed. Many excellent candidates have no open-source code, and if they show you any closed-source code that's an immediate signal against hiring them.
So now you've put candidates through a silly "live coding challenge" – offering an advantage to the sort of person who is chatty and communicative with strangers in a stressful environment and the expense of people who may be better engineers but slower to warm up.
I really wish folks would be way less dogmatic about this stuff. Every single time I see someone say "take-home exercises are bad", their proposed alternative is also critically flawed.
It might be flawed in terms of filtering out the type of person that does not do well in live pairing scenarios.
But I disagree it's _equally_ flawed. The time investment from a candidate is significantly smaller at the cost of more time investment from the employer's side (aka my side). Personally I consider this a reasonable trade-off.
Another huge problem with take home challenges is that they are one time use. At least if you're grinding leet code questions you're spending time that is useful at multiple companies.
I have a big folder on my computer of homeworks I've finished for various companies over the years. If I recall I passed all of them, it's sort of annoying that I can't simply hand over this list of other projects I've passed and say "here, all of these companies thought this was good".
What I like about your proposal is that if many other companies did it, the result would be every few years I should take a month of evenings and build out a cool project for myself demoing my ability to write code. If life gets to busy, it's fine if the project ages a bit.
This way you can scratch a personal itch and prep for interviews all the same time.
Did the companies tell you no when you asked them if you could share a previous project instead of completing theirs? From talking to many hiring managers, they also don't want to waste your time, and most will be happy to accept an alternative if it show similar skills to the ones they're looking for. I'm even seeing more teams offering this as an explicit option now.
There will always be a few who say no (IMO this could be a warning sign), but it doesn't hurt to ask :)
> it's sort of annoying that I can't simply hand over this list of other projects I've passed and say "here, all of these companies thought this was good".
Why not?
Put them in a repository, and in your cover letter when applying to a new company paste a link to it, and say exactly that.
It's your code after all.
As a software engineer with significant experience I think the option to explain some existing code is valuable. This option allows me to show my skill with something that is typically more interesting and demonstrative than a quick coding test. As a professional I also value my time and find explaining how I know I can help with what needs to be done takes far less time and effort than going through a coding test and then discussing that.