Encouraging interviewees to say “I don’t know” improves performance
morehappy.meI think part of this conditioning from taking tests in school. For twelve or more years of your life, in any situation where you are asked to demonstrate your knowledge, a null response is treated in the same negative fashion as an incorrect response, and getting outside help is disallowed. No wonder we try and guess, we've literally been trained for years that it's the best approach! Compounding that, an interview is the one situation in "the real world" that most resembles a testing situation from grade school, which will bring back all those bad habits.
My graduate advisor is Spanish, but is faculty at a university in the States. In his first year of teaching, he penalized for incorrect answers (as is apparently the Spanish custom). But his students threw fits during office hours, so he caved in the face of potentially terrible teaching evaluations.
After tenure, he said, things will be going right back to the way they were in the old country.
I could see throwing a fit, too, depending on how the weighting is handled for partially incorrect answers. If the question is "What shape is the Earth?", if a student answers "A cube", I can see taking off points. However, if they answer "A sphere", do they also lose points? Do they lose just as many points as the person who said sphere or do they get partial credit for being closer to the correct answer of "oblate spheroid"
To put it differently, would I get a better score in OS design than Linus Torvalds? After all, he wrote an operating system that contains bugs, while I wrote no operating system as all.
That's a general problem with the interpretation of test answers. What an appropriately penalized scoring scheme for a multiple choice tests does is set the expected value of guessing to zero, providing an unbiased estimator of performance for students who guess randomly.
Ah. I didn't realize he was doing this on a multiple choice test. It makes complete sense in that context. The Spanish threw me - none of my language courses had multiple choice exams.
I've seen people try and take this scoring scheme out of the multiple choice arena and into a more general program. However, when the answer space is infinite, the expected value for guessing is already zero, so there really isn't any point, besides sadistically torturing students.
Sorry, I should have clarified-- my boss is Spanish but works in computational biology, not Spanish lang/lit.
In the real world "I don't know" has more utility than a wrong answer. So you can argue that education should follow that route, too.
Actually, the SAT (American college entrance exam) penalizes wrong answers more than non-answers. A non-answer gets you no points, but a wrong answer gets points subtracted.
Barely. IIRC it's 0 points if blank, -0.25 if wrong, so if you eliminate 1 choice it's worth it to guess. I'd like to see it be -5 points if wrong, since this is more reflective of the real world. When I entered the industry as an electrical engineer this was a difficult transition for me. In school, you're incentivized to guess and BS. But in the real world, the penalty for being wrong is HUGE.
> in the real world, the penalty for being wrong is HUGE
In some situations (committing an integrated circuit design to manufacture probably being one), the cost of being wrong is pretty big.
Other situations where there's a tight feedback loop and you can make/propagate changes quickly (say, most web app software development), it's less so.
True. But generally the cost of guessing and doing something wrong is much higher than saying "I don't know" and asking someone else or Googling the result. School, through homeworks and especially exams, trains us to do our best without outside help. I would argue that this behavior is maladaptive to engineering in the real world.
In the real world, the penalty for being wrong is dependant on the risks involved, and sometimes you can manage the risks.
Guessing is a good skill. Managing risk is a good skill. Learning how to situationally value guessing vs. knowing is a awesome skill.
The SAT is the exception rather than the rule. Most standardized and non-standardized tests reward you for guessing.
The GMAT as well. It's interesting when you work with little kids - you have to encourage them to guess. They're more likely to admit what they don't know.
School basically teaches all the wrong lessons for people going into professional work. Memorize minutiae, plug and chug, devalue high level understanding, concentrate on trivial problems, etc. It's a shame that apprenticeship has fallen out of favor, it's such a vastly superior method to teach professional skills.
A comparison with Germany might be insightful here. Apprenciteship is going strong there.
Some commenters, and possibly the author, want this advice to extend to job interviews, but the reason an interviewer is pushing you to answer a question isn't that they want you to be right or wrong. A good interviewer wants to get a look into how you think and how you approach a problem.
An even better interviewer will make that clear, by telling you "there's no trick here", "I don't know the answer myself, let's see what we can figure out", or "there's no one right answer, I just want to know what you can see here". A trick I've employed a fair bit lately to get a reserved interviewee to start working with a question is "What is the worst solution we could provide to attack this problem?" I'll even possibly go as far as offering my own horrible solution, and asking them where we fo to improve on this.
And I do mean worst. I haven't met a candidate yet who can't at least throw out ideas on how to improve my horrible solutions, and at that point the ball is rolling.
Asking somebody for the worst possible solution seems like a really clever ice-breaker. I'm stealing that technique! Thanks patrickyeon!
I interviewed some folks a while ago for a simple desktop support position and I had a lot of trouble getting anything out of the candidates. It's possible that they were all just terrible, but I'm guessing it was more probably my approach.
I gave them a very simple test[1], and I was met with sort of deer-in-headlights stares. Hence forth I think I'll start with a problem and an example horrible solution.
[1]Presented the applicant with a screenshot from a desktop computer that couldn't connect to the internet. The screenshot showed several possible indicators (impossible network settings, cable unplugged, etc...).
Extending it to interviews doesn't work partly because the findings are so unsurprising.
Permitting "I don't know" doesn't actually seem to help candidates get more correct answers in the study (looking at the actual paper, the group discouraged from saying "I don't know" actually got a slightly higher number of questions right at the first time of asking)
It simply means those encouraged to say "I don't know" guess wrongly fewer times - no surprises there - whilst still revealing their ignorance to the interviewer.
Applied to job interviews: interviewing people for a job by asking simple factual questions the interviewer knows the answer to is doing it wrong. If they're indiscriminately negatively marking "wrong" answers that aren't prefaced with an explanation that guessing is inadvisable, they're doing it even more wrong - how candidates answer questions they don't know is valuable decision making heuristic.
A more usual interview will involve many questions where there is no clear "correct" answer, in which case "I don't know" will usually be one of the worst possible answers. Even where the questions are fact-based, a decent interviewer should usually give more credit for how a person guesses than an admission of ignorance, in which case "I don't know" is neutral at best. Sure, some candidates that aren't good at guessing or are especially bad at bullshitting will appear worse than if they're encouraged to say "I don't know", but that's valuable information for the interviewer which lost by encouraging everyone to give a non-answer. Same applies if humility is a key requirement and "I don't know" actually is a decent answer.
TLDR: Since correct answers are unaffected, allowing candidates to say "I don't know" only improves the interview performance of weak candidates.
> "And I do mean worst."
I am not convinced that throwing INTERCAL at a candidate is a good way to roll. ;)
But seriously, I think that you are spot-on with this approach. Interviewing, done effectively, is about getting a fleshed out idea of the capabilities of the candidate, not just getting correct answers. Perhaps the best response to give or expect, when the interview gets stuck, is "I don't know, but..." Even on questions that aren't coding questions, these are often good responses: "I don't know, but [this is how I would find out]" or "I don't know, but [this is a topic or problem that I believe is related]" .
For what it's worth, my 'worst solutions' always involve a guy with a van driving around hassling people in person. Somehow people never accept my proposed solution, so I haven't been able to test it.
> A good interviewer wants to get a look into how you think and how you approach a problem.
I consider that a red flag. In this case, the interviewer is almost certainly less concerned with whether or not you can do the job than they are concerned with whether you think like them.
Though I do like your suggestion to try offering a terrible solution to get the conversation started. Usually it's breaking the ice that's difficult.
I don't know who downvoted you, but I don't quite get what you are saying (why is that a red flag and what exactly do you mean that's a red flag?) Red flag as in it is theoretical (like "of course that would be nice.....but that's usually not true)? or red flag as in interviewer is tricking the interviewee to do bad job? (making bad impression to interviewer)?
I've been in interviews where the interviewer had me play the guess-what-answer-I'm-thinking-of game, and that may be what j_baker is afraid of. I don't want to do that either when I say "get a look at how you think"; I want to see that you can attack a problem, not necessarily that you'll try the same things I would.
Absolutely. I rather have an interviewer questions me why I think my solution is a solution (even if it is a terrible n^3 solution).
That's a great technique and one which I particularly like because it also tends to be more representative of the real job: you almost always want to work with the person who says “Slow down, let's make sure we understand this…” when handed a hasty spec and proposed solution
I have long suspected this to be true. That's my argument against mandatory voting, because if people can choose not to vote, they can express they don't know and hopefully leave the result on people who feel they do know.
However, it's a bit unclear how this ties together with Dunning-Kruger effect (which is also used as an argument against democracy). The D-K effect would suggest that if you allow people to express doubt, the experts will doubt more, and overall performance will decrease.
It would be interesting if someone did a psychology experiment on that. (I am actually in favor of doing these kinds of demonstrations in high school, because it's in my opinion important for people to understand how democratic voting works, for example the fact that voting won't get you a simple average of the results - which is usually used as an argument against democracy.)
> Dunning-Kruger effect (which is also used as an argument against democracy)
Democracy isn't about having an efficient system, it's about having a stable system. Democracy is an awful way of getting things done, it's just a reasonably good way of stopping very bad things from happening.
Favorite quotation from Kelly Johnson, former head of the Lockheed Martin Skunk Works:
> Voting on everything prevents anything very stupid from happening, but also anything very brilliant.
You can guess who called the shots in his aircraft design teams. He did, because he was truly fucking brilliant.
Yes, exactly. It's inefficient, that's the point of it.
He didn't have the same control over things as a dictator would. You can do these things in business more safely because it's less likely to kill large numbers of people.
Yeah, that's the value of a "benevolent dictator" in business. Skunk Works had one, Apple, Python, Ubuntu, and SpaceX too, and all blazed incredible trails in their fields.
I wish more people understood this. People alwayss complain about gridlock, but gridlock is great. It might block progress sometimes but it blocks regresssion just as often. It ensures that changes tend to have wide popular support.
That's not how mandatory voting works, at least in Australia, one if the few democracies to have it. It's mandatory to show up to the voting place and get your name ticked off the roll. It is not mandatory to actually cast a vote. You can turn around and walk back out as soon as your name has been crossed off. Or you can cast a blank ballot. Or an invalid ballot, or a donkey vote etc etc
> I have long suspected this to be true. That's my argument against mandatory voting, because if people can choose not to vote, they can express they don't know and hopefully leave the result on people who feel they do know.
Mandatory voting can include "I don't know" (and, in fact, it naturally falls out in a simple way from any use of preference ballots with a tallying system that accounts for ties and treats all unranked candidates as tied for the position immediately below all explicitly-ranked candidates.)
EDIT: Further, such a system would naturally handle the more common case where people do have known preferences, but they aren't accurately represented either as a forced preference ranking of all candidates (as with preference ballots that reject ties), or a unique most-preferred status for one candidate (as is the case with the "bullet" ballots typically used in FPTP elections.)
I'd actually be pretty surprised if the Dunning-Kruger effect applied significantly to politics. There are so many tribal pressures involved that I think that the phenomena involved with Dunning-Kruger would be overwhelmed.
In representative democracy, probably yes. But D-K effect is commonly used as an argument against (semi)direct democracy.
You can have mandatory voting with a "none of the above" option on the ballot.
And the point of such a system is? To spend more money and time of everyone?
In Australia, citizens have a duty to pay taxes, defend their country (if required to do so), serve on a jury (if their name is drawn), and vote. That's the cost of living in a democracy.
If voting was compulsory, only lunatics would vote. I wouldn't voluntarily pay taxes, sign up to fight a war, or serve on a jury either - I'm not crazy; but I think it's important that these things get done.
Hm, thinking about it, maybe I was wrong. Maybe you can actually express "I don't care" vote in the mandatory voting system as well, by voting for a randomly selected candidate (and these contribution will average out to same votes for all candidates). But it's rather impractical compared to simple non-mandatory voting.
What about the difference between "I want X, but there's no way they're getting in, so I won't bother" and "I don't want any of them"?
That's a technical issue. There are voting systems with points, and you can vote against someone rather than just for them. And you can vote for many people. I've said it once and I'll say it again, FPTP is a tyrant's idea of democracy.
One argument is to stop efforts to disfranchise voters. If voting is mandatory things like photo ID law in US won't fly (or at least would be much less effective). It also affect who politicians pander to. These days common strategy is to pander to radicals because it's easier to get them to vote. That would disappear if everybody has to vote.
When "none of the above" wins, it usually means new elections where all old candidates are banned or some other procedure to resolve complete distrust to current politicians.
But then the original point stands. Such a system doesn't fully allow you to express legitimate feeling that you don't know. If majority expresses "I don't know", it doesn't suddenly mean "I don't trust old guard" vote. That's misinterpretation of the voting result.
Also, you can express "I don't trust old guard" in the non-mandatory voting system as well. Just vote for the new candidate. So it's actually simpler system than yours and more expressive.
When reading the title, I assumed this was talking about job interviews, a much different type of interview. How do these ideas apply to job interviews? Is it a good idea for an interviewer in a job interview to encourage interviewees to say I don't know, and should you be more ok with saying I don't know and then elaborating in a job interview?
On the job interview point:
I'm always as truthful as possible in a job interview - especially technical ones. My background is not in tech, however I'm an autodidact with programming and whenever technical questions are asked that I may not know, I'll say it. And then I'll say how I think it should/would be done, or the steps I'd take to figure out the answer and solve the problem. Not sugarcoating some of my technical inability has faired me well so far. And then after I hear the question, think about the steps to do it, and/or get coached by the interviewee - I know the answer, and am ready to use it in my technical/creative arsenal. I feel like it's a win-win to not bullshit about something I don't know.
Purely anecdotal of course but I always set the scene for interviewee's that "I don't know" is a completely acceptable answer. If the question could be attempted with an educated guess then "I don't know" is perfectly fine as long as an educated guess is then attempted.
If you only get "I don't know" with no attempt at a guess or explanation for why they don't know then you may have issues.
Those are two separate questions, IMO. In the context of a job interview, however, I believe giving the interviewee permission to say "I don't know" makes the interview go better because they'll spend more time focusing on the interview and less time trying to figure out "what the interviewer wants."
I usually give a little preamble telling them to feel comfortable blurting out nonsense. I'd rather have them blurt out nonsense than sit there for 30 seconds in silence because they're afraid to say something stupid.
You can see the interviewee relax almost immediately.
My view is that if an interviewer dings me for saying "I don't know", that's a sign I wouldn't want to work there anyway.
Without encouragement, candidates answer with "I don't know" during a job interview easily stands out. If we are interviewing for a position, aren't we searching for just ONE outstanding specific individual? Should we really explicitly encourage?
A perfectly acceptable answer to me is "I don't know, but this is what I would do to find out", go on to describe what books you'd refer to, what google searches, etc.
I'll usually ask at least one question in an interview to see if the candidate will answer with "I don't know" or ramble on incoherently making something up. Guess which I prefer.
One of my best technical interviews was from a lead engineer who simply asked more and more difficult questions until he found your limit (believe me, he knew more than anyone he would interview). His interest wasn't in what you knew so much as how you reacted when you hit the limits of your knowledge.
There's a lot of value in that approach. I don't use it myself when interviewing people, but I do try to accomplish the same end.
>believe me, he knew more than anyone he would interview
He wouldn't really need to. By asking questions about very domain-specific knowledge, he could still achieve the same effect with anyone who didn't have a freakishly similar set of expertise.
Corollary: "There's no such thing as a stupid question", because you're not afraid to be wrong. All creative and innovative people are like this, without exception, IMO. As Richard Feynman said in the title of an autobiography: "What do you care what other people think?"
Everybody wins when people leave their egos behind.
Sure. But its not so hard to come up with a question that sounds pretty stupid. What is Monday? Can I eat your shoe for lunch? Why do you have to interview me before hiring me?
I've found that the ease with which someone answers "I don't know" has a positive correlation with their wisdom in general. (Of course, if they answer "I don't know" to everything, they probably just don't care, and that's a separate problem.)
I don't necessarily disagree with you. I think it is true that having too many I don't know means the person is probably not familiar with the area he's facing. The I don't care part is probably a bit rare. You probably have to look at body language (i.e. facial expression, for example).
I second this notion.
One thing i always ask on that chat before the trivia questions, is what tech news sites the candidate read. And where he usually goes to were he wants to know the correct argument order of a function he does not use too often.
That usually gives me a better grasp of how he will solve a real world problem than the trivia questions.
But when hiring for teams that need to get specific work done fast, i do not underestimate the value of the trivia questions as it will show me the ramp up time to get that work done. Of course i apply that on top of the other means to get an idea if the candidate is a good fit overall.
There is no silver bullet for hiring.
The answers may get better, but don't you want to find out which candidates are more likely to bluff when they're wrong? The point of the interview isn't for candidates to give you answers that are as correct as possible. If this were the case, you'd mail them a list of questions, and ask them to get back to you in a week. One goal of interviews is to find out how people behave when they don't know. This is why I try to take people to the end of their knowledge, no matter where that end is.
In an interview i have never had a problem saying that I don't know something or that I do know about something but can't recall the specific answer.
I normally try and turn that scenario into a quick discussion where I hopefully let the interviewer display their vast knowledge and insight. Assuming the question is relevant I am always geniuenly interested in finding out about something I don't know. Seeing that I am not being paid to take the interview I feel it is only fair that I get something out of it.
Nice study. I think it's quite intuitive. When you are forced to spell you are force to say something. So by saying "I don't know" it is natural to continue with "I don't know but I think" and then it continues with more words.
Whereas to tell someone in advance "don't you dare to tell me I don't know" the person will just hold on to the thoughts until he or she has to spell that thought out (because he or she has to say something to get out of the misery).
This is obvious. See also the child abuse panic of the 80s, where kids were pressured to make up stories of abuse. And the general fact that eyewitness testimony is unreliable.
Being comfortable with saying "I don't know" a lot is something I had to kind of train myself to do. It's part of realizing that this is a truly enormous field that we're in, and nobody could possibly know more than a fraction of it. Anyone who doesn't say it much is either spending all of their time in a subset of the field that they know very well, or is throwing around bullshit, either from fear or bravado.
How about just ditching contrived interview questions because companies have no good science or data to establish the correlation with hiring the best people?
It seems past performance would be more reliable. What have you created and what was your role in it? Can you articulate your passion for the problems we are solving? Can you show that your colleagues respected your work and benefited from collaborating with you?
A good interview question should give space for exploration - it shouldn't involve esoteric questions or things that are easily google-able. Good interview questions will give you insight into the candidate's thought process, not what they've memorized beyond fundamentals. So the results of this study seem irrelevant.
It's a nice idea, and it makes sense, but the sample size – 26 people per group, multiplied by 2 for two experiments – is too small to reach any definite conclusions.
lucidity as the fastest path to clarity ?