Explore this post
Need A Quick Summary?
Ask AI.
Pre-formulated prompts you can fire into your favorite AI assistant.
Introduction
Treat "forests" well. Not for the sake of nature, but for solving problems too!
Random Forest is one of the most versatile machine learning algorithms available today. With its built-in ensembling capacity, the task of building a decent generalized model (on any dataset) gets much easier. However, I've seen people using random forest as a black box model; i.e., they don't understand what's happening beneath the code. They just code.
In fact, the easiest part of machine learning is coding. If you are new to machine learning, the random forest algorithm should be on your tips. Its ability to solve—both regression and classification problems along with robustness to correlated features and variable importance plot gives us enough head start to solve various problems.
Most often, I've seen people getting confused in bagging and random forest. Do you know the difference?
In this article, I'll explain the complete concept of random forest and bagging. For ease of understanding, I've kept the explanation simple yet enriching. I've used MLR, data.table packages to implement bagging, and random forest with parameter tuning in R. Also, you'll learn the techniques I've used to improve model accuracy from ~82% to 86%.
Table of Contents
- What is the Random Forest algorithm?
- How does it work? (Decision Tree, Random Forest)
- What is the difference between Bagging and Random Forest?
- Advantages and Disadvantages of Random Forest
- Solving a Problem
- Parameter Tuning in Random Forest
What is the Random Forest algorithm?
Random forest is a tree-based algorithm which involves building several trees (decision trees), then combining their output to improve generalization ability of the model. The method of combining trees is known as an ensemble method. Ensembling is nothing but a combination of weak learners (individual trees) to produce a strong learner.
Say, you want to watch a movie. But you are uncertain of its reviews. You ask 10 people who have watched the movie. 8 of them said "the movie is fantastic." Since the majority is in favor, you decide to watch the movie. This is how we use ensemble techniques in our daily life too.
Random Forest can be used to solve regression and classification problems. In regression problems, the dependent variable is continuous. In classification problems, the dependent variable is categorical.
Trivia: The random Forest algorithm was created by Leo Breiman and Adele Cutler in 2001.
How does it work? (Decision Tree, Random Forest)
To understand the working of a random forest, it's crucial that you understand a tree. A tree works in the following way:
1. Given a data frame (n x p), a tree stratifies or partitions the data based on rules (if-else). Yes, a tree creates rules. These rules divide the data set into distinct and non-overlapping regions. These rules are determined by a variable's contribution to the homogeneity or pureness of the resultant child nodes (X2, X3).
2. In the image above, the variable X1 resulted in highest homogeneity in child nodes, hence it became the root node. A variable at root node is also seen as the most important variable in the data set.
3. But how is this homogeneity or pureness determined? In other words, how does the tree decide at which variable to split?
- In regression trees (where the output is predicted using the mean of observations in the terminal nodes), the splitting decision is based on minimizing RSS. The variable which leads to the greatest possible reduction in RSS is chosen as the root node. The tree splitting takes a top-down greedy approach, also known as recursive binary splitting. We call it "greedy" because the algorithm cares to make the best split at the current step rather than saving a split for better results on future nodes.
- In classification trees (where the output is predicted using mode of observations in the terminal nodes), the splitting decision is based on the following methods:
- Gini Index - It's a measure of node purity. If the Gini index takes on a smaller value, it suggests that the node is pure. For a split to take place, the Gini index for a child node should be less than that for the parent node.
- Entropy - Entropy is a measure of node impurity. For a binary class (a, b), the formula to calculate it is shown below. Entropy is maximum at p = 0.5. For p(X=a)=0.5 or p(X=b)=0.5 means a new observation has a 50%-50% chance of getting classified in either class. The entropy is minimum when the probability is 0 or 1.
Entropy = - p(a)*log(p(a)) - p(b)*log(p(b))
In a nutshell, every tree attempts to create rules in such a way that the resultant terminal nodes could be as pure as possible. Higher the purity, lesser the uncertainty to make the decision.
But a decision tree suffers from high variance. "High Variance" means getting high prediction error on unseen data. We can overcome the variance problem by using more data for training. But since the data set available is limited to us, we can use resampling techniques like bagging and random forest to generate more data.
Building many decision trees results in a forest. A random forest works the following way:
- First, it uses the Bagging (Bootstrap Aggregating) algorithm to create random samples. Given a data set D1 (n rows and p columns), it creates a new dataset (D2) by sampling n cases at random with replacement from the original data. About 1/3 of the rows from D1 are left out, known as Out of Bag (OOB) samples.
- Then, the model trains on D2. OOB sample is used to determine unbiased estimate of the error.
- Out of p columns, P ≪ p columns are selected at each node in the data set. The P columns are selected at random. Usually, the default choice of P is p/3 for regression tree and √p for classification tree.
-
Unlike a tree, no pruning takes place in random forest; i.e., each tree is grown fully. In decision trees, pruning is a method to avoid overfitting. Pruning means selecting a subtree that leads to the lowest test error rate. We can use cross-validation to determine the test error rate of a subtree.
- Several trees are grown and the final prediction is obtained by averaging (for regression) or majority voting (for classification).
Each tree is grown on a different sample of original data. Since random forest has the feature to calculate OOB error internally, cross-validation doesn't make much sense in random forest.
What is the difference between Bagging and Random Forest?
Many a time, we fail to ascertain that bagging is not the same as random forest. To understand the difference, let's see how bagging works:
- It creates randomized samples of the dataset (just like random forest) and grows trees on a different sample of the original data. The remaining 1/3 of the sample is used to estimate unbiased OOB error.
- It considers all the features at a node (for splitting).
- Once the trees are fully grown, it uses averaging or voting to combine the resultant predictions.
Aren't you thinking, "If both the algorithms do the same thing, what is the need for random forest? Couldn't we have accomplished our task with bagging?" NO!
The need for random forest surfaced after discovering that the bagging algorithm results in correlated trees when faced with a dataset having strong predictors. Unfortunately, averaging several highly correlated trees doesn't lead to a large reduction in variance.
But how do correlated trees emerge? Good question! Let's say a dataset has a very strong predictor, along with other moderately strong predictors. In bagging, a tree grown every time would consider the very strong predictor at its root node, thereby resulting in trees similar to each other.
The main difference between random forest and bagging is that random forest considers only a subset of predictors at a split. This results in trees with different predictors at the top split, thereby resulting in decorrelated trees and more reliable average output. That's why we say random forest is robust to correlated predictors.
Advantages and Disadvantages of Random Forest
Advantages are as follows:
- It is robust to correlated predictors.
- It is used to solve both regression and classification problems.
- It can also be used to solve unsupervised ML problems.
- It can handle thousands of input variables without variable selection.
- It can be used as a feature selection tool using its variable importance plot.
- It takes care of missing data internally in an effective manner.
Disadvantages are as follows:
- The Random Forest model is difficult to interpret.
- It tends to return erratic predictions for observations out of the range of training data. For example, if the training data contains a variable x ranging from 30 to 70, and the test data has x = 200, random forest would give an unreliable prediction.
- It can take longer than expected to compute a large number of trees.
Solving a Problem (Parameter Tuning)
Let's take a dataset to compare the performance of bagging and random forest algorithms. Along the way, I'll also explain important parameters used for parameter tuning. In R, we'll use MLR and data.table packages to do this analysis.
I've taken the Adult dataset from the UCI machine learning repository. You can download the data from here.
This dataset presents a binary classification problem to solve. Given a set of features, we need to predict if a person's salary is <=50K or >=50K. Since the given data isn't well structured, we'll need to make some modification while reading the dataset.
# set working directory
path <- "~/December 2016/RF_Tutorial"
setwd(path)
# Set working directory
path <- "~/December 2016/RF_Tutorial"
setwd(path)
# Load libraries
library(data.table)
library(mlr)
library(h2o)
# Set variable names
setcol <- c("age",
"workclass",
"fnlwgt",
"education",
"education-num",
"marital-status",
"occupation",
"relationship",
"race",
"sex",
"capital-gain",
"capital-loss",
"hours-per-week",
"native-country",
"target")
# Load data
train <- read.table("adultdata.txt", header = FALSE, sep = ",",
col.names = setcol, na.strings = c(" ?"), stringsAsFactors = FALSE)
test <- read.table("adulttest.txt", header = FALSE, sep = ",",
col.names = setcol, skip = 1, na.strings = c(" ?"), stringsAsFactors = FALSE)
After we've loaded the dataset, first we'll set the data class to data.table. data.table is the most powerful R package made for faster data manipulation.
>setDT(train)
>setDT(test)
Now, we'll quickly look at given variables, data dimensions, etc.
>dim(train)
>dim(test)
>str(train)
>str(test)
As seen from the output above, we can derive the following insights:
- The train dataset has 32,561 rows and 15 columns.
- The test dataset has 16,281 rows and 15 columns.
- Variable
targetis the dependent variable. - The target variable in train and test data is different. We'll need to match them.
- All character variables have a leading whitespace which can be removed.
We can check missing values using:
# Check missing values in train and test datasets
>table(is.na(train))
# Output:
# FALSE TRUE
# 484153 4262
>sapply(train, function(x) sum(is.na(x)) / length(x)) * 100
table(is.na(test))
# Output:
# FALSE TRUE
# 242012 2203
>sapply(test, function(x) sum(is.na(x)) / length(x)) * 100
As seen above, both train and test datasets have missing values. The sapply function is quite handy when it comes to performing column computations. Above, it returns the percentage of missing values per column.
Now, we'll preprocess the data to prepare it for training. In R, random forest internally takes care of missing values using mean/mode imputation. Practically speaking, sometimes it takes longer than expected for the model to run.
Therefore, in order to avoid waiting time, let's impute the missing values using median/mode imputation method; i.e., missing values in the integer variables will be imputed with median and in the factor variables with mode (most frequent value).
We'll use the impute function from the mlr package, which is enabled with several unique methods for missing value imputation:
# Impute missing values
>imp1 <- impute(data = train, target = "target",
classes = list(integer = imputeMedian(), factor = imputeMode()))
>imp2 <- impute(data = test, target = "target",
classes = list(integer = imputeMedian(), factor = imputeMode()))
# Assign the imputed data back to train and test
>train <- imp1$data
>test <- imp2$data
Being a binary classification problem, you are always advised to check if the data is imbalanced or not. We can do it in the following way:
# Check class distribution in train and test datasets
setDT(train)[, .N / nrow(train), target]
# Output:
# target V1
# 1: <=50K 0.7591904
# 2: >50K 0.2408096
setDT(test)[, .N / nrow(test), target]
# Output:
# target V1
# 1: <=50K. 0.7637737
# 2: >50K. 0.2362263
If you observe carefully, the value of the target variable is different in test and train. For now, we can consider it a typo error and correct all the test values. Also, we see that 75% of people in the train data have income <=50K. Imbalanced classification problems are known to be more skewed with a binary class distribution of 90% to 10%. Now, let's proceed and clean the target column in test data.
# Clean trailing character in test target values
test[, target := substr(target, start = 1, stop = nchar(target) - 1)]
We've used the substr function to return the substring from a specified start and end position. Next, we'll remove the leading whitespaces from all character variables. We'll use the str_trim function from the stringr package.
> library(stringr)
> char_col <- colnames(train)[sapply(train, is.character)]
> for(i in char_col)
> set(train, j = i, value = str_trim(train[[i]], side = "left"))
Using sapply function, we've extracted the column names which have character class. Then, using a simple for - set loop we traversed all those columns and applied the str_trim function.
Before we start model training, we should convert all character variables to factor. MLR package treats character class as unknown.
> fact_col <- colnames(train)[sapply(train,is.character)]
>for(i in fact_col)
set(train,j=i,value = factor(train[[i]]))
>for(i in fact_col)
set(test,j=i,value = factor(test[[i]]))
Let's start with modeling now. MLR package has its own function to convert data into a task, build learners, and optimize learning algorithms. I suggest you stick to the modeling structure described below for using MLR on any data set.
#create a task
> traintask <- makeClassifTask(data = train,target = "target")
> testtask <- makeClassifTask(data = test,target = "target")
#create learner
> bag <- makeLearner("classif.rpart",predict.type = "response")
> bag.lrn <- makeBaggingWrapper(learner = bag,bw.iters = 100,bw.replace = TRUE)
I've set up the bagging algorithm which will grow 100 trees on randomized samples of data with replacement. To check the performance, let's set up a validation strategy too:
#set 5 fold cross validation
> rdesc <- makeResampleDesc("CV", iters = 5L)
For faster computation, we'll use parallel computation backend. Make sure your machine / laptop doesn't have many programs running in the background.
#set parallel backend (Windows)
> library(parallelMap)
> library(parallel)
> parallelStartSocket(cpus = detectCores())
>
For linux users, the function parallelStartMulticore(cpus = detectCores()) will activate parallel backend. I've used all the cores here.
r <- resample(learner = bag.lrn,
task = traintask,
resampling = rdesc,
measures = list(tpr, fpr, fnr, fpr, acc),
show.info = T)
#[Resample] Result:
# tpr.test.mean = 0.95,
# fnr.test.mean = 0.0505,
# fpr.test.mean = 0.487,
# acc.test.mean = 0.845
Being a binary classification problem, I've used the components of confusion matrix to check the model's accuracy. With 100 trees, bagging has returned an accuracy of 84.5%, which is way better than the baseline accuracy of 75%. Let's now check the performance of random forest.
#make randomForest learner
> rf.lrn <- makeLearner("classif.randomForest")
> rf.lrn$par.vals <- list(ntree = 100L,
importance = TRUE)
> r <- resample(learner = rf.lrn,
task = traintask,
resampling = rdesc,
measures = list(tpr, fpr, fnr, fpr, acc),
show.info = T)
# Result:
# tpr.test.mean = 0.996,
# fpr.test.mean = 0.72,
# fnr.test.mean = 0.0034,
# acc.test.mean = 0.825
On this data set, random forest performs worse than bagging. Both used 100 trees and random forest returns an overall accuracy of 82.5 %. An apparent reason being that this algorithm is messing up classifying the negative class. As you can see, it classified 99.6% of the positive classes correctly, which is way better than the bagging algorithm. But it incorrectly classified 72% of the negative classes.
Internally, random forest uses a cutoff of 0.5; i.e., if a particular unseen observation has a probability higher than 0.5, it will be classified as <=50K. In random forest, we have the option to customize the internal cutoff. As the false positive rate is very high now, we'll increase the cutoff for positive classes (<=50K) and accordingly reduce it for negative classes (>=50K). Then, train the model again.
#set cutoff
> rf.lrn$par.vals <- list(ntree = 100L,
importance = TRUE,
cutoff = c(0.75, 0.25))
> r <- resample(learner = rf.lrn,
task = traintask,
resampling = rdesc,
measures = list(tpr, fpr, fnr, fpr, acc),
show.info = T)
#Result:
# tpr.test.mean = 0.934,
# fpr.test.mean = 0.43,
# fnr.test.mean = 0.0662,
# acc.test.mean = 0.846
As you can see, we've improved the accuracy of the random forest model by 2%, which is slightly higher than that for the bagging model. Now, let's try and make this model better.
Parameter Tuning: Mainly, there are three parameters in the random forest algorithm which you should look at (for tuning):
- ntree - As the name suggests, the number of trees to grow. Larger the tree, it will be more computationally expensive to build models.
- mtry - It refers to how many variables we should select at a node split. Also as mentioned above, the default value is p/3 for regression and sqrt(p) for classification. We should always try to avoid using smaller values of mtry to avoid overfitting.
- nodesize - It refers to how many observations we want in the terminal nodes. This parameter is directly related to tree depth. Higher the number, lower the tree depth. With lower tree depth, the tree might even fail to recognize useful signals from the data.
Let get to the playground and try to improve our model's accuracy further. In MLR package, you can list all tuning parameters a model can support using:
> getParamSet(rf.lrn)
# set parameter space
params <- makeParamSet(
makeIntegerParam("mtry", lower = 2, upper = 10),
makeIntegerParam("nodesize", lower = 10, upper = 50)
)
# set validation strategy
rdesc <- makeResampleDesc("CV", iters = 5L)
# set optimization technique
ctrl <- makeTuneControlRandom(maxit = 5L)
# start tuning
> tune <- tuneParams(learner = rf.lrn,
task = traintask,
resampling = rdesc,
measures = list(acc),
par.set = params,
control = ctrl,
show.info = T)
[Tune] Result: mtry=2; nodesize=23 : acc.test.mean=0.858
After tuning, we have achieved an overall accuracy of 85.8%, which is better than our previous random forest model. This way you can tweak your model and improve its accuracy.
I'll leave you here. The complete code for this analysis can be downloaded from Github.
Summary
Don't stop here! There is still a huge scope for improvement in this model. Cross validation accuracy is generally more optimistic than true test accuracy. To make a prediction on the test set, minimal data preprocessing on categorical variables is required. Do it and share your results in the comments below.
My motive to create this tutorial is to get you started using the random forest model and some techniques to improve model accuracy. For better understanding, I suggest you read more on confusion matrix. In this article, I've explained the working of decision trees, random forest, and bagging.
Did I miss out anything? Do share your knowledge and let me know your experience while solving classification problems in comments below.
Subscribe Now
Stay ahead, one post at a time.
Get expert tips, hacks, and how-tos from the world of tech recruiting to stay on top of your hiring!
Thank you for subscribing!
We're so pumped you're here! Welcome to the most amazing bunch that we are, the HackerEarth community. Happy reading!
Get in touch with our friendly team and we’ll get back to you soon.
How to Run a Panel Interview That Gets a Decision
Meta title: How to run a panel interview that produces a decision Meta description: How to run a panel interview that produces a decision, not a debate — a practical guide to structure, rubrics, and debrief that actually close roles.
How to run a panel interview that produces a decision, not a debate
A panel interview is a hiring session in which multiple interviewers evaluate the same candidate against a shared rubric, then reconcile their independent judgments into a single decision. To run one that produces a decision rather than a debate, assign each panelist a specific competency to evaluate, require independent written scorecards before any group discussion, and structure the debrief to focus only on scoring disagreements.
Learning how to run a panel interview that produces a decision, not a debate, starts with accepting that panels don't fail during the interview. They fail in the 20 minutes after — when four people who watched the same candidate walk out with four different conclusions and no way to reconcile them. If your panels regularly end in a Slack thread that stretches for three days, the interview isn't the problem. The debrief structure is.
Most guides on how to run a panel interview treat the session itself as the event. That's backwards. The session is a data-collection exercise. The decision is a separate exercise, and it needs its own rules. Research on structured interviewing consistently shows it outperforms unstructured formats on predictive validity — but only when the structure extends into how the panel makes its decision.
Why panel interviews turn into debates
Panels debate for three reasons, and they're almost never about the candidate.
The first is coverage overlap. Two interviewers ask about system design. Both form opinions. Neither has data on how the candidate handles ambiguity, code quality, or collaboration — because no one was assigned to look for it. In the debrief, the two design interviewers argue with each other while the actual gaps go undiscussed.
The second is rubric drift. The team agreed on a scoring guide six months ago. Since then, two interviewers have started weighing "communication" more heavily, one has quietly stopped caring about testing, and the newest panelist is calibrating against their last company's bar. Same rubric, five interpretations. If you don't already have a shared scoring language, our guide on designing interview rubrics that reduce bias is a useful starting point.
The third is timing. When interviewers submit scorecards after the debrief starts — or worse, during it — the loudest voice in the room anchors the discussion. Everyone else adjusts to fit. This is well-documented in decision science. Research on group polarization — including work by Cass Sunstein at Harvard Law School in Wiser: Getting Beyond Groupthink to Make Groups Smarter (2015) — suggests that groups amplify errors when members share opinions before independent judgment is captured, a dynamic that plausibly applies to hiring panels.
The pre-panel work that makes running a panel interview possible
Before the interview happens, three things need to be locked. Skip any of them and you're building the debate you're trying to avoid.
Assign coverage explicitly. Each panelist gets one or two competencies to evaluate — coding, system design, debugging, cross-functional collaboration, whatever the rubric names. No two panelists cover the same thing. If your rubric has six dimensions and your panel has four people, some dimensions get double-coverage and some get one owner. Decide which before the loop starts, not after.
Calibrate the rubric on a real example. Take a scorecard from a recent hire — ideally one where the panel disagreed — and have the current interviewers score it independently. Then compare results. Where the scores diverge by more than one point on a five-point scale, you have a calibration gap. Fix the rubric language, not the interviewers. This takes an hour. Most teams don't do it, then spend that hour every week arguing in debriefs instead.
Set the scorecard deadline before the debrief. Every panelist submits their scorecard independently, in writing, within 24 hours of their interview and before the debrief begins. No exceptions. If a scorecard isn't in, the debrief doesn't start. This is the single highest-leverage rule in the process and the one most teams refuse to enforce.
How to run the panel interview itself
The interview is the easy part if the pre-work is done. A few operational rules make it easier.
Cap each session at 45 to 60 minutes. In practitioner experience, anything longer tends to correlate with fatigue rather than better signal. Keep transitions between interviewers under five minutes — long gaps degrade the candidate experience and give panelists time to compare notes, which contaminates independent judgment.
Interviewers should not attend each other's sessions unless the format explicitly requires it (a senior hire's system design round, for example, sometimes benefits from a silent observer). Otherwise, the observation becomes a discussion, and the discussion becomes the anchor.
Give the candidate one contact for logistics — usually the recruiter. Panelists focus on evaluation; coordination lives outside the panel. If your interview process still routes reschedules through the hiring manager, that's a workflow problem, not a panel problem. Tools like FaceCode enforce the independent-scorecard rule by storing each interviewer's scores against the rubric before the debrief begins, so the loop lead can see at a glance who has submitted and block the debrief from starting until every panelist is in. That doesn't fix an uncalibrated rubric, but it removes the most common excuse for skipping the rule.
The debrief structure that produces a decision
Here is where most panels lose the plot. This is the part of how to run a panel interview that most teams get wrong. The debrief is not a discussion. It's a structured decision meeting with a specific sequence.
Step one: read the scorecards silently. Everyone opens the submitted scores and comments. No talking for the first five minutes. This forces every panelist to encounter the others' reasoning before hearing their tone.
Step two: identify the disagreements, not the agreements. The hiring manager or loop lead names the specific rubric dimensions where scores diverge by more than one point. Those are the only items discussed. If four panelists gave the candidate a 4 on coding, don't spend 10 minutes agreeing about it.
Step three: each disagreement gets a five-minute cap. The two panelists with divergent scores present their evidence — what the candidate said, what they did, what the rubric asks for. Other panelists ask questions. No new scores are assigned; the goal is to surface what the disagreement is actually about. In our observation across structured debriefs we've seen, a large share of "disagreements" — often the majority — collapse in under two minutes once both sides describe what they saw. They were evaluating different things.
Step four: the hiring manager makes the call. Panel input is data. The hiring manager owns the decision. This is not a democracy, and pretending it is produces the drawn-out debates that panels are famous for. If the hiring manager overrides a strong dissent, they document why. That documentation matters for future calibration and, in regulated industries, for defensibility. SHRM's guidance on structured hiring decisions reinforces the value of documented rationale for later review.
The whole debrief should take 30 to 45 minutes. If yours regularly runs longer, the pre-work is broken.
What to do when the panel is genuinely split
Sometimes the disagreement is real. Two experienced engineers watched the same candidate solve the same problem and reached opposite conclusions about whether the candidate can handle the role. That's a signal, not a bug.
The default move in most companies is to add another round. This is usually wrong. Adding a round rewards the loudest dissenter and punishes the candidate for a process failure. It also signals to the panel that disagreement gets resolved by more interviewing, which encourages performative doubt in future loops.
A better move: name the specific competency in dispute, and design a 30-minute targeted follow-up focused only on that dimension. If two panelists disagree about the candidate's ability to debug production issues, run a debugging exercise. Don't run another general interview. This respects the candidate's time and produces evaluable data on the actual disagreement.
If the split is about seniority rather than skill — the candidate can do the job but not at the level being hired for — that's a leveling conversation, not a hiring decision. Loop the recruiter in to renegotiate the offer level with the candidate before rejecting.
Trade-offs worth naming when you run a panel interview this way
Structured panels give up some things. Serendipity is one — the moment where a candidate mentions a project that unlocks a completely different role fit. Rigid coverage assignments make those moments less likely. Build in a five-minute open-question slot per interview if that matters to you.
Structured panels can also feel bureaucratic to interviewers who take pride in "reading" candidates. That instinct is real, and sometimes right, but it's also where most bias enters the process. If your interviewers resist calibration because it constrains their judgment, that resistance is exactly the reason to do it.
Finally, structured debriefs put more work on the hiring manager. They have to run the meeting, own the decision, and document overrides. If your hiring managers won't do this, no interview format will save you. That's a management problem, not a process one.
Frequently asked questions
How many people should be on a panel interview?
A common practitioner recommendation is three to five, with four as a frequent default. Fewer than three concentrates decision weight on one or two people. More than five produces coverage overlap and slower debriefs without meaningfully better signal. Senior hires sometimes justify a fifth or sixth panelist for a specific competency, but that panelist should have a named coverage area, not a floating observer role.
Should the hiring manager be on the panel?
Yes, but not as the deciding voice inside the panel. The hiring manager interviews for their own rubric dimension, submits a scorecard like everyone else, and then runs the debrief as decision-owner. Conflating panelist and decision-maker inside the panel session is what produces the anchoring problem — everyone else calibrates to the hiring manager in real time.
How do we prevent one senior panelist from dominating the debrief?
Silent scorecard review first, then discuss only disagreements, then five-minute caps per disputed dimension. The structure does the work. If a senior panelist still dominates, the hiring manager needs to actively redirect — "we've heard your view on this dimension; let's hear from the other interviewers." If they won't do that, the debrief structure isn't the fix.
What if the candidate performs differently across interviewers?
Inconsistent performance across interviewers most often signals a calibration problem, not a candidate problem — the panel isn't asking comparable questions or applying comparable rubrics. Occasionally it reflects real candidate variability under different interviewer styles, which is worth knowing. Name the pattern in the debrief: "Interviewer A saw strong debugging, Interviewer B saw hesitation. What was different about the two sessions?" That question usually surfaces the actual issue.
How long should the full panel loop take?
For most engineering roles, four interviews of 45 to 60 minutes plus a 30-minute debrief — so a same-day loop of four to five hours, or a distributed loop over two to three days. Practitioner experience suggests that loops longer than six total interview hours tend to correlate with candidate drop-off rather than better decisions.
Key takeaways
- Panel debates are usually caused by unassigned coverage, uncalibrated rubrics, and scorecards submitted after discussion starts — fix those first.
- Independent, written scorecards submitted before the debrief are the single highest-leverage rule; refuse to start the debrief without them.
- Debriefs should discuss disagreements only, cap each disputed dimension at five minutes, and end with the hiring manager owning the decision.
- When panels genuinely split, run a targeted 30-minute follow-up on the specific competency in dispute — not another full round.
- Structured panels trade serendipity for consistency; make the trade deliberately, and document override decisions for calibration and defensibility.
See it in action
If your panels are producing debates instead of decisions, the fastest audit is to pull the last 10 loops and count how many had all scorecards submitted before the debrief started. If it's fewer than eight, start there. For teams looking to standardize the interview session itself across distributed panels, take a look at how FaceCode structures multi-interviewer coding rounds or schedule a walkthrough of HackerEarth's assessment and interview stack.
How to Get Hiring Managers to Complete Scorecards
Meta title: How to get hiring managers to complete scorecards Meta description: How to get hiring managers to complete scorecards: the conversation, the timing, and the systems that actually move debrief compliance past 80%.
How to get hiring managers to complete scorecards: a recruiter's guide to the conversation that actually works
Getting hiring managers to complete scorecards is less a workflow problem than a negotiation problem. The recruiters who consistently pull scorecards on time have figured out how to make completion feel like the hiring manager's win — not the recruiter's chore. This guide is about the specific conversation, timing, and lightweight systems that move debrief compliance from "chased for three days" to "in the ATS before the next interview."
If you have ever sent the fourth "gentle nudge" on a Thursday afternoon, you already know the standard advice — "make it part of your process" — doesn't survive contact with a hiring manager whose sprint just slipped. What follows is a recruiter-to-recruiter playbook on how to get hiring managers to complete scorecards without becoming the person they mute in Slack.
Why hiring managers don't complete scorecards (be honest about the cause)
Scorecard non-compliance is almost never about laziness. In our experience running assessments and interview loops for hundreds of hiring teams, the pattern breaks down into four causes, roughly in this order:
- The scorecard asks the wrong questions. Fields like "Culture fit: 1–5" with no rubric are impossible to fill in without feeling either dishonest or exposed to a bias complaint. Hiring managers stall because the form itself is broken.
- The debrief window closed. By the time a hiring manager sits down on Friday, the Tuesday interview is a blur. They either fabricate a score or avoid the task.
- No one has explained what the scorecard is for. If the hiring manager thinks it's an HR compliance artifact, it goes to the bottom of the list. If they think it's how the panel calibrates on the next candidate, it doesn't.
- The recruiter is the only person following up. When escalation never happens, the deadline is fictional.
Naming the cause changes the intervention. A recruiter who chases harder solves none of these. A recruiter who fixes the rubric, shrinks the window, reframes the purpose, or builds an escalation path solves all of them.
The conversation that actually works before the interview
The single highest-leverage moment for scorecard completion is the intake conversation with the hiring manager before the first interview is scheduled — not the reminder afterward.
In that meeting, three things get agreed:
- The rubric. What are we actually evaluating? Three to five competencies, each with a behavioral anchor. "System design at senior level" beats "technical strength." If the hiring manager can't articulate what "good" looks like, the scorecard will fail regardless of tooling.
- The completion window. Scorecard due within 24 hours of the interview, no exceptions. This is the number to negotiate hard on. Anything longer than 24 hours correlates with lower quality and higher attrition of detail — the research on memory decay is well-established, and interview debriefs are no exception (see the classic work summarized in Kahneman and Klein, 2009, on expert judgment, foundational but still cited).
- The escalation. "If a scorecard isn't in by end of day the following day, I'll ping you once. If it's not in 24 hours after that, I'll loop in [the hiring manager's manager or the VP of Engineering]." Say it out loud. Get the nod.
Recruiters often skip the third item because it feels aggressive. It isn't. It's the only thing that turns the deadline into a real one. The hiring manager who agrees to escalation up front rarely needs it invoked.
How to get hiring managers to complete scorecards after the interview (the 24-hour play)
Once the interview happens, the mechanics matter more than the reminders. Here is the sequence that works:
T+0 (immediately after the interview): Send a single Slack message with the scorecard link, the candidate's name, and the specific rubric competencies to score. Not a calendar invite. Not an email. A message they can act on from their phone between meetings.
T+4 hours: If not submitted, a second message. This one includes a one-line prompt: "Quick take — recommend/no recommend and one sentence on why. You can flesh out the rubric later." Lowering the bar to a directional answer often unblocks the full submission within the hour.
T+24 hours: If still not submitted, a call — not a Slack ping. Two minutes of "walk me through what you saw" and a recruiter typing the scorecard live. This is the least popular tactic among recruiters and the most effective. It costs 10 minutes. It closes the loop.
T+48 hours: Escalation, as agreed in the intake. Once. Publicly enough that the hiring manager remembers next time.
The recruiters who complain that they "can't get scorecards in" have almost always skipped step three. They pinged four times and never picked up the phone.
Redesign the scorecard so it can be completed in five minutes
If completion still lags after the conversation and timing fixes, the form itself is the problem. A scorecard that takes 20 minutes to fill in will not get filled in.
The scorecard that gets completed on time has:
- Three to five competencies, not 12
- A hire/no-hire recommendation at the top, not the bottom
- Behavioral anchors under each rating so a "3" means the same thing to every interviewer
- One free-text field for "what would change your mind"
- No "culture fit" field without a defined rubric — it invites bias complaints and produces no signal
The trade-off is real: shorter scorecards capture less nuance, and some engineering managers will push back that a five-competency rubric can't evaluate a staff hire. Fair point. For senior roles, add one rubric-anchored deep-dive competency rather than expanding all fields. Depth in one place beats shallowness across ten.
For teams running high-volume technical hiring, structured skills-based assessments can carry more of the evaluative load upstream, so the post-interview scorecard becomes a calibration document rather than the primary signal. That shifts the hiring manager's job from "assess from scratch" to "confirm or challenge the rubric-applied score" — which is a five-minute task, not a twenty-minute one.
The systems layer: what to automate and what to leave human
Automation helps at the edges. It doesn't fix the underlying accountability problem.
What to automate: - Scorecard link delivery immediately post-interview (most ATS platforms — Greenhouse, Lever, Ashby — do this natively) - Reminder pings at T+4 and T+24 - Dashboard visibility for the hiring manager's manager showing outstanding scorecards by owner
What to keep human: - The intake conversation and the escalation agreement - The T+24 phone call - The quarterly review of which hiring managers consistently miss and why
An honest note: vendor dashboards that promise "automated scorecard compliance" tend to overstate what automation alone can do. Reminders don't create accountability; agreements do. The system exists to make the agreement visible, not to replace it.
For teams where interview volume is high enough that the debrief bottleneck is structural — 40+ interviews a week per hiring manager — the upstream fix is reducing the number of interviews that need debriefs, not automating the debriefs harder. Tools like OnScreen handle initial screening with a deterministic rubric so the hiring manager only debriefs candidates who cleared a structured filter. Fewer interviews, tighter scorecards, better calibration.
When to stop chasing and start reporting
Some hiring managers will never comply consistently. That is a data point, not a failure of the recruiter. Track scorecard completion rate by hiring manager as a quarterly metric and share it with the head of TA and the hiring manager's own leader.
The pattern usually breaks one of three ways: - The hiring manager improves once completion is visible - Their leader intervenes - The organization decides that hiring manager shouldn't be leading loops
All three are acceptable outcomes. What isn't acceptable is a recruiter absorbing the compliance cost silently, quarter after quarter, while candidates drop out because feedback took eight days.
Frequently asked questions
How long should hiring managers have to complete scorecards? 24 hours from the end of the interview. Beyond that, memory decay and calendar pressure combine to produce either fabricated scores or no scores at all. Some teams allow 48 hours for senior loops with system design components; that's the outer limit worth defending.
What's a realistic scorecard completion rate to target? Above 85% within the agreed window is achievable for teams that run the intake conversation and the T+24 phone call. Above 95% requires the escalation path to be real and occasionally invoked. Teams that report 100% compliance are usually not measuring accurately.
Should recruiters fill in scorecards on the hiring manager's behalf? Only during a live 10-minute call where the hiring manager talks and the recruiter types, with the hiring manager reviewing and submitting. Recruiters filling in scorecards asynchronously creates a defensibility problem — the person who observed the interview didn't document it — and undermines calibration.
How do you handle a hiring manager who refuses to use the rubric? Escalate once, then involve the head of TA. Rubric-free hiring is a defensibility risk under most fair-hiring frameworks and a calibration risk regardless of geography. This isn't a preference conversation; it's a program-level decision that a recruiter shouldn't be absorbing alone.
Does AI-generated candidate content change how scorecards should work? Yes. If your screening upstream doesn't verify that the candidate you interviewed is the candidate who did the take-home, the scorecard rubric should include a "consistency with prior signal" check. Interviewers flag divergence; recruiters investigate. This is one of the fastest-growing sources of late-stage no-hires we see.
Key takeaways
- The conversation before the first interview matters more than the reminder after — negotiate the rubric, the 24-hour window, and the escalation path up front.
- Redesign scorecards to five minutes of work: three to five competencies, behavioral anchors, and a hire/no-hire at the top.
- The T+24 phone call is the highest-leverage recruiter move for scorecard completion and the most consistently skipped.
- Automation supports accountability but doesn't create it — agreements do.
- Track completion rate by hiring manager quarterly; make the data visible to their leader.
Next steps
If scorecard compliance is downstream of an interview process that's simply running too hot, the upstream fix — structured screening that reduces the number of full-loop interviews — often does more than any workflow change. See how HackerEarth's assessment and interview platform helps hiring teams tighten the funnel before the debrief bottleneck starts.
How to Run a Hiring Intake Meeting That Builds a Rubric
Meta title: How to run a hiring intake meeting that builds a rubric Meta description: How to run a hiring intake meeting that produces a usable rubric, not a wish list. A 60-minute agenda, questions, and traps to avoid.
How to run a hiring intake meeting that produces a usable rubric, not a wish list
Most technical hiring fails at the intake meeting. The recruiter walks out with a job description, a list of "must-haves" that reads like a LinkedIn profile of the departing engineer, and no shared definition of what "strong" actually looks like. Learning how to run a hiring intake meeting that produces a usable rubric — not a wish list — is the highest-leverage thing a recruiter can do for a req.
This is not a strategy exercise. A hiring intake meeting done well takes 60 to 90 minutes, produces a scoring rubric two interviewers can apply to the same candidate and reach the same score, and gets calibrated once with a real resume before the first candidate hits the pipeline. Done badly, it produces a wish list, three months of misaligned debriefs, and a closed req that took twice as long as it should have.
Why most intake meetings produce wish lists, not rubrics
The default intake meeting is a monologue. The hiring manager describes an ideal person, the recruiter takes notes, and both parties leave feeling productive. Six weeks later, when a candidate scores 4/5 on "communication" from one interviewer and 2/5 from another, nobody can point to the source of the disagreement — because the source is that "communication" was never defined.
A wish list has three tells: it lists traits instead of behaviors, it does not distinguish must-haves from nice-to-haves, and it cannot be applied to two different candidates and produce comparable scores. A rubric fixes all three. Research from Google's Project Oxygen and the widely cited Kahneman, Rosenfield, Gandhi, and Blaser work on noise in judgment shows that structured evaluation criteria — not smarter interviewers — reduce inconsistency in hiring decisions.
The wish-list-to-rubric conversion is the actual work of the intake meeting. Everything else is paperwork.
What a usable rubric looks like
A usable rubric names 5 to 8 skills, defines each with an observable behavior, assigns a weight, and specifies which interview stage evaluates it. It fits on one page. Two interviewers reading it independently and scoring the same candidate should land within one point of each other on a 5-point scale.
Here is the minimum viable structure:
- Skill: the capability being evaluated (e.g., "system design for services at 1K+ RPS")
- Definition: one sentence describing what "meets bar" looks like in behavior, not adjectives
- Weight: must-have, strong-preference, or nice-to-have
- Stage: which interview round tests this — take-home, technical screen, panel, or hiring-manager round
- Anchor examples: one description of a 3/5 answer and one of a 5/5 answer
If any row in the rubric cannot be filled in during the intake, that skill is not ready for evaluation. Either the hiring manager needs to think harder, or the skill needs to be cut.
The 60–90 minute intake agenda
Block a full 90 minutes. Meetings under 45 minutes almost always produce wish lists because there is no time to force the specificity conversation. The agenda below assumes the recruiter runs the meeting and the hiring manager is the primary participant, with an optional second interviewer joining for the last 30 minutes to pressure-test the rubric.
Minutes 0–10: Confirm the role's business context
Open with the question the hiring manager has probably not been asked: what does this person deliver in their first six months that makes the hire worth it? Not their responsibilities. Their outputs.
If the answer is vague ("contribute to the team," "help us scale"), keep pressing. A senior backend hire whose first six months are "ship the payments-service rewrite" is a different rubric from one whose first six months are "stabilize on-call and reduce SEV1s." Both are legitimate, but they weight skills differently.
Minutes 10–25: List the skills, then cut half
Ask the hiring manager to list every skill they think matters. Write them all down without pushback. You will typically get 12 to 20 items — some technical, some behavioral, some cultural, some that are actually the same thing renamed.
Then do the cut. Force the hiring manager to rank the list and mark only 5 to 8 as must-haves. The rest become nice-to-haves or get removed. A rubric with 15 must-haves is a rubric that will fail candidates for the wrong reasons and will not survive contact with a real pipeline.
This is the moment where hiring managers push back. A common objection: "But I need someone who has all of these." The honest answer: candidates with all of them exist but will not accept your offer at the salary band you have approved. Pick the 5 to 8 you will actually reject on.
Minutes 25–50: Convert each skill into observable behavior
For each must-have, ask three questions:
- What does a candidate say or do that shows they have this? Not "they seem confident" — "they explain the trade-off between eventual consistency and strong consistency without prompting."
- What would a candidate say or do that shows they don't? This one is harder and more useful. Interviewers score more reliably when they have a clear negative anchor.
- Which interview stage tests this? If the answer is "the whole loop," the skill is not defined tightly enough.
This is the section where 30 minutes disappears fast. It is also the section that determines whether the rubric is usable.
Minutes 50–70: Assign weights and design the loop
With the skills defined, decide what fails a candidate. If a staff engineer candidate is weak on system design, is that a rejection or a discussable? If they are weak on cross-team communication, same question.
Then map each skill to a stage. A useful test: no stage should evaluate more than three skills, and no skill should be evaluated by more than two stages. If your take-home is trying to evaluate coding quality, system design, testing discipline, and communication, it is evaluating none of them well.
For teams using platforms like HackerEarth Assessments or FaceCode, this is the point to decide which skills get an automated assessment and which need a live evaluator. Automated scoring is more consistent for well-defined coding skills; live evaluation is more useful for judgment, communication, and edge-case reasoning.
Minutes 70–90: Calibrate with a real resume
Pull a resume from a candidate the team has hired in the past 12 months, ideally one everyone agrees was a good hire. Score them against the rubric you just built.
If the rubric would have rejected the person you just agreed was a good hire, the rubric is wrong. Fix it now. If two people at the meeting score the same resume more than one point apart on any skill, the definition for that skill is not tight enough. Fix it now.
Then do the same exercise with a candidate who was hired and did not work out. The rubric should have flagged them.
The three questions that separate rubrics from wish lists
When you find yourself running low on time, these are the three questions that do the most work:
"What behavior would I see?" Cuts through trait language ("smart," "driven," "collaborative") and forces observable definitions.
"Would I reject a candidate for this alone?" Sorts must-haves from nice-to-haves faster than any ranking exercise.
"Where in the loop does this get tested?" Exposes skills the team wants to evaluate but has no mechanism for.
If the hiring manager cannot answer these three for a given skill, the skill does not belong in the rubric yet.
Where intake meetings still fail — and honest trade-offs
Even a well-run intake meeting has limits. Three failure modes we see repeatedly:
Rubric drift after six weeks. The rubric is calibrated once at intake and then never revisited. By the tenth candidate, each interviewer is applying their own drift. The fix is not more training — it is a 15-minute re-calibration meeting after the first three candidates go through the full loop.
The hiring manager wasn't the hiring manager. In matrixed orgs, the person in the intake meeting is not always the person who approves the offer. If the actual decision-maker is a skip-level, get them in the room or accept that the rubric will be relitigated.
The rubric is right and the pipeline is wrong. A tight rubric applied to a weak pipeline produces the same result as a loose rubric applied to a strong one — closed reqs and unhappy hiring managers. Rubric work does not fix sourcing.
A rubric is also not a substitute for judgment on senior hires. For staff-and-above roles, the rubric constrains the debrief; it does not make the decision. That is a feature, not a bug.
Frequently asked questions
How long should a hiring intake meeting actually take?
60 to 90 minutes for a new role. 30 minutes for a backfill on an existing rubric. Meetings under 45 minutes for new roles almost always skip the specificity conversation and produce wish lists. If the hiring manager cannot give you 90 minutes, split the intake into two 45-minute meetings — one for skills, one for weights and calibration.
Who needs to be in the intake meeting besides the recruiter and hiring manager?
At minimum, one senior interviewer who will be on the loop. They pressure-test the rubric in the last 30 minutes and catch skills the hiring manager over- or under-weights. For roles where the hiring manager does not have the deepest technical expertise (common for eng managers hiring specialists), a technical peer is not optional.
How does a rubric differ from a scorecard?
A rubric defines what is being evaluated and what "meets bar" looks like. A scorecard is the form an interviewer fills out during or after the round. The rubric is the source of truth; the scorecard is the artifact. Most teams have scorecards without rubrics, which is why their scorecards do not agree with each other.
What if the hiring manager refuses to cut skills from the must-have list?
Ask them to rank the list and identify the bottom three. Then ask: "If a candidate was strong on the top five and weak on these three, would you reject them?" If the answer is no, those three are nice-to-haves. If the answer is yes, you have a compensation-band problem, not a rubric problem.
Can AI interview tools replace the intake meeting?
No. AI interview tools like HackerEarth's OnScreen apply a rubric consistently across candidates, which is valuable. They do not build the rubric. The intake meeting is where humans decide what to evaluate; the tooling decides how consistently to evaluate it.
Key takeaways
- A usable rubric has 5–8 must-haves with observable behaviors, weights, and stage assignments — not a wish list of traits.
- Block 60–90 minutes for a new-role intake; anything shorter skips the specificity conversation that separates rubrics from wish lists.
- Calibrate the rubric against a real past hire before the first candidate enters the pipeline — if the rubric would have rejected a known good hire, fix it.
- Re-calibrate after the first three candidates go through the loop; rubric drift is the most common post-intake failure.
- Rubrics constrain debriefs but do not replace judgment on senior hires — and no rubric fixes a weak pipeline.
See it in action
Want to see how a structured rubric translates into a repeatable assessment loop? Schedule a demo of HackerEarth Assessments and walk through a rubric-to-assessment mapping with our team.
Discover powerful tools designed to streamline hiring, assess talent efficiently, and run seamless hackathons. Explore HackerEarth’s top products that help businesses innovate and grow.
Assessments
AI-driven advanced coding assessments
Learn More
OnScreen
Interview every candidate. Defend every decision.
Learn More
Hackathons
Engage global developers through innovation
Learn More
L & D
Tailored learning paths for continuous assessments
Learn More