A few days ago I ran a mock interview for a friend interviewing for a Data Scientist – Product role at a food delivery company. It was my first time running one, so this isn’t a how-to. Just a few things I noticed while thinking it through.
They tend to come in two parts:
explore the problem with data
measure a change the team wants to launch
Two things I’d pin down first: is the problem real, and what’s the business goal?
For whether it’s real, I’d list the key metrics, then say how they’d look if the problem were real, and how they’d look if it weren’t.
In our case, a few merchants complained they were getting more orders than they could handle. I’d check whether prep time and carrier wait time climb as order volume goes up. If the problem is real, I’d expect something like: past 20 orders, those times jump sharply, and carriers end up waiting around at the merchant instead of grabbing and going. If it isn’t, the times still rise, but not enough to really hold up deliveries.
For the business goal, I’d ask the interviewer what they have in mind, and right away offer a few guesses of my own.
In our case, the short-term goal is to avoid long wait times and cancellations. Longer term, we want to keep the marketplace healthy: if wait times stay long, carriers may start leaving the platform, and if orders keep overwhelming merchants, they may unlist themselves. Both are things we want to avoid.
The interviewer would pick a change to launch. They usually ask what’s your success metric?
Some cases let you go straight to an A/B test. Some don’t. When there’s no experiment telling you which metric is correct, you have to define one from scratch and argue why it’s reasonable.
If a few seem about equally good, I’d lean toward the one without an obvious flaw.
In the merchant-complaint case, the platform decides to launch a “pause” button. Merchants can pause when they’re getting too many orders, or whenever they want. This turns out to be a case where the experiment’s target metric is hard to choose. When a merchant pauses, order volume drops as a result, and prep time and wait time are all tied to order volume. There are long-term metrics we could track, but each has an obvious flaw:
merchant retention — it means gambling the platform’s revenue just to test a button
order error rate or ratings — errors are rare and ratings are noisy, so it takes forever to reach significance
long-term order volume per merchant — the button lowers volume by design, so a drop could mean it’s working, or that merchants are hurting; you can’t read the direction
There’s no clean experiment to define here, so we have to find a success metric without an experiment.
I’d measure it in two layers.
First, adoption. Before asking whether the button worked, I’d ask whether merchants use it at all, and whether they use it at the right time. How many overwhelmed merchants actually pause, and do the pauses cluster at high-volume moments? This is measurable, and it isn’t polluted by the volume problem, because it’s about behavior, not outcomes. If nobody uses it, or they only pause when they’re idle, nothing downstream is worth reading.
Then impact, but only among merchants who actually paused. I’d go back to the Part 1 signature: when orders pile up, do wait time and cancellations still spike, or does the merchant pause before that happens? It’s an observation, not an experiment, since these are merchants who chose to use the button, not a random group. But we don’t have an experiment, so this is the closest we can get to knowing whether the problem improved.
None of this needs a perfect answer. I think the interviewer mostly wants to see that you know what a good answer would even look like.