Long-Form Video Understanding - Part 2: Evaluation and Benchmarks

12 min read Original article ↗

In my previous writeup, I demystified Multimodal LLM-as-a-Judge; in Part 1 on long-form video understanding, I discussed current bottlenecks and design choices. This post finishes the small trilogy, diving into evaluation and benchmarks, and tying the other two together.

Before we start, let me define benchmarks and evaluation:

  • A benchmark is typically two things: what you are measuring - a set of tasks; and how you measure - a way to score answers against those tasks. It’s just one component of evaluation.

  • Evaluation is the harder question of whether that score is actually useful to you. A benchmark can mislead on either side:

    • Tasks may not require the capability you care about - e.g. answerable from one frame or the subtitles alone, when your use case needs real long-form understanding.

    • Scoring - e.g. LLM judges for open-ended answers - may not be calibrated to your use case.

The rest of this writeup dissects both:

  • Benchmark survey: I categorize benchmarks by task and scoring complexity, and point out clear gaps and the reasons behind them.

  • Evaluation and Agentic Judge: there are clear gaps in open-ended, multi-hop questions - which is where a judge becomes essential. I connect back to my previous writeup.

  • Judge for training: I close with the training-time uses of a judge - for example, the agentic methods from Part 1.

Two different kinds of property matter here:

  • Task axes - how hard the answer is to produce: where the evidence sits (scope), and how many reasoning steps it takes (depth).

  • Scoring axes - how hard the answer is to verify: whether it’s closed or open-ended, and whether it’s objective or subjective.

And, as we’ll see, the two can be quite correlated.

The earlier survey by Tang et al. slices video tasks by granularity - abstract (e.g., video classification) vs. temporal (e.g., moment retrieval) vs. spatiotemporal (e.g., object tracking), along with varying degrees of language involvement. At a high level, this captures where the relevant evidence sits (scope). It’s a useful axis, but on its own it can mislead: where the evidence sits doesn’t tell you how hard the task is.

I think there’s a second, equally important dimension: how much planning and decomposition a task takes (depth) - i.e., how many steps the model has to chain together. Take two questions that both need the whole video (same scope), and both expect closed, objective answers:

  • “Does anyone in a red coat ever appear?” - a presence check: find one match and stop - shallow.

  • “How many distinct people appear, counting each only once?” - scan the whole video, re-identify, dedup, and tally. Several dependent steps - deep.

So I propose two task axes:

  • Evidence scope - where the relevant evidence sits: single frame → single clip → dispersed segments → whole video.

  • Compositional depth - how many steps the model must chain:

    • L0 - Perceive: the relevant moment is specified; just recognize what’s there.

    • L1 - Locate / retrieve: the target is specified; find whether and where it occurs.

    • L2 - Single-hop reason: find the relevant evidence and reason once to an answer.

    • L3 - Multi-hop / decompose: break into interdependent sub-questions, chain retrieve-then-reason steps, and aggregate.

To show that scoring is a separate dimension, take two questions of similar task difficulty:

  • “How many distinct people appear in the whole video?” - wide scope and multi-hop, but the answer is a number you can check automatically.

  • “Who are the distinct people, and what distinguishes each?” - similar underlying work, but now the answer is free-form text with no exact match; you need a mechanism to read and judge it.

So I propose two scoring axes:

  • Closed vs. open - can you auto-score it (exact match, tIoU), or do you need a model to read the answer (a judge)?

  • Objective vs. subjective - is there a single defensible answer, or legitimate variation? (”Is the cinematography good?” - even a perfect judge can’t guarantee agreement without more objective standards.)

It's worth noting that these axes together trace the line that separates verifiable from non-verifiable rewards in reinforcement learning.

I categorize the major long-video benchmarks along all four axes below: scope and depth are a center-of-mass range (most benchmarks span several rungs); format and ground truth reflect each benchmark's primary scoring mode. (The full list of the 16 benchmarks is attached at the end)

A few patterns worth highlighting:

  • Almost every benchmark is closed and objective. Only MLVU and InfiniBench mix in open-ended questions - and those open slices are also the more subjective ones - while LVSum is the one recent benchmark that is squarely open-ended and subjective throughout.

  • Most benchmarks live at locate-and-retrieve (L1) or single-hop reasoning (L2). Genuine L3 - multi-hop decomposition - shows up only as subsets: the multi-detail split of MLVU, the event-linking and spoiler questions in InfiniBench, the adaptation track of Video-MMMU, route planning in VSI-Bench, forward-active responding in OVO-Bench.

  • Long is not deep. The longest benchmarks are still mostly L1-L2 - retrieve-and-answer over a long video, not decompose-and-reason: LVBench at ~68 min average, InfiniBench at ~53 min.

  • The benchmarks that aim deep often don’t test it. EgoSchema, Video-MMMU’s adaptation track, and OVO-Bench’s forward-active responding all target multi-step reasoning, but force it into multiple choice - where a model can pattern-match or eliminate its way to the answer without actual step-by-step decomposition.

Here is the evidence that the task and scoring axes are closely related: the only benchmarks that climb to L3 and whole-video scope are exactly the ones forced open-ended and subjective (LVSum, the open slices of MLVU and InfiniBench) - and there are barely any of them.

That isn’t a coincidence. Current benchmarks live in the closed + objective + low-depth corner, even as video length scales up, and the two halves reinforce each other. The closed + objective bias likely reflects a desire to grade cheaply, with no Multimodal LLM-as-a-Judge required. And that format, in turn, limits the depth you can actually test - a genuinely multi-hop question posed as multiple choice invites exactly the shortcutting above, and a deep, holistic task rarely has one objective answer. So the tasks that genuinely need long-form understanding - deep, whole-video, open-ended - are also the ones we can’t grade with a cheap exact-match check. Which is exactly where a judge comes in.

The benchmark gap above - open-ended, multi-hop questions - is exactly where auto-scoring runs out and you need a model to read the answer and judge it. This is where I’ll pick up the thread from my previous writeup on Multimodal LLM-as-a-Judge.

For multi-hop tasks specifically, a judge unlocks something a benchmark score can’t: process evaluation. Instead of only checking the final answer, it can verify the reasoning trajectory - did the model actually find the right evidence at each step, or did it guess its way to a plausible conclusion? In the judge writeup I called this agentic step verification.

That points to a design I find compelling for L3 questions: an agentic judge. A main judge decomposes the evaluation the way the multi-hop task decomposes - spawning smaller, specialized judges for each step and aggregating their verdicts. The decomposition that makes producing the answer hard is the same move that makes grading it tractable; it’s the natural extension of the rubric- and checklist-style judging I discussed before - breaking one fuzzy judgment into many checkable pieces.

But here’s the part I find most interesting, and it’s not just interpretability: decomposing the evaluation redistributes where the hard perceptual work lands. Take a genuinely L3 question over a two-hour film - “Why does the protagonist turn on their mentor in the final act?” - and say the model answers, “because the mentor’s betrayal, revealed at 1:12, recontextualizes the debt from act one.” A single judge would have to re-watch the whole film to grade that. An agentic judge instead asks a few different kinds of question:

  • Localization: do the cited scenes exist and show what’s claimed? - a couple of checks over short, pinpointed spans.

  • Grounding: does the 1:12 clip actually reveal the betrayal? - again, local.

  • Entailment: given those verified facts, does the “recontextualizes the debt” link actually follow? - pure text and logic, no video at all.

  • Completeness: did the answer miss another motive set up somewhere else in the film? - this one still needs to scan the whole thing.

Notice what happened: most of the grading collapsed into text-only logic checks - where a judge is very reliable - and short-span perceptual checks, where it’s far stronger than on long video. For all of those, the judge never has to take in the whole two hours. The long-video burden didn’t disappear, though - it got concentrated into that last check, which is where the real difficulty now lives.

And that difficulty is recall, not grounding. The localization and grounding checks only validate the evidence the model chose to cite - a model that points at the wrong scene can still pass every local check while having missed the real evidence. So the irreducible core is finding the evidence the answer should have used but didn’t, which means searching the whole video again. This is exactly where “agentic” differs from “checklist”: a checklist is static, but an agentic judge can retrieve spans on demand and actively hunt for that missed evidence - the same decompose-retrieve-aggregate loop the Part 1 agents use to answer, turned around to verify.

Two caveats, even so:

  • Every sub-judge needs its own calibration, and the errors compound up the tree - you’ve traded one shaky judgment for many.

  • If the judge runs on the same kind of model as the policy it grades, the two share a perceptual blind spot - the judge will quietly miss exactly the visual mistakes the model is most prone to.

So decomposing the evaluation narrows the blind spot to its hard core - whole-video recall, and the perception the judge shares with the model - but it doesn’t remove it.

So far I’ve treated the judge as a measuring instrument. But as I argued in the judge writeup, that’s the smaller half of its value - the more consequential use is at training time, and that’s where evaluation loops directly back into the design choices from Part 1.

Recall that Part 1 ended with trained agents: a model that decomposes a query, calls tools, and is optimized with RL or preference optimization. RL needs a reward, and for anything open-ended that reward is increasingly a judge.

This is where the agentic judge does something a final-answer judge can’t. A scalar “this answer is a 6” tells the model it was wrong but not where; per-step verdicts - localization fine, grounding fine, the step-2-to-3 link unsupported - tell it which step to fix. That’s the credit-assignment problem for multi-hop RL, and it’s exactly the kind of dense process reward the agentic methods in Part 1 are reaching for.

But the second caveat above turns into a real failure mode here. A process reward is only as honest as the judge behind it - and if that judge shares the policy’s perceptual blind spot, you’re rewarding the model for answers that look right to a grader that can’t see what’s wrong. The key unlock would be to de-correlate the judge’s perception gaps from the generator’s.

For current long-video benchmarks, length is no longer the bottleneck - we can already build two-hour benchmarks. The bottleneck is that the tasks we care about most - deep, whole-video, open-ended - and the only tool that could measure them at scale - a judge - are stuck in the same blind spot as articulated above.

That’s the connection I wanted to highlight across this trilogy:

  • Part 1 built increasingly sophisticated machinery for deep, long-context, open-ended problems.

  • The benchmarks surveyed in this post barely cover those.

  • And the judge that could fill the gap doesn't exist yet - the hard part is keeping it from being weakest in that very spot.

So perhaps the real frontier in long-form video understanding isn’t a longer benchmark or a bigger model - it’s an evaluation we can trust on the tasks we can’t yet measure. A well-calibrated, possibly multi-hop, genuinely multimodal judge is the most promising path I can see - but it’s far from solved.

General long-video QA

Movies / narrative

Knowledge-intensive

Spatial

Needle-in-a-haystack

Instance-level grounding

Timestamp-aware summarization

Streaming

Discussion about this post

Ready for more?