The Low Frequency Trap: Why Video Models Can't Keep a Ledger

Paper: 2608.06361 Authors: Sarvesh Baskar, Zikui Cai, Shayan Shabihi, Anirudh Satheesh, Muhammad R. Islam, Udari Madhushani Sehwag, Tom Goldstein, Furong Huang Categories: cs.AI

The Gap

Video benchmarks come in two flavors, and both leave the same hole.

Real-world suites (Video-MME, TempCompass, EgoSchema and friends) give you authentic clips and broad coverage, but every clip is a fixed bundle: the number of events, how fast they come, how long the video runs, and how cluttered the scene is all move together. When a model fails, you cannot say whether it failed because there were fourteen events, because they arrived at 3 Hz, because the video was two minutes long, or because the background was busy. The independent variables are welded together.

Programmatic and synthetic benchmarks (CLEVRER-style causal video QA, procedurally generated temporal tasks) fix the control problem — you can dial one knob at a time. But they inherit the grading convention of QA benchmarks: they check the final answer. If the model says “9 bounces” and the truth is 9, it scores 1.0, whether it saw the right nine bounces, hallucinated four while missing four, or guessed the modal answer. A count is a scalar, and scalars are the easiest thing in the world to get right for the wrong reasons. Compensating errors — one miss cancelling one double-count — are invisible to exact-match accuracy.

So the gap is specifically this: nobody was auditing the itemization. This paper’s move is to make the generator emit, alongside each video, an executable event trace with timestamps, so that the model’s reported events can be lined up against ground truth event by event.

   [PROBLEM]
   Video evals cannot localize temporal failure
      |
      +-- real clips: N, rate F, duration, clutter all entangled
      +-- synthetic clips: controlled, but graded on the final scalar only
      |
      v
   [ASSUMPTION]
   If you own the renderer, you own the ground-truth ledger.
   So: hold rendering fixed, sweep only (N, F), and keep the trace.
      |
      v
   [METHOD]
   Trace-grounded parametric profiling
      +-- 3 tasks: bounce contacts / blinks / state transitions
      +-- 2,190 videos on an (N x F) grid
      +-- 2 scores per video: final count  AND  timestamp-level recall
      |
      v
   [EVIDENCE]
   * capability surface: state transitions OK to N=12 at 0.5-1.0 Hz
   * transient blinks: no reliable region at all, anywhere
   * high N, high F: 0.2% counts correct, 18.1% events recovered
   * 2x frames: 19.6% -> 29.3% accuracy, but 3.7% sequence agreement
      |
      v
   [CONCLUSION]
   Failure is staged, not gradual. Event *representation* gates access;
   count and frequency then compound. And accuracy can rise while
   faithfulness does not -- so accuracy alone is a misleading metric.

The Increment

One sentence: Before, a video model that reported the right number of events was scored as having counted correctly; after, you can show it got the number right while recovering fewer than one in five of the actual events — and you can name the exact (count, frequency, event-type) region where it stops working.

Core Mechanism

The machinery is deliberately unglamorous. A parametric generator renders three families of videos. In bounce, a ball travels and makes wall contacts; the event is an impulse — an instant with no aftermath. In blink, an element turns on and off; the event is transient, present for a short window and then gone without trace. In state transition, an object changes category and stays changed; the event leaves a standing residue that any later frame still testifies to. Two knobs are swept: N, the number of events, and F, the frequency at which they occur. Everything else — resolution, palette, geometry, motion style — is pinned, so the only thing varying between two cells of the grid is event count and event rate. That yields 2,190 videos on an (N x F) lattice.

The second half is the actual contribution. Because the generator scripted the events, it can emit them as an executable trace: a list of (event, timestamp) pairs that is ground truth by construction rather than by annotation. The model is asked not just for a total but for the events it saw, and each report is scored twice. The scalar score is the old exact-count accuracy. The trace score aligns reported timestamps against the true trace within a tolerance and reports recall, precision, and whether the whole reported sequence agrees with the ledger. Sweeping this over the grid produces a capability surface: for each event type, the contour in (N, F) space inside which the model clears an 80% reliability bar.

What falls out is a staged picture rather than a smooth decay. Persistent state transitions are handled cleanly out to twelve events at 0.5 and 1.0 Hz. Transient blinks have no reliable positive-count region anywhere on the grid — the model never gets a foothold, regardless of how few events there are. In the high-N high-F corner, 0.2% of totals are correct and 18.1% of true events are recovered. Then the diagnostic pays off: doubling the frame budget lifts bounce accuracy from 19.6% to 29.3%, which looks like progress until you check the ledger and find that the reported sequence matches ground truth only 3.7% of the time. The extra frames bought better guesses, not better bookkeeping.

  PARAMETRIC GENERATOR (rendering held fixed)
  +--------------------------------------------------+
  |  sweep N = event count     sweep F = event rate  |
  |  task = bounce | blink | state-transition        |
  +--------------------------------------------------+
        |                                  |
        v                                  v
   [ VIDEO FRAMES ]                 [ EVENT TRACE ]
        |                            (e_1, t_1)
        |                            (e_2, t_2)  ... executable
        v                                  |
   [ VLM + prompt ]                        |
        |                                  |
        v                                  |
   report: total = k                       |
           events = (e', t'), ...          |
        |                                  |
        +---------------+------------------+
                        |
             +----------+-----------+
             |                      |
             v                      v
      [ SCALAR SCORE ]      [ TRACE ALIGNMENT ]
      exact count match     match t' to t within tol
      (the old metric)      -> recall / precision
             |              -> sequence agreement
             +----------+-----------+
                        v
              [ CAPABILITY SURFACE ]
        F ^  . . . x x x        F ^  x x x x x x
          |  . . . x x x          |  x x x x x x
          |  o o o o o o          |  x x x x x x
          |  o o o o o o          |  x x x x x x
          +--------------> N      +--------------> N
          state transitions        transient blinks
        o = >= 80% reliable   x = below   . = mixed

The metaphor: this is a forensic audit, and the auditors wrote the company’s books themselves.

Normally when you audit an accountant you compare their reported total against your own total. If both say $4,000, you move on. The trick here is that the researchers *are* the company: they scripted every transaction, so they hold the general ledger with a timestamp on every line. Now they can tie out line by line. And the accountant who reports$4,000 turns out to have missed three payments and invented three others — the bottom line reconciles, the ledger does not. That is exactly the 0.2%-correct-but-18.1%-recall situation, and exactly why 29.3% accuracy with 3.7% sequence agreement should worry you: a bookkeeper whose totals improved while their line items stayed wrong got luckier, not better.

The three event types are three kinds of transaction. A state transition is a permanent change to an account balance: even if you only glance at the books once a month, the new balance is sitting there, so you can reconstruct that something happened. A blink is a same-day deposit-and-withdrawal that nets to zero — unless you happen to open the ledger during those few hours, the event leaves no trace in any snapshot. A bounce is a single instantaneous transfer, somewhere in between. This is why the failure is staged rather than gradual: whether the auditor can see anything at all is decided by transaction type before volume even matters, and only then does volume (N) and velocity (F) grind down whatever accuracy remained.

The (N, F) sweep is auditing companies of increasing transaction volume and velocity to find the largest books this accountant can still be trusted with. The 80% contour is where you draw the line on the engagement letter. And raising the frame budget is sampling more days out of the month: it raises the odds your total comes out right, without making you any better at knowing which transactions actually happened.

Key Concepts

• Trace-grounded evaluation: Most benchmarks grade a compressed answer — a number, a letter, a yes/no. Compression destroys evidence about *how the answer was produced. Suppose you ask someone to count the cars that pass a window in a minute and they say “seven.” Right or wrong, you learn almost nothing. Now ask them to write down the time of each car. If the truth was 7 cars at seconds 3, 11, 19, 28, 40, 47, 55 and they wrote 7 cars at seconds 5, 12, 20, 21, 22, 48, 56, you can see that they hallucinated a cluster around second 21 and missed the one at 40 — and that their “correct” total was two errors cancelling. Trace grounding is just: make the ground truth itemized and executable, and require the model’s answer to be itemized too, so cancellation cannot hide.

• Capability surface and the reliability threshold: A single benchmark number says “this model is 62% good at video counting,” which is a statement about the benchmark’s difficulty mix, not about the model. Instead, put the two things that actually make the task hard on two axes — how many events (N) and how fast (F) — run the model at every cell, and shade the region where it clears some bar you’d actually accept in production, here 80%. What you get is shaped like a flight envelope: an operating region with a boundary. “Reliable up to 12 events at 1 Hz, unreliable above” is a usable engineering fact. “62%” is not.

• Persistent vs transient events (and why representation gates access): A frame-sampled model does not watch video; it looks at a handful of stills. That makes some events structurally recoverable and others structurally invisible. If a traffic light changes from red to green and stays green, any later still tells you the change happened — the event is written into the state. If the light flickers green for 200 ms and returns to red, no still outside that 200 ms window contains any evidence at all; you would need to have sampled inside the flicker. This is the paper’s sharpest point: before you talk about temporal reasoning ability, ask whether the evidence is even in the frames the model received. Transient events at frequency F sampled at rate R are simply gone when R is too low, no matter how good the reasoning is — and the paper’s finding that blinks have *no reliable region anywhere is consistent with that being a hard access failure, not a reasoning failure.

Framework Shift

Before (mainstream approach):          After (this paper):

  fixed real clip                        parametric generator
  +-------------------+                  +-------------------+
  | N, F, duration,   |                  |  sweep N  x  F    |
  | clutter: entangled|                  |  render: FIXED    |
  +-------------------+                  +-------------------+
          |                                   |          |
          v                                v          v
      [ model ]                        [ frames ]  [ trace ]
          |                                   |          |
          v                                v          |
     answer = k                        report = k, (t'..)|
          |                                   |          |
          v                                +----+-----+
     k == truth ?                                  |
          |                                +-------+-------+
          v                                |               |
     ACCURACY: 62%                         v               v
     (one scalar,                    count match     line-item
      no localization)                                tie-out
                                           |               |
                                           +-------+-------+
                                                   v
                                       F ^ o o o x x x
                                         | o o x x x x
                                         +------------> N
                                       envelope + faithfulness

From one aggregate number to an operating envelope with an audit trail, the core shift is that evaluation stops asking how good is this model and starts asking where exactly does it break, and is it right for the right reasons.

Expert Assessment

Problem choice: Real gap, and a well-chosen one. The “correct answer, wrong reasoning” problem is widely acknowledged in text (chain-of-thought faithfulness, process reward models) but has barely been operationalized for video, where the compressed-answer convention is even more entrenched. Counting is the ideal wedge: it is the simplest possible temporal aggregation, so failure here is genuinely damning, and it happens to be the one task where the ground truth is naturally itemizable. The framing also lands on something the field keeps stumbling over — that frame sampling silently defines what is knowable — and gives it a measurement apparatus instead of a hand-wave. This sits squarely in the diagnostic-benchmark turn (BIG-Bench Hard to targeted probes, aggregate leaderboards to capability surfaces), which is where video eval was overdue to go.

Method maturity: Clever in framing, light in machinery. There is no new model, no new algorithm — it is a generator, a sweep, and an alignment scorer. That is fine; the insight is that owning the generator gets you a free ledger, and the value is in having actually built it and run the grid. But it *is thin, and the thinness shows in one place: the timestamp alignment tolerance is a hyperparameter that directly manufactures the headline numbers. If “3.7% sequence agreement” is sensitive to whether tolerance is 0.2 s or 1.0 s, that number is soft, and I would want the sensitivity curve, not a footnote. A simpler alternative the paper seems to skip: multiset agreement on event identities without timestamps, which is tolerance-free and would still expose compensating errors. Reporting both would make the claim much harder to argue with.

Experimental integrity: The internal comparisons are clean — rendering held fixed is the right control, and the accuracy-up-faithfulness-flat result is the kind of finding that only shows up if you were not fishing for it. Two red flags. First, the headline rests on a single model (Gemini 3.6 Flash, a cheap fast-tier model); the abstract mentions prompting variants and real-world checks but not a spread of models, and “fast-tier model is bad at dense temporal detail” is a much weaker claim than “video LMs fail.” Second, and more serious: there is no oracle or information-availability baseline. If the model receives K frames and the video contains N events at frequency F, there is a hard ceiling on what any system could recover, and for the high-N high-F corner that ceiling may be near the observed performance. Without computing it — or running a trivial frame-differencing CV pipeline on the same sampled frames — you cannot separate “did not reason” from “was never shown.” The paper gestures at this by raising the sampling rate, which is the right instinct, but a measured ceiling would convert a suggestive result into a proof. The 18.1% recall figure in particular is uninterpretable until I know the ceiling at that grid cell.

Writing quality: The abstract is doing heavy nominalization work — “trace-grounded parametric profiling for capability-surface estimation” is four abstractions stacked on an idea a smart reader gets in one sentence (“we ship the answer key with timestamps”). That kind of naming tends to signal padding out a light contribution. “Staged temporal failure” is the one coinage that earns its keep. The section I would rewrite is whatever handles sampling: it needs an explicit information-availability analysis with a Nyquist-style argument for the transient tasks and an oracle bound per grid cell. Done well, that single addition upgrades the paper from “models do badly on our probe” to “here is the boundary between what is unknowable given frame budget and what is a genuine reasoning failure” — which is a substantially more citable result and probably the real paper hiding inside this one.

Verdict: weak accept — the trace-grounded audit and the accuracy-rises-while-faithfulness-does-not result are genuinely useful and will change how people report counting evals, but the single-model evidence base and the missing information-theoretic ceiling leave the headline claims less pinned down than the framing implies.

Takeaways

Things you can lift directly:

Never report exact-match accuracy on an aggregation task alone. Any metric that grades a scalar derived from many sub-decisions can be satisfied by compensating errors. Add a set-level or sequence-level agreement metric next to it. This transfers immediately to counting, tallying, deduplication, log summarization, “how many times did X occur in this document,” financial extraction — anywhere the answer is a reduction over items. The specific diagnostic signature to watch for: accuracy improves, item-level agreement does not. That means you bought better guessing.

When you build a synthetic eval, make the generator emit the answer key in itemized, executable form. It costs nearly nothing at generation time and is impossible to retrofit later. The general pattern: build the eval so that ground truth is a *program trace, not a label. Then any intermediate claim the model makes is checkable.

Separate event representation from event count in your own task analysis. Before blaming a model’s reasoning, classify your events: does each one leave a persisting residue in state, or does it vanish? Persisting events are recoverable from sparse snapshots; transient events require sampling inside their window. If your production task involves transient events, the fix is not a better prompt or a bigger model — it is sampling policy, and you should compute the sampling rate your event frequency demands before anything else. The blink-versus-state-transition result says this distinction dominates everything downstream.

Replace single-number evals with a two-axis reliability envelope. Pick the two parameters that actually drive difficulty in your domain, sweep them, and shade where the system clears the bar you would actually ship at. “Reliable to 12 events at 1 Hz” is something you can design a system around; “62% on the benchmark” is not. This is cheap to do and disproportionately useful for capability-based routing and for deciding when to hand a subtask to code instead of a model.

Architectural implication: if your application needs event bookkeeping, do not ask a video LM for a total. Have it detect within short windows where evidence is dense, and aggregate in code. The paper is essentially a measurement of why the monolithic-count approach fails, and the failure is not the kind that scales away with prompting.