VideoNet: Reviving Action Recognition with Domain-Specific Challenges

Paper: 2605.02834 Authors: Tanush Yadav, Mohammadreza Salehi, Jae Sung Park, Vivek Ramanujan, Hannaneh Hajishirzi, Yejin Choi, Ali Farhadi, Rohun Tripathi, Ranjay Krishna Categories: cs.CV, cs.LG

The Gap

Action recognition used to be the flagship task for video understanding. But modern vision-language models (VLMs) aren’t tested on it anymore because existing benchmarks lack diversity and challenge. The field moved on to other tasks, leaving a blind spot: we don’t know if VLMs can actually recognize actions, especially domain-specific ones like “suturing” in surgery or “pirouette” in ballet. Prior datasets (Kinetics, UCF-101) focus on generic everyday actions. No one has systematically tested whether VLMs understand specialized actions that require domain knowledge.

Problem: VLMs untested on domain-specific actions
    |
    v
Assumption: Generic action datasets insufficient
    |
    v
Method: Build VideoNet (1000 actions, 37 domains)
        + Test VLMs in multiple settings
        + Create 500k training pairs
    |
    v
Evidence: Closed models 69.9%, open models 45%
          Few-shot helps some, hurts others
          Fine-tuning on new data beats all 8B models
    |
    v
Conclusion: VLMs struggle with domain actions
            Training data closes the gap

The Increment

One sentence: Before this paper, we assumed VLMs could handle action recognition because they excel at other video tasks; after, we know they fail at domain-specific actions and need targeted training data.

Core Mechanism

VideoNet is a benchmark and dataset combo. The benchmark side: 1,000 distinct actions spanning 37 domains (medical, sports, cooking, manufacturing, etc.). Each action gets multiple video clips. The evaluation has three difficulty levels: multiple-choice (pick 1 of N actions), binary (is this action X?), and few-shot (here are k examples, now classify). The dataset side: 500k video-question-answer pairs for training, collected by scraping domain-specific videos and generating QA pairs.

The evaluation flow: feed a video to a VLM, ask “what action is happening?”, measure accuracy. In multiple-choice, the model picks from a list. In binary, it answers yes/no. In few-shot, it sees k examples before answering. The training flow: take the 500k pairs, fine-tune a VLM (they use Molmo2-4B), then re-evaluate on the benchmark.

Evaluation Pipeline:
  Video --> VLM --> Answer --> Compare to ground truth
             ^
             |
    [Multiple-choice / Binary / Few-shot prompt]

Training Pipeline:
  500k pairs --> Fine-tune VLM --> Improved model
                      |
                      v
              [Molmo2-4B baseline]

Think of VideoNet as a driving test for VLMs, but instead of parallel parking and lane changes, it’s testing whether the model can distinguish a “double axel” from a “salchow” in figure skating. The benchmark is the test itself—37 different road conditions (domains), 1,000 specific maneuvers (actions). The dataset is the practice material—500k example scenarios with correct answers. Most VLMs have only practiced on generic roads (everyday actions). When you throw them into specialized terrain (surgery, ballet), they crash. The few-shot setting is like giving a quick tutorial before the test: “here’s what a pirouette looks like, now identify one.” Some models learn from the tutorial, others get confused. Fine-tuning is like actual driving school—repeated practice until the maneuvers become second nature.

Key Concepts

• Domain-specific actions: Actions that require specialized knowledge to distinguish. “Walking” is generic—anyone can recognize it. “Suturing” vs “cauterizing” in surgery requires medical context. “Sauté” vs “blanch” in cooking requires culinary knowledge. These actions often look similar to outsiders but have precise meanings within their domain. The challenge: VLMs trained on internet-scale data see mostly generic actions. Domain-specific actions are rare in training, so models haven’t learned the fine distinctions. It’s like asking someone who’s never cooked to tell apart simmering and boiling—they both involve hot water and bubbles, but the temperature and purpose differ.

• Few-shot in-context learning: Giving the model k examples of the target action before asking it to classify a new video. For example: “Here are 3 videos of a ‘pirouette’ [shows videos]. Now, is this new video a pirouette?” The model doesn’t update its weights—it just uses the examples as context. Humans improve +13.6% with examples. Some VLMs improve (Qwen +7.0%), others degrade (Gemini -4.8%). Why the difference? Models that can’t effectively use visual context get distracted by the extra information. It’s like showing someone reference photos before a quiz—helpful if they can compare features, confusing if they just memorize irrelevant details.

• Closed vs open models: Closed models (Gemini, GPT-4) are proprietary, trained on unknown data with unknown compute. Open models (Qwen, LLaVA) have public weights and training details. On VideoNet, the gap is massive: Gemini 69.9% vs Qwen 45.0%. This suggests closed models either saw more domain-specific video data during training or have better video understanding architectures. The gap matters because open models are what researchers can actually improve. The paper’s contribution: showing that targeted training data (the 500k pairs) can close this gap—fine-tuned Molmo2-4B beats all 8B open models.

Framework Shift

Before (mainstream approach):        After (this paper):

Generic action datasets              Domain-specific actions
    |                                     |
    v                                     v
[Walk, Run, Jump, Eat]              [Suture, Pirouette, Sauté]
    |                                     |
    v                                     v
VLMs tested on easy actions         VLMs tested on hard actions
    |                                     |
    v                                     v
High accuracy assumed               Low accuracy revealed
                                          |
                                          v
                                    500k training pairs
                                          |
                                          v
                                    Fine-tuned models improve

From “test on what models already know” to “test on what they should know but don’t, then provide training data to fix it.”

Expert Assessment

Problem choice: Real gap. Action recognition faded from VLM evaluation not because it was solved, but because existing benchmarks became too easy. Domain-specific actions are a natural next frontier—they’re common in real applications (medical video analysis, sports coaching, manufacturing QA) but absent from benchmarks. The timing is right: VLMs are mature enough that we need harder tests.

Method maturity: Straightforward but effective. The benchmark design is solid—37 domains is comprehensive, 1,000 actions is substantial. The evaluation settings (multiple-choice, binary, few-shot) systematically isolate different failure modes. The training dataset is the real contribution: 500k pairs is large enough to matter, and the fine-tuning results prove it works. No architectural novelty, but that’s fine—this is infrastructure work. One concern: the paper doesn’t detail how they ensured action label quality across 37 domains. Domain experts validating labels would strengthen claims.

Experimental integrity: Baselines are fair—they test major closed and open models. The few-shot results are interesting: showing that some models degrade with examples is an honest negative result. The fine-tuning comparison is clean: Molmo2-4B vs other 8B models, apples to apples. One weakness: no ablation on what aspects of the training data matter most (domain diversity? video quality? QA format?). Also, the human baseline (+13.6% with few-shot) is mentioned but not deeply analyzed—why do humans benefit more than models?

Writing quality: The abstract and intro are crisp. The related work section is thin—doesn’t engage deeply with prior action recognition datasets or explain why they’re insufficient. The results section is thorough but could use more error analysis: which domains are hardest? Which actions get confused? The dataset collection section is vague—“scraping domain-specific videos” needs more detail on sources, filtering, and quality control. Rewriting the dataset section with concrete examples and statistics would elevate the paper.

Verdict: weak accept — Solid benchmark and useful training data, but lacks depth in analysis and dataset methodology. The contribution is more “here’s a resource” than “here’s an insight.”

Takeaways

• Evaluation design: The three-setting approach (multiple-choice, binary, few-shot) is a clean way to diagnose model failures. Steal this for any benchmark: start hard, relax constraints, see where models break.
• Domain-specific as a forcing function: When generic benchmarks saturate, go domain-specific. This applies beyond vision—language models, audio models, any modality. Find the specialized niches where models haven’t been tested.
• Training data still matters: In the era of billion-parameter models, targeted datasets (500k pairs) can still move the needle. If your model fails on a task, don’t just scale up—collect focused training data for that task.
• Few-shot as a diagnostic: Models that degrade with in-context examples have a problem with context utilization. Use few-shot performance as a signal for architectural weaknesses, not just a performance boost technique.