CoPE-VideoLM: Codec Primitives For Efficient Video Language Models

Paper: 2602.13191 Authors: Sayan Deb Sarkar, Rémi Pautrat, Ondrej Miksik, Marc Pollefeys, Iro Armeni, Mahdi Rad, Mihai Dusmanu Categories: cs.CV, cs.AI, cs.CL

Abstract

Video Language Models (VideoLMs) face a fundamental challenge: balancing temporal coverage with computational efficiency. Current approaches rely on keyframe sampling to fit within context window constraints, but this sparse sampling misses critical macro-level events and micro-level details. CoPE-VideoLM introduces a paradigm shift by leveraging video codec primitives—specifically motion vectors and residuals—that naturally encode video redundancy and sparsity. This approach eliminates the need for expensive full-image encoding on most frames, achieving up to 86% reduction in time-to-first-token and 93% reduction in token usage while maintaining or exceeding performance across 14 diverse video understanding benchmarks.

Key Contributions

• Introduction of codec primitives (motion vectors and residuals) as efficient video representations for VideoLMs, exploiting native video compression structures
• Lightweight transformer-based encoders that aggregate codec primitives and align them with image encoder embeddings through strategic pre-training
• Dramatic efficiency gains: 86% reduction in time-to-first-token and 93% reduction in token usage compared to standard VideoLMs
• Flexible density control mechanism allowing trade-offs between keyframe and codec primitive densities for different use cases
• Comprehensive evaluation across 14 benchmarks spanning general QA, temporal reasoning, long-form understanding, and spatial scene understanding

Technical Methodology

The core innovation of CoPE-VideoLM lies in its exploitation of video codec structures. Modern video codecs (H.264, H.265) compress videos by encoding keyframes (I-frames) fully and representing subsequent frames (P-frames, B-frames) through motion vectors and residuals. Motion vectors capture pixel displacement between frames, while residuals encode the prediction error. These primitives inherently capture temporal dynamics with minimal redundancy.

The architecture consists of three main components:

1. Keyframe Encoder: Processes selected I-frames using standard vision transformers to extract spatial features
2. Motion Vector Encoder: A lightweight transformer that processes motion vector fields, capturing inter-frame motion patterns
3. Residual Encoder: Another lightweight transformer that processes residual information, capturing fine-grained changes not explained by motion

The pre-training strategy aligns codec primitive representations with image encoder embeddings, enabling the model to learn meaningful correspondences between compressed representations and full image features. This alignment accelerates convergence during end-to-end fine-tuning on downstream tasks.

The model’s flexibility comes from adjustable sampling rates: users can vary keyframe density (how many I-frames to process) and codec primitive density (how many P/B-frames to include) based on computational budgets and task requirements.

Performance Analysis

CoPE-VideoLM demonstrates remarkable efficiency-performance trade-offs across diverse benchmarks:

Efficiency Metrics:

• Time-to-first-token reduced by up to 86%, enabling faster inference and better user experience
• Token usage reduced by up to 93%, allowing longer videos to fit within context windows
• Computational overhead dramatically decreased by avoiding full-image encoding for most frames

Benchmark Performance: The model maintains or exceeds baseline performance across 14 benchmarks:

• General video QA tasks show comparable or superior accuracy
• Temporal reasoning tasks benefit from denser temporal coverage enabled by efficient codec primitives
• Long-form video understanding improves due to ability to process more frames within token budgets
• Spatial scene understanding remains strong despite reduced full-image processing

The results suggest that codec primitives capture sufficient information for video understanding tasks, and the sparse keyframe sampling of traditional VideoLMs may be unnecessarily wasteful. The motion vectors effectively encode “what moved where,” while residuals capture appearance changes, together providing a compact yet information-rich representation.

Implications and Future Directions

CoPE-VideoLM opens several promising research directions:

Practical Impact: The dramatic efficiency gains make video understanding more accessible, enabling deployment on resource-constrained devices and processing of longer videos. The 93% token reduction is particularly significant for applications requiring extensive temporal context.

Architectural Insights: The success of codec primitives challenges the assumption that full-image processing is necessary for every frame. This suggests that video understanding models should be designed with video-native representations rather than treating videos as sequences of independent images.

Scalability: The approach naturally scales to longer videos and higher resolutions, as codec primitives remain compact regardless of video length. This contrasts with keyframe-based methods where token count grows linearly with video duration.

Limitations and Extensions: While codec primitives are efficient, they depend on video codec availability. Future work could explore:

• Learned codec-like representations for raw video
• Adaptive primitive density based on video content complexity
• Integration with other compression techniques like neural codecs
• Extension to multi-modal scenarios with audio codec primitives

The work also raises questions about optimal pre-training strategies for aligning compressed and uncompressed representations, and whether similar principles apply to other temporal modalities.

Takeaways

1. Video codec primitives (motion vectors and residuals) provide an efficient alternative to full-frame processing in VideoLMs, reducing token usage by up to 93%
2. Lightweight transformer encoders can effectively aggregate codec primitives and align them with image embeddings through strategic pre-training
3. The approach maintains or exceeds performance across 14 diverse benchmarks while dramatically improving computational efficiency
4. Flexible density control allows practitioners to trade off between efficiency and performance based on specific application requirements
5. The success of codec primitives suggests that video understanding models should embrace video-native representations rather than treating videos as image sequences
6. Time-to-first-token reduction of 86% enables more responsive video understanding applications and better user experiences
7. The paradigm shift toward compression-aware architectures may inspire similar approaches in other temporal and multi-modal domains