ActDiff-VC: Compressing Video to Almost Nothing by Teaching Diffusion Models to Hallucinate Wisely

Paper: 2605.02849 Authors: Amirhosein Javadi, Shirin Saeedi Bidokhti, Tara Javidi Categories: cs.CV

The Gap

Traditional video codecs (H.264, H.265) hit a wall below 0.1 bpp — they produce blocky, blurry mush. Learned codecs (neural compression) improve perceptual quality but still struggle under extreme rate constraints. Recent diffusion-based approaches can hallucinate plausible content, but they lack a principled way to control what gets hallucinated using minimal transmitted data. The core tension: you need rich conditioning signals to guide generation, but you have almost no bits to spare.

Prior work either transmits dense features (too expensive) or uses fixed keyframe intervals (wastes bits on static scenes, starves dynamic ones). No one has solved: how do you adaptively decide what minimal information to send, and how do you make a diffusion model reconstruct video from that sparse signal alone?

Problem: Ultra-low bitrate video is perceptually broken
   |
   v
Assumption: Diffusion models can hallucinate plausible content
            if given the right sparse conditioning
   |
   v
Method: Adaptive keyframe selection + sparse point trajectories
        -> feed into conditional diffusion decoder
   |
   v
Evidence: 64.6% bitrate reduction at matched perceptual quality
          vs. learned codecs (UVG, MCL-JCV benchmarks)
   |
   v
Conclusion: Content-adaptive conditioning enables practical
            diffusion-based video compression at <0.1 bpp

The Increment

One sentence: Before this paper, diffusion-based video compression required dense conditioning signals that defeated the purpose of compression; after, you can compress video to keyframes + sparse trajectories and still get perceptually plausible reconstruction.

Core Mechanism

ActDiff-VC operates in three stages. First, it partitions the video into variable-length segments based on content complexity — static scenes get longer segments (fewer keyframes), dynamic scenes get shorter ones. Second, for each segment, it selects a keyframe and tracks a small set of salient points across frames, encoding their trajectories. Third, it transmits only the keyframe and trajectory data, then uses a conditional diffusion model to synthesize the missing frames.

The diffusion decoder is conditioned on two signals: the keyframe (provides appearance) and the point trajectories (provide motion). The trajectories are sparse — typically 50-200 points per frame — but they capture the dominant motion patterns. The decoder learns to interpolate plausible content between these sparse constraints, leveraging the diffusion prior’s ability to generate realistic textures and structures.

Two adaptive mechanisms make this work. Content-adaptive keyframe selection monitors scene complexity and inserts keyframes only when the current segment’s motion exceeds a threshold. Budget-aware trajectory selection ranks points by saliency (using optical flow magnitude and spatial distribution) and transmits only the top-k points that fit within the rate budget. Together, these mechanisms allocate bits where they matter most.

Input Video
    |
    v
[Segment Partitioner] ---> variable-length segments
    |                      (adaptive to content)
    v
[Keyframe Selector] -----> transmit keyframe (JPEG/learned)
    |
    v
[Point Tracker] ---------> track salient points
    |
    v
[Trajectory Encoder] ----> compress trajectories (entropy coding)
    |
    v
    ========== TRANSMISSION (ultra-low bitrate) ==========
    |
    v
[Conditional Diffusion Decoder]
    ^       ^
    |       |
keyframe  trajectories
    |
    v
Reconstructed Video

Think of it like animating a cartoon with keyframes and motion paths. A traditional codec is like filming every frame with a camera — you capture everything, but the file is huge. ActDiff-VC is like an animator’s workflow: you draw keyframes (the important poses), sketch motion paths for key features (the trajectories), then let a skilled artist (the diffusion model) fill in the in-between frames. The artist has seen thousands of videos during training, so it knows how fabric wrinkles, how faces move, how shadows fall. You don’t need to transmit every pixel — just enough constraints to guide the artist’s hand. The magic is in choosing which keyframes and which motion paths are essential, and trusting the artist to hallucinate the rest plausibly.

Key Concepts

• Conditional Diffusion Decoder: Standard diffusion models generate images from pure noise by iteratively denoising. Conditional diffusion adds control: you inject extra information (here, keyframes and trajectories) at each denoising step, steering the generation toward content consistent with those constraints. The model learns during training to respect these conditions while still producing perceptually realistic outputs. Think of it as sculpting clay: the conditions are your armature (the skeleton), and the diffusion process fills in the flesh around it. Without conditions, you get random plausible images; with conditions, you get plausible images that match your constraints.

• Sparse Point Trajectories: Instead of transmitting dense optical flow (which costs too many bits), ActDiff-VC tracks a small set of salient points across frames and encodes their (x, y) positions over time. These points are chosen to capture the dominant motion — corners, edges, high-texture regions. The decoder interpolates motion between these sparse samples. It’s like navigating with a few landmarks instead of a detailed map: you know the river bends here, the mountain is there, and you infer the terrain in between. The key insight: human perception is forgiving of interpolation errors in low-saliency regions, so you can get away with extreme sparsity.

• Content-Adaptive Keyframe Selection: Not all video segments need the same keyframe density. A static talking-head shot can go 60 frames without a keyframe; a fast action scene needs one every 10 frames. ActDiff-VC monitors accumulated motion (via optical flow magnitude) and inserts a keyframe when motion exceeds a threshold. This is rate-distortion optimization at the segment level: spend bits on keyframes only when the diffusion model’s hallucination would drift too far from ground truth. It’s like deciding when to recalibrate a compass — if you’re walking straight, you can go a long time; if you’re zigzagging through a forest, you need frequent checks.

Framework Shift

Before (learned codecs):              After (ActDiff-VC):

Video --> [Encoder] --> latent        Video --> [Segment] --> keyframes
             |          features                    |            + 
          [Quantize]       |                   [Track Points]  trajectories
             |             |                        |             |
          [Entropy    <----+                   [Compress]        |
           Coding]                                  |             |
             |                              ========|=============|====
          TRANSMIT                                  v             v
             |                            [Diffusion Decoder]
             v                                 (hallucinate
          [Entropy                           missing frames)
           Decoding]                               |
             |                                     v
          [Dequantize]                      Reconstructed Video
             |
          [Decoder]
             |
             v
       Reconstructed Video

Dense representation,                 Sparse conditioning,
every frame encoded                   generative reconstruction

One sentence: From transmitting compressed representations of every frame to transmitting only sparse control signals and letting a generative model hallucinate the rest.

Expert Assessment

Problem choice: This is a real gap. Ultra-low-bitrate video compression (<0.1 bpp) is practically important (streaming over poor networks, archival storage) but perceptually broken with traditional methods. The insight to use diffusion models as a generative prior is timely — diffusion models have matured enough to be controllable, and the field is ready for this application. The problem sits at the intersection of compression and generative modeling, which is a hot area.

Method maturity: The core idea is elegant: adaptive keyframes + sparse trajectories as conditioning. The two adaptive mechanisms (keyframe selection, trajectory selection) are straightforward but effective. However, the paper doesn’t deeply explore failure modes — what happens when point tracking fails (occlusions, fast motion)? The reliance on optical flow and point tracking is a potential brittleness. A more robust approach might use learned motion representations, but that would cost more bits. The method feels like a solid first step, not a final solution.

Experimental integrity: Baselines are reasonable (H.265, learned codecs like DCVC, diffusion-based methods like LDMVC). The perceptual metrics (NIQE, KID, FID) are appropriate for ultra-low bitrates where pixel-level metrics (PSNR, SSIM) are meaningless. The 64.6% bitrate reduction claim is at matched NIQE, which is fair. However, the paper doesn’t show user studies — perceptual quality is ultimately subjective, and automated metrics can be gamed. The benchmarks (UVG, MCL-JCV) are standard but small. I’d want to see results on more diverse content (sports, animation, low-light) before declaring victory.

Writing quality: The abstract and introduction are clear. The method section is dense but complete. The experiments section could be stronger — the ablation studies are minimal (only one table on keyframe/trajectory selection), and there’s no analysis of where the method fails. The related work section name-drops many papers but doesn’t clearly position the contribution relative to concurrent diffusion-based video compression work. If I were reviewing, I’d ask for: (1) failure case analysis, (2) user study, (3) deeper ablations on trajectory sparsity vs. quality tradeoff.

Verdict: weak accept — The problem is important, the method is novel and shows strong empirical results, but the evaluation could be more thorough and the failure modes need exploration. This is a solid contribution that will likely inspire follow-up work, but it’s not a slam dunk.

Takeaways

Adaptive conditioning for generative models: The idea of dynamically allocating your “conditioning budget” based on content complexity transfers beyond video compression. If you’re using a generative model (diffusion, VAE, GAN) and have limited capacity to condition it (tokens, features, etc.), you can monitor when the model’s prior is sufficient vs. when you need to inject more control. This applies to image editing (where to place control points), text-to-image (which regions need detailed prompts), and even LLM steering (when to provide examples vs. let the model freewheel).

Sparse trajectories as motion representation: Point trajectories are a surprisingly compact way to encode motion. If you’re building any system that needs to transmit or store motion data (animation, robotics, AR/VR), consider tracking sparse salient points instead of dense flow fields. The key is choosing the right points (high-gradient regions, corners) and trusting interpolation for the rest. This is a 10x-100x compression over dense representations.

Perceptual metrics for generative tasks: The paper’s use of NIQE, KID, FID instead of PSNR/SSIM is a reminder that pixel-level metrics are often the wrong objective for generative tasks. If your goal is perceptual quality, optimize for perceptual metrics from the start. This applies to any domain where you’re generating content for human consumption (image synthesis, audio generation, text rendering).