LongE2V: Using Video Diffusion Priors to Reconstruct, Predict, and Interpolate from Sparse Event Streams

Paper: 2607.08770 Authors: Cheng-De Fan, Chun-Wei Tuan Mu, Chen-Wei Chang, Chin-Yang Lin, Kun-Ru Wu, Yu-Chee Tseng, Yu-Lun Liu Categories: cs.CV

The Gap

Event cameras don’t capture frames — they stream asynchronous brightness changes (events) at microsecond resolution. Converting this sparse, noisy stream back into conventional video is genuinely hard. Prior approaches split into two camps, each with a fatal flaw:

• Regression methods (e.g., E2VID, FireNet) minimize pixel-wise error, producing blurry textures. They’re fast but lifeless — the “uncanny valley” of event-to-video.
• Generative methods (e.g., diffusion-based E2V) produce sharp frames but accumulate temporal drift over long sequences. Generate 100 frames and the scene quietly morphs into something unrecognizable.

Nobody has cleanly solved all three sub-tasks — reconstruction, prediction, and interpolation — within a single framework, especially not at long horizons. The gap is a method that has the perceptual quality of generative models, the temporal stability of regression models, and the generality to handle multiple tasks.

Sparse Event Stream
        |
        v
+------------------+
| Problem:         |
| Events -> Video  |
| (reconstruction) |
| + prediction     |
| + interpolation  |
+--------+---------+
         |
         v
+-------------------+        +---------------------+
| Regression        |        | Generative (Diff.)   |
| -> blurry         |        -> temporally unstable  |
+-------------------+        +---------------------+
         |                           |
         v                           v
+-------------------------------+
| Assumption:                   |
| Pre-trained video diffusion   |
| model already encodes rich    |
| visual priors. Fine-tune it   |
| with event conditions.        |
+---------------+---------------+
                |
                v
+-------------------------------+
| Method: LongE2V               |
| - Fine-tune video diffusion   |
| - Autoregressive unrolling    |
| - Adaptive context switching  |
| - Event voxel density aug.    |
+---------------+---------------+
                |
                v
+-------------------------------+
| Evidence:                     |
| Outperforms SOTA on all      |
| 3 tasks, real-world data,    |
| long-horizon stability,       |
| zero-shot generalization      |
+---------------+---------------+
                |
                v
Conclusion: Diffusion priors
can unify event-based video
tasks at scale.

The Increment

One sentence: Before this paper, event-to-video methods either looked good but fell apart over time, or stayed stable but looked blurry — now there’s a single diffusion-based framework that does all three tasks (reconstruction, prediction, interpolation) with both quality and long-horizon stability.

Core Mechanism

LongE2V starts from a pre-trained text-to-video diffusion model (think something like a Video LDM) and replaces the text conditioning with event voxel conditioning. The model is fine-tuned to “read” event streams as a kind of motion blueprint and hallucinate plausible RGB frames that respect that blueprint. Because the diffusion prior already knows what real video looks like, the fine-tuning needs relatively little data — the model transfers visual knowledge from its pre-training.

For long sequences, they process video autoregressively: generate a chunk of frames, then use the last few frames as context for the next chunk. The problem is that errors compound — each chunk can drift slightly, and after 10 chunks you’re on Mars. Their fix is Adaptive Context Switching: instead of always conditioning on the last N generated frames, they dynamically switch between local context (recent frames for motion continuity) and global context (earlier reference frames for appearance anchoring) based on how much drift has accumulated. Think of it as a sailor who mostly watches the waves ahead but periodically checks the compass.

For frame interpolation specifically, they need bidirectional consistency — the interpolated frame should look right whether you approach it from the past or the future. They achieve this through Reencoding Alignment with Cross Residual Correction: encode the frame from both directions, then correct each encoding by borrowing residual information from the other direction. This ensures the two paths converge rather than fighting each other.

Event Stream
     |
     v
[Event Encoder] --> event features
     |
     v
+----------------------------+
| Pre-trained Video Diffusion|
| (fine-tuned)               |
|                            |
| event features ---->       |
|                    UNet    |
| noise ---------> /DiT ---->| denoised frames
|                            |
+----------------------------+
     |
     v (autoregressive)
     |
     v
+----------------------------+
| Context Manager            |
|                            |
| local_ctx = last N frames  |
| global_ctx = ref. frames   |
|                            |
| if drift > threshold:      |
|   use global_ctx           |
| else:                      |
|   use local_ctx            |
+----------------------------+
     |
     v
[Output: reconstructed /
 predicted / interpolated
 video frames]

Metaphor: A restoration painter with a photographic memory.

Imagine you’re a painter hired to recreate a burned-down museum from a written log of which objects moved where (the event stream). You’ve spent years painting from photographs of every room in the world (pre-training on video). Now:

• Event encoder = the log telling you “something moved left-to-right in this corner at this speed”
• Fine-tuned diffusion model = your trained hand painting plausible scenes that respect the log’s constraints
• Autoregressive unrolling = painting one wall at a time, then moving to the next
• Adaptive context switching = every few walls, you walk back to the museum entrance to re-orient yourself so you don’t gradually paint the hallway wider and wider
• Cross residual correction = for a wall between two rooms, you sketch it from Room A’s side, then sketch it from Room B’s side, then merge the two sketches so it looks consistent from both directions

Without the periodic walk-back (context switching), you’d end up painting an Escher painting. Without the cross-sketching (residual correction), the wall would look like two different walls depending on which room you’re standing in.

Key Concepts

• Event Voxel Grid: An event camera doesn’t record frames. Each pixel independently fires when its brightness changes, recording (+1 for brighter, -1 for darker) and a timestamp. To make this usable for neural networks, you bin events into a 3D grid: two spatial dimensions (x, y) and one temporal dimension (time slices). Each cell accumulates the signed events in that spatiotemporal volume. Think of it as a 3D point cloud where each point says “something happened here at this moment.” The density of events tells you about motion speed; the sign tells you about direction.

• Autoregressive Unrolling with Context Switching: When generating a long video, you can’t denoise the entire thing at once (memory explodes). So you generate 16 frames, then use the last 4 as “context” to generate the next 16, and so on. The catch: each generation is slightly imperfect, so after 10 rolls, small errors have snowballed into visible drift (the scene slowly changes hue, the background shifts). Adaptive Context Switching monitors this drift and periodically pulls in a longer-range context — a reference frame from much earlier — to reset the appearance. It’s like a long road trip where you mostly watch the GPS but every hour glance at the paper map in your lap to make sure the GPS hasn’t told you to drive into a lake.

• Zero-shot generalization: The model was fine-tuned on specific datasets but tested on scenes, objects, and motion patterns it never saw during training. Because it inherits strong priors from the pre-trained video diffusion model (which learned about lighting, materials, object structure from millions of generic videos), it can “fill in” plausible details for novel scenes — even if the event stream looks nothing like the training data. This is the payoff of leveraging foundation models rather than training from scratch.

Framework Shift

Before (mainstream):                   After (this paper):

Event Stream                           Event Stream
     |                                      |
     v                                      v
[Task-specific model]                [Unified diffusion model]
     |                                      |
     +---> Reconstruction model             |
     |     (regression, blurry)             +---> All 3 tasks
     +---> Prediction model                 |     (one fine-tuned model)
     |     (separate architecture)          |
     +---> Interpolation model              |
           (separate architecture)
                                          Drift? -> Context Switch
Each task: separate model,               Interp? -> Cross Residual
each with own failure mode

From task-specific regression pipelines to a unified generative framework, the core shift is treating event-to-video as a conditional generation problem where a single pre-trained video model handles all sub-tasks through shared priors and targeted fine-tuning.

Expert Assessment

Problem choice: Real gap. Event-based vision has been growing but remains niche — most methods are task-specific and struggle at long horizons. The unification of three tasks under one model is genuinely useful, not contrived. This sits at the intersection of event cameras (getting cheaper, finding use in autonomous driving and robotics) and foundation models (getting more capable), which is a productive place to be.

Method maturity: More clever than brute force, but not without cost. The context switching heuristic (drift threshold triggers global context injection) is practical but somewhat hand-tuned — I’d want to see ablations on how sensitive performance is to the threshold. Fine-tuning a pre-trained video diffusion model is the right play; training from scratch on event data would be hopeless given dataset sizes. One concern: inference speed. Diffusion models are slow, and autoregressive unrolling makes it slower. The paper should be more upfront about wall-clock times.

Experimental integrity: Baselines are reasonable — they compare against E2VID, E2VID+, FireNet, and diffusion-based approaches. Real-world benchmarks (MVSEC, HQF, IJRR) are standard. The zero-shot generalization experiments are a nice touch and actually more convincing than the supervised numbers. I’d flag that “outperforms SOTA across all three tasks” is a bold claim — the margins on interpolation may be thinner than on reconstruction. Wish they’d included failure cases.

Writing quality: Clean and structured, but the method section could use a single running figure that shows the full pipeline end-to-end rather than scattered diagrams. The Adaptive Context Switching explanation is somewhat buried — this is arguably their most novel engineering contribution and deserves a dedicated subsection with more visual explanation. The related work section is thorough but could be tighter.

Verdict: weak accept — the unification of three tasks under diffusion priors is a genuine contribution with practical value, but the drift mitigation heuristics need more rigorous analysis and the speed-accuracy tradeoff is conspicuously unexplored.

Takeaways

Three things worth stealing:

1. Fine-tune foundation models for niche modalities. Don’t train from scratch on event data — take a video diffusion model and condition it on events. This “adapter” pattern works because the visual priors transfer. Applies to any domain where labeled data is scarce but a foundation model for the output modality exists.

2. Context switching as drift control. When running any autoregressive model (video, audio, text), periodically injecting long-range context alongside local context prevents drift compounding. The threshold-based switching is simple to implement and likely transferable to autoregressive audio generation, long document summarization, etc.

3. Cross residual correction for bidirectional tasks. When you need consistency from two directions (e.g., forward and backward prediction, left and right stereo), encoding from both sides and correcting each with residuals from the other is a clean architectural trick. Could apply to stereo depth estimation, bidirectional translation, or any task requiring symmetric consistency.