HERMES++: Bridging Scene Understanding and Future Prediction in Autonomous Driving

Paper: 2604.28196 Authors: Xin Zhou, Dingkang Liang, Xiwu Chen, Feiyang Tan, Dingyuan Zhang, Hengshuang Zhao, Xiang Bai Categories: cs.CV

The Gap

Existing driving world models split into two camps. One camp (like UniAD, VAD) focuses on future scene generation—predicting where objects will be—but treats the present as a black box. They simulate forward in time but can’t answer “what is that object?” or “why did it move there?” The other camp uses Large Language Models for reasoning about scenes, but LLMs live in text space. They can tell you “the pedestrian might cross” but can’t predict the actual 3D trajectory or geometry evolution. You get either physics without semantics or semantics without physics.

The core problem: understanding (what/why) and prediction (where/when) require different representations. Scene understanding wants high-level semantic features compatible with language models. Future prediction needs dense geometric representations that capture spatial structure. Prior work treats these as separate tasks with separate models.

Problem: Semantic reasoning <--gap--> Geometric prediction
         (LLM space)                   (3D space)
            |                              |
            v                              v
Assumption: Can we unify them in one model?
            |
            v
Method: Shared BEV backbone + dual branches
        + knowledge transfer + temporal bridge
            |
            v
Evidence: Outperforms specialists on both tasks
          (nuScenes, Waymo benchmarks)
            |
            v
Conclusion: Unified model > separate models
            (synergy between understanding & prediction)

The Increment

One sentence: Before HERMES++, you needed separate models for “what’s in the scene” and “where things will go”—now one model does both by making LLM reasoning guide geometric prediction.

Core Mechanism

The architecture has three main components working in sequence. First, a BEV (bird’s-eye view) encoder processes multi-view camera images and converts them into a unified spatial representation—think of it as creating a top-down map from multiple camera angles. This BEV representation is crucial because it’s structured enough for geometric operations but also compatible with language model processing.

Second, the model splits into two branches that share this BEV backbone. The understanding branch uses LLM-enhanced world queries—learnable tokens that extract semantic information and pass through a language model to get high-level scene interpretation. The prediction branch takes the same BEV features but processes them through a different path focused on geometric evolution. Here’s the clever part: the understanding branch’s output (semantic context) gets injected into the prediction branch through a “Current-to-Future Link” module. This is where semantic reasoning guides geometric prediction.

Third, to keep the predicted geometry physically plausible, they use Joint Geometric Optimization—a dual regularization strategy. Explicit constraints enforce known geometric rules (like ground plane consistency), while implicit latent regularization aligns the model’s internal representations with geometry-aware priors learned during training. The prediction branch outputs future point clouds, while the understanding branch outputs semantic labels and relationships.

Multi-view Images
      |
      v
  [BEV Encoder] --> shared spatial representation
      |
      +------------------+------------------+
      |                                     |
      v                                     v
[Understanding Branch]              [Prediction Branch]
  LLM-enhanced queries              Geometric decoder
      |                                     ^
      |                                     |
      +---> [Current-to-Future Link] ------+
            (semantic guides geometry)
                                            |
                                            v
                                    [Joint Geometric Opt]
                                    (explicit + implicit)
                                            |
                                            v
                                    Future Point Clouds

Think of HERMES++ like a film director working with both a screenwriter and a cinematographer. The BEV encoder is the raw footage—all the visual information captured from multiple angles. The understanding branch (screenwriter) interprets what’s happening: “the pedestrian is waiting to cross, the car is slowing down.” The prediction branch (cinematographer) plans the next shots: “the pedestrian will be here, the car will be there.” The Current-to-Future Link is the director’s notes—the screenwriter’s interpretation directly informs how the cinematographer frames future shots. Without the director (the link), the cinematographer might predict physically plausible but contextually wrong movements (a car accelerating when it should brake). The Joint Geometric Optimization is like the continuity editor, ensuring the predicted shots don’t violate physical laws (cars don’t float, objects don’t teleport).

Key Concepts

• BEV (Bird’s-Eye View) Representation: Imagine you’re playing a strategy game where the camera looks straight down at the battlefield. That’s BEV—a top-down 2D grid where each cell contains information about what’s at that location in 3D space. Why is this useful? Multi-view cameras give you fragmented perspectives (front, left, right, back), but BEV consolidates them into one coherent spatial map. It’s like taking four photos of a room from different corners and stitching them into a floor plan. For autonomous driving, BEV naturally aligns with how we think about navigation (lanes, intersections, distances) and it’s computationally efficient because you’re working in 2D instead of full 3D. The key trick: BEV preserves spatial relationships (what’s next to what, how far apart) while being structured enough that language models can process it.

• Current-to-Future Link: This is the bridge between “what I understand now” and “what I predict next.” In most models, understanding and prediction are separate—like having a weather analyst and a fortune teller who never talk. The Current-to-Future Link makes them collaborate. Concretely, it takes semantic features from the understanding branch (high-level concepts like “pedestrian intending to cross”) and injects them into the prediction branch’s geometric processing. It’s implemented as a cross-attention mechanism where future queries attend to current semantic context. Example: without this link, the model might predict a pedestrian continues walking straight (geometrically plausible). With the link, semantic understanding says “pedestrian is looking at phone, likely to stop,” so the geometric prediction adjusts accordingly.

• Joint Geometric Optimization: Predicting future geometry is tricky because there are infinite ways to arrange points in 3D space, but only a tiny fraction are physically valid. This optimization strategy enforces validity through two complementary mechanisms. Explicit constraints are hard rules: “predicted points must lie on or above the ground plane,” “object boundaries must be continuous,” “motion must respect physics.” These are implemented as loss terms during training. Implicit regularization is softer: the model learns latent representations that naturally favor plausible geometries by training on real-world data. It’s like teaching someone to draw faces—you can give explicit rules (eyes above nose, nose above mouth) but they also develop an implicit sense of “this looks right” from seeing thousands of examples. Together, these prevent the model from generating nonsensical futures (cars sinking into the ground, objects fragmenting).

Framework Shift

Before (mainstream approach):        After (HERMES++):

Task 1: Scene Understanding          Unified Model:
[Images] -> [Specialist Model A]     [Images]
         -> Semantic Labels                |
                                           v
Task 2: Future Prediction            [Shared BEV Encoder]
[Images] -> [Specialist Model B]          |
         -> Future Geometry          +----+----+
                                     |         |
(Two separate models,                v         v
 no information sharing)        [Understand] [Predict]
                                     |         ^
                                     +----+----+
                                          |
                                    Semantic guides
                                    geometric evolution

From parallel specialists to collaborative branches, the core shift is making semantic understanding a first-class input to geometric prediction rather than treating them as independent tasks.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The autonomous driving community has been building increasingly sophisticated future prediction models (occupancy forecasting, trajectory prediction) while separately developing scene understanding systems. The insight that these should inform each other is obvious in hindsight but underexplored in practice. It sits at a natural convergence point in the field’s trajectory—LLMs are mature enough to handle structured spatial inputs, and geometric prediction has hit a ceiling without semantic context.

Method maturity: The approach is more architectural engineering than algorithmic breakthrough. The individual components (BEV encoding, cross-attention, geometric losses) are established techniques. The contribution is in the integration—showing that these pieces can work together synergistically. There’s a risk of over-engineering (four major components, each with hyperparameters), but the ablations suggest each piece pulls its weight. A simpler approach might be to just condition a prediction model on LLM embeddings, but the authors argue (convincingly) that bidirectional information flow matters.

Experimental integrity: The baselines are fair—they compare against strong specialists (BEVFormer for understanding, OccWorld for prediction) rather than strawmen. The numbers are solid across multiple benchmarks (nuScenes, Waymo). One yellow flag: the “outperforms specialists” claim is true on average but not uniformly—some specialist models still win on specific metrics. The paper is honest about this in tables but the abstract oversells slightly. The ablations are thorough, showing each component’s contribution. No major red flags, but I’d want to see failure case analysis (when does the semantic-geometric coupling hurt rather than help?).

Writing quality: The paper front-loads motivation well but buries implementation details. Section 3.3 (Joint Geometric Optimization) is dense and would benefit from a concrete example walkthrough. The related work section is comprehensive but reads like a literature dump—cutting it by 30% and integrating key comparisons into the method section would improve flow. The figures are clear but Figure 3 (architecture diagram) tries to show too much at once. Breaking it into two figures (understanding branch, prediction branch) would help. The experimental section is well-structured.

Verdict: weak accept — Solid integration work with thorough experiments, but the novelty is in composition rather than invention, and the complexity cost is non-trivial.

Takeaways

Cross-task knowledge transfer via attention: The Current-to-Future Link pattern is generalizable. If you have two related tasks (detection + tracking, segmentation + depth, etc.), instead of training separate models, use cross-attention to let one task’s features inform the other. The key is identifying which direction the information should flow and at what layer.

BEV as a universal interface: For any multi-view spatial reasoning problem, BEV is a strong default representation. It’s more structured than image features, more efficient than full 3D voxels, and compatible with both geometric operations and language model processing. If you’re working with camera arrays (robotics, surveillance, AR), consider BEV as your intermediate representation.

Dual regularization for structured prediction: When predicting structured outputs (geometry, graphs, sequences), combine explicit constraints (hard rules you know) with implicit regularization (learned priors from data). Explicit constraints prevent catastrophic failures; implicit regularization handles the long tail of edge cases you didn’t think to encode.

When to unify vs specialize: This paper shows unification wins when tasks share underlying structure (spatial reasoning) and one task’s output is a natural input to the other (semantics → geometry). But unification has costs—more complex training, harder debugging, potential for negative transfer. The decision heuristic: unify when the tasks have asymmetric information flow (one clearly informs the other), not just when they’re vaguely related.