HOMIE: Unifying Human-Object Video Personalization with Multimodal Intelligence

Paper: 2607.18217 Authors: Yiyang Cai, Nan Chen, Rongchang Xie, Junwen Pan, Chunyang Jiang, Cheng Chen, Wen Zhou, Zhenbang Sun, Wei Xue, Wenhan Luo Categories: cs.CV

The Gap

Existing research in subject-driven video generation, particularly for human-object centric video personalization (HOCVP), is stuck in a tough trade-off. Methods focusing on inter-subject personalization can make a person and an object look right individually, but they often botch the interaction between them—like getting the face perfect but the handshake weird. This is even harder when the “object” is abstract, like a logo. On the other hand, using extra reference inputs (like multiple views or text maps, i.e., intra-subject references) could help, but current systems lack a good way to understand and use this extra information without breaking the model’s controllability or getting too expensive to retrain.

[Problem: HOCVP struggles with]
|         |
v         v
[Inter-Subject Personalization] [Intra-Subject References]
|                            |     |                   |
v                            v     v                   v
[Poor Human-Object           [Lack of            [Costly Re-alignment]
 Interaction & Control]        Correspondence]     |
                           |   |   |               v
                           v   v   v               [No Unified
                       [Need for a unified solution]    Framework]
                           |
                           v
[HOMIE: Integrates MLLM Knowledge via Novel Mechanisms]
                           |
                           v
[SOTA Results & Ablation on Inter/Intra Tasks]
                           |
                           v
[Conclusion: Unified, controllable, and efficient HOCVP]

The Increment

One sentence: Before this paper, achieving high-fidelity, interactive personalized videos required fragile, disconnected methods; after, a single framework gracefully integrates rich semantic understanding from multimodal LLMs into the generation process, setting a new performance standard.

Core Mechanism

HOMIE’s core idea is to smartly inject knowledge from a powerful Multimodal Large Language Model (MLLM) into a standard diffusion-based video generator. The MLLM is great at understanding relationships (like “person holding cup” or “person wearing shirt with logo”), but its features live in a different “language” than the image features the diffusion model works with. Directly mixing them causes chaos, losing the precise control of text prompts.

HOMIE’s solution has two key parts. First, it uses a Global Multimodal Guidance mechanism within the diffusion model’s self-attention layers. Think of it as the MLLM giving the video model a high-level “semantic briefing” at every step of generation, guiding the overall scene composition without overwriting the detailed instructions from the text prompt. Second, to handle the tricky intra-subject references (e.g., “here’s a picture of the exact person I want”), it introduces Modality-Reference Embedding. This acts like a smart tagging system. It tags tokens coming from the MLLM and from the VAE (the image feature extractor) differently, and crucially, it explicitly links the tokens from the reference image to the corresponding tokens in the generated video frame, ensuring the subject’s identity is preserved consistently.

[Reference Image(s)] --> [VAE Encoder] --> [VAE Tokens (Tag A)]
         |                                         |
         +-------> [MLLM] --> [MLLM Features] --> [Multimodal Guidance Module]
         |                                         |
[Text Prompt] --> [Text Encoder] --> [Text Tokens] |
         |                                         | (Integrated via self-attention)
         v                                         v
[Diffusion Transformer (Self-Attention Layer)] <--- [Modality-Reference Embedding: Tags & Links]
         |
         v
[Denoising Process] --> [Generated Video Frame]

Structural Metaphor: A Film Director’s Crew Imagine the diffusion model is the cinematographer (handles the camera, lighting, visuals). The text prompt is the screenplay (the scene description). The MLLM is the veteran art director who understands deep relationships (e.g., “this character’s emotional state should be reflected in how they hold the prop”).

• Global Multimodal Guidance: This is like the art director (MLLM) constantly whispering semantic advice (“More tension in their posture”) to the cinematographer (diffusion model) on set, improving the scene’s logic without rewriting the screenplay (text prompt).
• Modality-Reference Embedding: This is the set continuity manager. They take a reference photo of the actor (VAE token from reference image) and physically put them on set, tagging them as “the main actor.” They ensure the same actor (same tag) is filmed from every angle, maintaining perfect continuity (intra-subject fidelity). They also make sure the actor interacts properly with the prop (art director’s advice). Without the art director’s guidance, the scene is technically correct but emotionally hollow. Without the continuity manager, you get a different-looking actor in every shot. HOMIE hires both for the same production.

Key Concepts

• Human-Object Centric Video Personalization (HOCVP): This is the specific job. You’re given reference pictures of a person and an object (which could be a physical thing or an abstract logo), and you need to generate a new video where that specific person is doing something with that specific object. The hard part isn’t just putting them in a video; it’s making the interaction look natural and true to life.
• Multimodal Large Language Model (MLLM) Integration: Instead of just using an MLLM to caption images, HOMIE uses it as a deep relationship analyzer. The key is the *integration strategy: it doesn’t replace the text encoder (which would break controllability) or require a massive retraining of the whole system. It adds a new “guidance stream” of semantic features that the generation process can attend to.
• Intra-Subject vs. Inter-Subject: “Inter-subject” means the relationship *between different subjects (human and object). “Intra-subject” means the consistency within one subject (the person in the video looking like the person in the reference photo). Previous methods often focused on one; HOMIE unifies them. A simple example: inter-subject fidelity is about making sure the person is holding the logo-branded cup correctly. Intra-subject fidelity is about making sure that person looks like the same person in every frame of the video.

Framework Shift

Before (mainstream approach):            After (this paper):
[Text Prompt] --> [Generator]            [Text Prompt] --> [Generator] <---+
[Ref Image 1] -----> (Maybe attention)   [Ref Image(s)] --> [VAE] -->[Tagged Tokens]-+
[Ref Image 2] -----> (Maybe separate)                                          |    |
                                        [MLLM] -------> [Semantic Features]---->[Global &
Multiple paths, fragile balancing.      Unified pathway with explicit            Modality-
MLLM knowledge hard to inject.          tagging and guidance mechanisms.         Reference]

From X to Y, the core shift is from fragmented, post-hoc adjustments to a unified, architecturally integrated guidance system for semantics and identity.

Expert Assessment

Problem choice: This is a real and important gap. HOCVP is where the rubber meets the road for subject-driven video generation. The inter/intra-subject tension is a fundamental bottleneck, and the problem of “abstract objects” (logos) is cleverly highlighted. It sits squarely at the frontier of making diffusion models truly controllable for complex, multi-entity scenes.

Method maturity: It’s clever engineering more than a brute-force throw of compute. The insight to use self-attention for *global guidance and a special embedding for explicit token association is sound and targeted. They avoid the simpler (but destructive) approach of replacing the text encoder. Are there simpler approaches? Maybe, but they likely wouldn’t achieve the same balance. The design feels intentional.

Experimental integrity: The experiments are comprehensive. They test on both inter and intra tasks, include challenging abstract concepts (logos), and provide meaningful ablations (e.g., removing guidance vs. removing embedding). Comparisons against strong baselines (like DreamVideo and VideoComposer) appear fair. The numbers look convincing, though real-world robustness always needs more diverse testing. No obvious red flags.

Writing quality: The paper is generally clear but can be dense. The “Method” section, while detailed, could be more pedagogical. Rewriting the first two pages of the method to walk through the film crew metaphor (or similar) before diving into the technical formulas would make the core contribution instantly graspable to a broader audience. The related work section is solid.

Verdict: Weak Accept — The contribution is incremental but significant. It unifies two key problems in HOCVP with a well-motivated and effective architectural solution, backed by solid experiments. It moves the state-of-the-art forward in a clear, practical way.

Takeaways

1. The Unified Framework Principle: When you have two related but distinct sub-problems (here, inter- and intra-subject fidelity), don’t treat them as separate tasks with separate fixes. Design a single architectural component (like Modality-Reference Embedding) that can naturally handle both by construction.
2. Semantic Guidance via Attention: For injecting knowledge from a powerful external model (like an MLLM) into a generative process, consider using cross- or self-attention layers as the “integration point.” This allows for continuous, nuanced guidance without disrupting the core controllability of the original model (e.g., the text encoder).
3. Tag, Don’t Mix: When combining features from different modalities (image features from VAE, semantic features from MLLM), don’t just concatenate or add them. Use a learned embedding or positional tag to explicitly distinguish their source. This lets the downstream network treat them appropriately and maintain their unique characteristics.