Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device

Paper: 2602.20161 Authors: Abdelrahman Shaker, Ahmed Heakl, Jaseel Muhammad, Ritesh Thawkar, Omkar Thawakar, Senmao Li, Hisham Cholakkal, Ian Reid, Eric P. Xing, Salman Khan Categories: cs.CV

Abstract

Mobile-O introduces the first practical unified multimodal model capable of both understanding and generating visual content directly on mobile devices. Unlike existing unified models that require substantial computational resources and cloud infrastructure, Mobile-O achieves real-time performance on edge devices through its novel Mobile Conditioning Projector (MCP) architecture. The model efficiently fuses vision-language features with diffusion-based generation using depthwise-separable convolutions and layerwise alignment, enabling cross-modal conditioning with minimal overhead. Trained on only a few million samples using a novel quadruplet format (generation prompt, image, question, answer), Mobile-O demonstrates competitive or superior performance compared to larger unified models while running significantly faster—achieving 74% on GenEval and processing 512×512 images in approximately 3 seconds on an iPhone.

Key Contributions

• Mobile Conditioning Projector (MCP): A lightweight architecture that efficiently bridges vision-language understanding and diffusion-based generation using depthwise-separable convolutions and layerwise alignment mechanisms
• Quadruplet Training Format: Novel post-training approach using (generation prompt, image, question, answer) tuples that jointly enhances both understanding and generation capabilities
• Efficient On-Device Deployment: First unified multimodal model achieving real-time performance on mobile devices (~3s per 512×512 image on iPhone) without cloud dependency
• Superior Efficiency-Performance Trade-off: Outperforms Show-O by 5% and JanusFlow by 11% on GenEval while running 6× and 11× faster respectively; surpasses them by 15.3% and 5.1% on visual understanding benchmarks

Architecture and Methodology

The Mobile-O architecture consists of three primary components working in concert: a vision encoder for image understanding, a language model for text processing and reasoning, and a diffusion generator for image synthesis. The innovation lies in the Mobile Conditioning Projector (MCP), which serves as the critical bridge between these modalities.

The MCP employs depthwise-separable convolutions—a technique that factorizes standard convolutions into depthwise and pointwise operations, dramatically reducing computational complexity from $O(D_K \cdot D_K \cdot M \cdot N)$ to $O(D_K \cdot D_K \cdot M + M \cdot N)$, where $D_K$ is kernel size, $M$ is input channels, and $N$ is output channels. This architectural choice enables efficient feature transformation while maintaining representational capacity.

Layerwise alignment in the MCP ensures that features from the vision-language pathway are properly conditioned at multiple scales within the diffusion generator. Rather than injecting conditioning information at a single point, the MCP distributes alignment across multiple layers, allowing the diffusion process to leverage both high-level semantic information and low-level visual details. This multi-scale conditioning proves crucial for generating coherent images that accurately reflect textual descriptions.

The training methodology introduces a quadruplet format that simultaneously addresses understanding and generation tasks. Each training sample consists of four elements: a generation prompt describing desired visual content, the corresponding image, a question about visual content, and the answer. This format enables the model to learn bidirectional mappings between vision and language—both generating images from text and understanding images through question-answering. The joint training approach creates synergies between tasks, with understanding capabilities informing better generation and vice versa.

Performance Analysis

Mobile-O demonstrates remarkable performance across both generation and understanding benchmarks despite its compact size and efficiency focus. On GenEval, a comprehensive benchmark for text-to-image generation quality, Mobile-O achieves 74% accuracy, outperforming Show-O by 5 percentage points and JanusFlow by 11 percentage points. More impressively, these gains come alongside dramatic speed improvements—Mobile-O runs 6× faster than Show-O and 11× faster than JanusFlow.

For visual understanding tasks, Mobile-O shows even more pronounced advantages. Averaged across seven standard benchmarks including VQA, image captioning, and visual reasoning tasks, Mobile-O surpasses Show-O by 15.3% and JanusFlow by 5.1%. This substantial gap suggests that the quadruplet training format and efficient architecture don’t merely preserve understanding capabilities—they actively enhance them, possibly by creating richer cross-modal representations through joint training.

The efficiency metrics are particularly striking for mobile deployment. Processing a 512×512 image takes approximately 3 seconds on an iPhone, making Mobile-O the first unified multimodal model practical for real-time mobile applications. This performance is achieved with a model trained on only a few million samples, orders of magnitude less data than typical large-scale vision-language models require. The data efficiency stems from the focused training approach and architectural design that maximizes information extraction from each sample.

Implications and Future Directions

Mobile-O represents a paradigm shift in multimodal AI deployment, demonstrating that unified understanding and generation capabilities need not be confined to cloud infrastructure or high-end hardware. The ability to run sophisticated multimodal models entirely on-device has profound implications for privacy, latency, and accessibility. Users can generate and analyze images without transmitting sensitive data to remote servers, applications can function without internet connectivity, and the computational costs of serving millions of users shift from centralized data centers to distributed edge devices.

The architectural innovations in Mobile-O—particularly the MCP and quadruplet training—offer blueprints for future efficient multimodal systems. The depthwise-separable convolution approach and layerwise alignment could be adapted to other cross-modal tasks beyond vision and language. The quadruplet training format suggests that carefully designed multi-task learning can achieve better results with less data than separate task-specific training.

Several research directions emerge from this work. Extending Mobile-O to handle video understanding and generation would enable temporal reasoning on mobile devices. Incorporating additional modalities like audio could create truly comprehensive multimodal assistants. Exploring quantization and neural architecture search could further reduce computational requirements. Investigating how Mobile-O’s joint training approach scales to larger models and datasets could inform the design of next-generation unified models.

The open-source release of code, models, datasets, and mobile application democratizes access to unified multimodal AI, enabling researchers and developers to build upon this foundation. This accessibility could accelerate innovation in on-device AI and inspire new applications that leverage real-time multimodal intelligence.

Takeaways

1. Mobile-O achieves the first practical unified multimodal understanding and generation on mobile devices, processing 512×512 images in ~3 seconds on iPhone without cloud dependency
2. The Mobile Conditioning Projector (MCP) efficiently bridges vision-language and diffusion models using depthwise-separable convolutions and layerwise alignment with minimal computational overhead
3. Novel quadruplet training format (generation prompt, image, question, answer) jointly enhances both understanding and generation capabilities using only a few million samples
4. Mobile-O outperforms Show-O and JanusFlow by 5% and 11% on GenEval while running 6× and 11× faster, and surpasses them by 15.3% and 5.1% on visual understanding benchmarks
5. The work demonstrates that efficient architecture design and focused training can achieve competitive performance with dramatically reduced computational requirements and training data