Micro Language Models Enable Instant Responses

Paper: 2604.19642 Authors: Wen Cheng, Tuochao Chen, Karim Helwani, Sriram Srinivasan, Luke Zettlemoyer, Shyamnath Gollakota Categories: cs.AI, cs.LG

Abstract

Edge devices such as smartwatches and smart glasses cannot continuously run even the smallest 100M-1B parameter language models due to power and compute constraints, yet cloud inference introduces multi-second latencies that break the illusion of a responsive assistant. We introduce micro language models (μLMs): ultra-compact models (8M-30M parameters) that instantly generate the first 4-8 words of a contextually grounded response on-device, while a cloud model completes it; thus, masking the cloud latency.

Key Contributions

• Extreme Scale Efficacy: Trained 8.8M-29.5M parameter decoder-only μLMs from scratch on chat-style data
• Collaborative Generation Framework: Reframes cloud model as continuator rather than respondent, enabling seamless mid-sentence handoffs
• Error Recovery Methods: Three correction modes (Explicit, Natural Recovery, Humor-aware) for graceful recovery when local opener goes wrong
• Sub-second Latency: 28M μLM achieves 45ms time to first token on embedded hardware

Methodology

The approach trains ultra-compact decoder-only Transformers with:

• Pre-norm residual blocks, rotary positional embeddings, RMSNorm
• Grouped-query attention with fewer key-value heads
• Tied input/output embeddings

Training is compute-matched across variants, with larger models trained for fewer steps. The collaborative framework uses instruction prompting to make the cloud model act as a continuator.

Results

• Despite their extreme size, 14-29M μLMs are competitive with 70M-256M baseline models
• 4-8 words emerges as optimal operating point with error recovery rates below 3.7-8.4%
• User study shows 49% equivalence, 28% preference for collaborative over standalone
• On Orange Pi: 45ms TTFT, 3ms first token decode, 4 words in 55ms

Takeaways

• Useful language generation survives at extreme parameter scales (8M-30M)
• Orders-of-magnitude asymmetric collaboration is achievable
• Opens door to responsive AI on smartwatches, glasses, and other wearable devices