TiCo: Time-Controllable Training for Spoken Dialogue Models

Paper: 2603.22267 Authors: Kai-Wei Chang, Wei-Chih Chen, En-Pei Hu, Hung-yi Lee, James Glass Categories: cs.CL

Abstract

This paper proposes TiCo, a simple post-training method for enabling spoken dialogue models (SDMs) to follow time-constrained instructions and generate responses with controllable duration. While existing SDMs can generate natural spoken responses, they lack time awareness and struggle to follow duration-related instructions (e.g., “Please generate a response lasting about 15 seconds”). The authors introduce Spoken Time Markers (STMs) — special tokens indicating estimated cumulative speaking duration at given positions in the intermediate representation. TiCo uses a two-stage training process: first learning time awareness through self-generation and SFT, then optimizing time controllability through RLVR. The paper also introduces TiCo-Bench, the first benchmark for evaluating time-controllable instruction following in SDMs. Experimental results show that TiCo significantly improves adherence to duration constraints while preserving response quality.

Key Contributions

• TiCo framework: A two-stage post-training method (SFT + RL) that equips spoken dialogue models with time-controllable generation through Spoken Time Markers
• TiCo-Bench: The first benchmark designed to evaluate time controllability of SDMs, with 1,440 evaluation samples across diverse tasks and modalities
• Strong empirical results: TiCo achieves 4.54s MAE and 14.9% MAPE overall, substantially outperforming both open-source SDMs and commercial models
• Generalization: The learned time-control capability extends beyond training duration ranges and transfers from speech queries to text queries

Why Time Control Matters

In real-world applications such as voice assistants, wearable devices, and medical/emergency scenarios, controlling response duration is critical. A voice assistant providing a traffic update while driving must be concise; a wearable device may require brief feedback due to battery constraints; emergency instructions must be delivered within strict time pressure. Despite this importance, time controllability in spoken dialogue models has remained largely unexplored.

Controlling duration in speech is considerably harder than controlling text length. A single word may span different numbers of syllables, and speech duration varies with phonetic composition, speaking rate, and prosodic structure. Simply constraining word count does not guarantee accurate duration control.

Methodology

Stage 1: Time-Awareness Training

The first stage teaches the model to generate Spoken Time Markers (STMs) as part of its intermediate representation. The process works as follows:

1. Self-generation: Given input speech queries, the model freely generates responses (both intermediate text and speech output)
2. ASR-based alignment: Whisper is used to obtain word-level timestamps, establishing temporal correspondence between the intermediate representation and the spoken output
3. STM insertion: Time markers are interleaved with the text tokens at sentence-level punctuation marks (e.g., Well, <0.9 seconds> milk<1.8 seconds> ...)
4. SFT training: The model is fine-tuned to predict these augmented sequences autoregressively

Self-generation has two advantages: it eliminates the need for paired QA data, and the generated responses follow the model’s own output distribution, improving training stability.

Stage 2: Time-Controllable Training

The second stage uses Reinforcement Learning with Verifiable Rewards (RLVR) to optimize the model for following explicit duration instructions. Key design choices:

• GRPO optimization: The model generates multiple candidate responses for each time-constrained prompt
• Main reward: A Gaussian function measuring accuracy of predicted total duration relative to the instructed target: $R_{\text{main}} = \exp\left(-(\Delta t)^2 / 2\sigma^2\right)$
• Auxiliary rewards: Presence reward (generate at least one STM), monotonicity reward (time markers should increase), repetition penalty, and copy penalty (prevent trivially copying the target duration)
• CHORD regularization: Dynamically weighted SFT loss from Stage 1 data is mixed with the GRPO objective to prevent reward hacking and catastrophic forgetting

Architecture

TiCo is implemented on Qwen2.5-Omni-7B, which uses a “Thinker-Talker” architecture. Only the Thinker (intermediate sequence generator) is trained; the Talker (speech synthesizer) remains frozen. During inference, STMs are used for planning and removed before passing the sequence to the Talker. Importantly, TiCo is architecture-agnostic and can be applied to any SDM with an intermediate representation.

TiCo-Bench

The benchmark draws from four sources:

• InstructS2S: Speech question answering (180 queries)
• UROBench: Reasoning subset, both speech and text (180 queries each)
• LIFEBench: Text-based QA, creative generation, reasoning, and summarization (180 queries)

Each query is evaluated under two temporal constraints (short: 10-30s, long: 30-60s), yielding 1,440 total evaluation samples. Metrics include MAE (seconds), MAPE (%), and GPT-score (1-5 quality rating).

Results

Overall Performance

TiCo achieves the best overall performance across all settings:

Model	MAE (s)	MAPE (%)	GPT-score
TiCo (Proposed)	4.54	14.9	3.56
Cascade (GPT)	10.41	29.0	4.15
Qwen2.5-Omni-7B	13.01	42.3	3.57
GPT-audio	13.93	49.2	3.88
Kimi Audio	29.29	114.5	1.67

TiCo reduces MAE by 65% compared to the backbone model and by 56% compared to the strongest cascaded baseline, while maintaining comparable response quality (GPT-score 3.56 vs. 3.57 for the backbone).

Generalization

• Longer durations: Despite being trained with a maximum of ~41 seconds of speech, TiCo generalizes well to durations up to 60 seconds, maintaining low error across all duration bins
• Text queries: Trained exclusively on speech queries, TiCo transfers effectively to text-query inputs (5.35s MAE, 18.0% MAPE on text benchmarks), outperforming all baselines including Cascade (GPT)

Qualitative Examples

The model adapts content length to match target durations while preserving core semantics. For the same question (“How deep is the ocean?”):

• 15s target: Concise answer covering key facts (actual: 15.2s)
• 40s target: Extended answer with additional elaboration and follow-up content (actual: 41.6s)

Takeaways

TiCo demonstrates that time controllability can be effectively learned as a post-training capability for spoken dialogue models. The Spoken Time Marker mechanism provides an elegant solution for bridging the gap between semantic planning and temporal realization in speech. The two-stage approach — first learning time awareness, then optimizing for time-constrained instruction following — proves both simple and effective. The framework’s generalization to unseen duration ranges and input modalities suggests that temporal control can serve as a robust intermediate planning capability, opening the door for more controllable and practical spoken language systems.