DDTSR: Enabling Human-Like Responsiveness in Spoken Dialogue Through Dual-Track Streaming

Paper: 2602.23266 Authors: Siyuan Liu, Jiahui Xu, Feng Jiang, Kuang Wang, Zefeng Zhao, Chu-Ren Huang, Jinghang Gu, Changqing Yin, Haizhou Li Categories: cs.CL

Abstract

Traditional spoken dialogue systems suffer from high latency due to their strictly sequential ASR-LLM-TTS pipeline, where each component must complete before the next begins. This paper introduces the Discourse-Aware Dual-Track Streaming Response (DDTSR) framework, which fundamentally reimagines the architecture to enable concurrent processing across modalities. By leveraging discourse connectives as minimal-committal early responses generated by a small auxiliary model while a large model performs deep reasoning in parallel, DDTSR achieves human-like responsiveness. The framework incorporates streaming-based cross-modal collaboration to overlap ASR, LLM inference, and TTS operations, and employs curriculum learning to maintain discourse coherence between early and later responses. Experimental results demonstrate 19-51% latency reduction across benchmarks while preserving response quality, with strong compatibility across different LLM backbones and utterance lengths.

Key Contributions

• Dual-track architecture: Introduces parallel processing where a small model generates discourse connectives (e.g., “Well,” “Actually”) while a large model performs knowledge-intensive reasoning, enabling immediate response initiation
• Streaming cross-modal collaboration: Dynamically overlaps ASR transcription, LLM token generation, and TTS synthesis to minimize the time-to-first-speech
• Curriculum-based coherence training: Develops a training strategy that ensures logical consistency between early minimal-committal responses and subsequent reasoning outputs
• Plug-and-play compatibility: Demonstrates framework applicability across diverse LLM backbones without architecture-specific modifications

Technical Architecture

The DDTSR framework operates through three interconnected stages that break the sequential bottleneck of traditional pipelines.

Connective-Guided Small-Large Model Synergy: The system employs a lightweight auxiliary model specifically trained to generate discourse connectives—linguistically minimal phrases that signal turn-taking without committing to specific content. While this small model produces immediate verbal feedback (typically within 200-300ms), the primary large language model begins parallel processing of the complete query. This dual-track approach mirrors human conversation patterns where speakers often begin responses with filler phrases while formulating complete thoughts.

Streaming-Based Cross-Modal Collaboration: Rather than waiting for complete outputs from each component, DDTSR implements a token-level streaming mechanism. As ASR produces partial transcriptions, the LLM begins generating tokens, and TTS synthesizes speech from the earliest available tokens. The framework calculates the “earliest speakable moment” $t_{speak}$ as:

$t_{speak} = \min(t_{ASR} + t_{LLM}^{first} + t_{TTS}^{first})$

where $t_{ASR}$ represents partial transcription time, $t_{LLM}^{first}$ is the time to generate the first reasoning token, and $t_{TTS}^{first}$ is the time to synthesize the first speech segment.

Discourse Continuity Enhancement: To prevent jarring transitions between connective-based early responses and reasoning-based complete responses, DDTSR employs curriculum learning. The training process gradually increases the complexity of maintaining coherence, starting with simple connective-to-statement transitions and progressing to complex multi-turn discourse patterns. This ensures that early responses naturally lead into subsequent content without semantic discontinuity.

Experimental Results and Analysis

The authors evaluated DDTSR on two spoken dialogue benchmarks: a customer service dataset and an open-domain conversation corpus. The framework achieved latency reductions of 19% on the customer service benchmark and 51% on the open-domain benchmark compared to traditional sequential pipelines.

Latency Breakdown: Analysis reveals that the latency gains come primarily from three sources: (1) 30-40% reduction from parallel small-large model processing, (2) 25-35% reduction from streaming cross-modal collaboration, and (3) 10-15% reduction from optimized discourse planning. Notably, the framework maintains response quality metrics, with BLEU scores within 2% of baseline systems and human evaluation scores showing no significant degradation in coherence or informativeness.

Robustness Analysis: The system demonstrates consistent performance across varying utterance lengths (50-500 tokens) and different LLM backbones (tested with GPT-style, LLaMA-style, and instruction-tuned models). The plug-and-play nature stems from the framework’s modular design, where the connective generation module operates independently of the specific LLM architecture.

Discourse Quality Preservation: Human evaluators rated DDTSR responses on naturalness, coherence, and informativeness. Results show that 87% of responses were rated as “natural” or “very natural,” with discourse connectives perceived as appropriate conversation markers rather than awkward fillers. The curriculum learning approach proved critical—ablation studies removing this component resulted in a 23% increase in perceived discontinuity between early and late response segments.

Implications for Real-Time Interaction

DDTSR represents a paradigm shift in spoken dialogue system design, moving from sequential processing to concurrent multi-track execution. The framework’s ability to reduce latency while maintaining quality addresses a fundamental challenge in human-computer interaction: the expectation of immediate responsiveness.

The use of discourse connectives as a bridging mechanism is linguistically grounded—human speakers naturally employ such devices to maintain conversational flow while formulating complex responses. By computationally modeling this behavior, DDTSR achieves more natural interaction patterns.

The framework’s scalability is particularly noteworthy. As LLMs continue to grow in size and capability, the latency problem typically worsens. DDTSR’s architecture provides a path forward where increased model capability doesn’t necessarily translate to degraded user experience. The small model overhead is minimal (typically <5% of total compute), making the approach practical for production deployment.

Future directions include extending the framework to multilingual scenarios, incorporating visual modalities for multimodal dialogue, and exploring adaptive connective selection based on user preferences and conversation context.

Takeaways

1. Breaking the sequential ASR-LLM-TTS pipeline through parallel processing can reduce spoken dialogue latency by up to 51% without sacrificing response quality
2. Discourse connectives serve as effective minimal-committal responses that maintain conversational flow while complex reasoning occurs in parallel
3. Streaming-based cross-modal collaboration enables token-level overlap between transcription, generation, and synthesis, advancing the earliest speakable moment
4. Curriculum learning is essential for maintaining discourse coherence between early connective-based responses and subsequent reasoning outputs
5. The framework functions as a plug-and-play module compatible with diverse LLM architectures, demonstrating strong practical applicability for real-time spoken interaction systems