Speech LLMs are Contextual Reasoning Transcribers

View PDF HTML (experimental)

Abstract:Despite extensions to speech inputs, effectively leveraging the rich knowledge and contextual understanding of large language models (LLMs) in automatic speech recognition (ASR) remains non-trivial, as the task primarily involves direct speech-to-text mapping. To address this, this paper proposes chain-of-thought ASR (CoT-ASR), which constructs a reasoning chain that enables LLMs to first analyze the input speech and generate contextual analysis, thereby fully exploiting their generative capabilities. With this contextual reasoning, CoT-ASR then performs more informed speech recognition and completes both reasoning and transcription in a single pass. Moreover, CoT-ASR naturally supports user-guided transcription: while designed to self-generate reasoning, it can also seamlessly incorporate user-provided context to guide transcription, further extending ASR functionality. To reduce the modality gap, this paper introduces a CTC-guided Modality Adapter, which uses CTC non-blank token probabilities to weight LLM embeddings, efficiently aligning speech encoder outputs with the LLM's textual latent space. Experiments show that, compared to standard LLM-based ASR, CoT-ASR achieves a relative reduction of 8.7% in word error rate (WER) and 16.9% in entity error rate (EER).

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2604.00610 [cs.CL]

(or arXiv:2604.00610v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2604.00610

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Keqi Deng [view email] [v1] Wed, 1 Apr 2026 08:13:50 UTC (248 KB)