Paper: 2604.21928 Authors: Research Team Categories: cs.CL, cs.LG

Problem

Traditional ASR evaluation relies on Word Error Rate (WER), which:

  • Measures surface-level transcription accuracy only
  • Treats all errors equally (substitutions, deletions, insertions)
  • Ignores semantic meaning and context
  • Fails to capture functional adequacy for downstream tasks

LLM-based Evaluation

The paper proposes using generative LLMs to evaluate ASR output:

Evaluation Dimensions

  1. Semantic Accuracy: Does the transcription preserve meaning?
  2. Contextual Coherence: Is the output contextually appropriate?
  3. Task Usefulness: Is the transcription useful for downstream tasks?
  4. Error Severity: Are errors critical or minor?

Method

  • Feed ASR output + reference into LLM
  • Ask LLM to assess functional equivalence
  • Use confidence scores and explanations

Results

Experiments show:

  • LLM evaluation correlates better with human judgment than WER
  • Captures semantic equivalence even with surface differences
  • Identifies “acceptable” ASR outputs that WER would flag as errors

Takeaways

  • WER is necessary but not sufficient for ASR evaluation
  • LLMs can provide nuanced, semantic-level assessment
  • Future ASR benchmarks should include LLM-based evaluation

论文: 2604.21928 作者: 研究团队 分类: cs.CL, cs.LG

问题

传统的ASR评估依赖于词错误率（WER），它：

  • 仅衡量表面级别的转录准确率
  • 对所有错误一视同仁（替换、删除、插入）
  • 忽略语义含义和上下文
  • 无法捕捉下游任务的功能适用性

基于LLM的评估

论文提出使用生成式LLM评估ASR输出：

评估维度

  1. 语义准确率：转录是否保留了含义？
  2. 上下文连贯性：输出是否上下文适当？
  3. 任务有用性：转录对下游任务有用吗？
  4. 错误严重性：错误是关键还是次要？

方法

  • 将ASR输出 + 参考输入LLM
  • 让LLM评估功能等效性
  • 使用置信度分数和解释

实验结果

实验表明：

  • LLM评估与人类判断的相关性优于WER
  • 即使有表面差异也能捕捉语义等效性
  • 识别WER会标记为错误的”可接受”ASR输出

要点总结

  • WER对于ASR评估是必要的但不够充分
  • LLM可提供细致的、语义级别的评估
  • 未来的ASR基准测试应包括基于LLM的评估