# Factual Consistency Oriented Speech Recognition

## TL;DR

This paper introduces a new optimization framework for speech recognition that reduces hallucinations by maximizing factual consistency between hypotheses and ground-truth, leading to more accurate and reliable transcriptions.

## Contribution

It proposes a novel framework that enhances ASR factual accuracy by integrating a separate estimator into the training process, improving consistency without increasing word error rates.

## Key findings

- Higher factual consistency scores in ASR hypotheses
- Maintains comparable word error rates to traditional models
- Improves speech summarization quality via increased factual accuracy

## Abstract

This paper presents a novel optimization framework for automatic speech recognition (ASR) with the aim of reducing hallucinations produced by an ASR model. The proposed framework optimizes the ASR model to maximize an expected factual consistency score between ASR hypotheses and ground-truth transcriptions, where the factual consistency score is computed by a separately trained estimator. Experimental results using the AMI meeting corpus and the VoxPopuli corpus show that the ASR model trained with the proposed framework generates ASR hypotheses that have significantly higher consistency scores with ground-truth transcriptions while maintaining the word error rates close to those of cross entropy-trained ASR models. Furthermore, it is shown that training the ASR models with the proposed framework improves the speech summarization quality as measured by the factual consistency of meeting conversation summaries generated by a large language model.