VHASR: A Multimodal Speech Recognition System With Vision Hotwords

TL;DR

VHASR introduces a multimodal speech recognition system that leverages vision hotwords through a dual-stream architecture, significantly improving recognition accuracy over unimodal models and achieving state-of-the-art results in image-based ASR.

Contribution

The paper presents a novel multimodal ASR system utilizing vision hotwords with a dual-stream architecture, enhancing speech recognition performance.

Findings

VHASR outperforms unimodal ASR models.

Achieves state-of-the-art results on multiple datasets.

Effectively utilizes image information to improve recognition.

Abstract

The image-based multimodal automatic speech recognition (ASR) model enhances speech recognition performance by incorporating audio-related image. However, some works suggest that introducing image information to model does not help improving ASR performance. In this paper, we propose a novel approach effectively utilizing audio-related image information and set up VHASR, a multimodal speech recognition system that uses vision as hotwords to strengthen the model's speech recognition capability. Our system utilizes a dual-stream architecture, which firstly transcribes the text on the two streams separately, and then combines the outputs. We evaluate the proposed model on four datasets: Flickr8k, ADE20k, COCO, and OpenImages. The experimental results show that VHASR can effectively utilize key information in images to enhance the model's speech recognition ability. Its performance not only surpasses unimodal ASR, but also achieves SOTA among existing image-based multimodal ASR.