Mi-Go: Test Framework which uses YouTube as Data Source for Evaluating Speech Recognition Models like OpenAI's Whisper

TL;DR

Mi-Go is a new testing framework that uses YouTube videos to evaluate speech recognition models like OpenAI's Whisper across diverse real-world scenarios, enhancing robustness and adaptability assessment.

Contribution

Introduces a novel framework leveraging YouTube as a data source for comprehensive evaluation of speech recognition models' performance and adaptability.

Findings

YouTube is effective for testing speech recognition models.

Whisper's performance varies across languages and audio qualities.

Mi-Go can identify potential misuse of subtitles.

Abstract

This article introduces Mi-Go, a novel testing framework aimed at evaluating the performance and adaptability of general-purpose speech recognition machine learning models across diverse real-world scenarios. The framework leverages YouTube as a rich and continuously updated data source, accounting for multiple languages, accents, dialects, speaking styles, and audio quality levels. To demonstrate the effectiveness of the framework, the Whisper model, developed by OpenAI, was employed as a test object. The tests involve using a total of 124 YouTube videos to test all Whisper model versions. The results underscore the utility of YouTube as a valuable testing platform for speech recognition models, ensuring their robustness, accuracy, and adaptability to diverse languages and acoustic conditions. Additionally, by contrasting the machine-generated transcriptions against human-made subtitles, the Mi-Go framework can help pinpoint potential misuse of YouTube subtitles, like Search Engine Optimization.