# Benchmarking Large Pretrained Multilingual Models on Qu\'ebec French Speech Recognition

## TL;DR

This paper assesses large pretrained multilingual speech recognition models on Quebec French, revealing their performance varies significantly on regional speech data and highlighting the need for regional benchmarks.

## Contribution

The paper introduces a new benchmark and evaluation pipeline for Quebec French speech recognition, focusing on regional speech variations and real-world conditions.

## Key findings

- Models perform poorly on regional Quebec French data compared to standard benchmarks.
- Benchmarking reveals discrepancies between well-known benchmark results and real-world regional speech recognition.
- Results inform practitioners on challenges of deploying speech models in regional language contexts.

## Abstract

We evaluate the performance of large pretrained multilingual speech recognition models on a regional variety of French spoken in Qu\'ebec, Canada, in terms of speed, word error rate and semantic accuracy. To this end we build a benchmark and evaluation pipeline based on the CommissionsQc datasets, a corpus of spontaneous conversations recorded during public inquiries recently held in Qu\'ebec. Published results for these models on well-known benchmarks such as FLEURS or CommonVoice are not good predictors of the performance we observe on CommissionsQC. Our results should be of interest for practitioners interested in building speech applications for realistic conditions or regional language varieties.