Pantagruel: Unified Self-Supervised Encoders for French Text and Speech

TL;DR

Pantagruel introduces a unified self-supervised encoder for French text and speech that learns in the feature space, enabling effective multimodal representation and outperforming existing models on various downstream tasks.

Contribution

The paper presents a new family of self-supervised models for French text and speech that learn in the feature space, with a shared architecture for both modalities, and introduces a large diverse speech corpus.

Findings

Pantagruel models outperform strong French baselines on multiple benchmarks.

Unified architecture effectively handles both speech and text inputs.

Feature-space self-supervised learning proves effective for French multimodal tasks.

Abstract

We release Pantagruel models, a new family of self-supervised encoder models for French text and speech. Instead of predicting modality-tailored targets such as textual tokens or speech units, Pantagruel learns contextualized target representations in the feature space, allowing modality-specific encoders to capture linguistic and acoustic regularities more effectively. Separate models are pre-trained on large-scale French corpora, including Wikipedia, OSCAR and CroissantLLM for text, together with MultilingualLibriSpeech, LeBenchmark, and INA-100k for speech. INA-100k is a newly introduced 100,000-hour corpus of French audio derived from the archives of the Institut National de l'Audiovisuel (INA), the national repository of French radio and television broadcasts, providing highly diverse audio data. We evaluate Pantagruel across a broad range of downstream tasks spanning both modalities, including those from the standard French benchmarks such as FLUE or LeBenchmark. Across these tasks, Pantagruel models show competitive or superior performance compared to strong French baselines such as CamemBERT, FlauBERT, and LeBenchmark2.0, while maintaining a shared architecture that can seamlessly handle either speech or text inputs. These results confirm the effectiveness of feature-space self-supervised objectives for French representation learning and highlight Pantagruel as a robust foundation for multimodal speech-text understanding.