The Claire French Dialogue Dataset

TL;DR

The paper introduces the Claire French Dialogue Dataset (CFDD), a large, standardized corpus of French dialogue texts aimed at advancing multilingual open source language models.

Contribution

It compiles and standardizes 24 diverse French dialogue corpora into a unified dataset, facilitating research in multilingual language modeling.

Findings

Dataset contains approximately 160 million words.

CFDD is publicly available for research and development.

The dataset is organized into eight subcategories for targeted use.

Abstract

We present the Claire French Dialogue Dataset (CFDD), a resource created by members of LINAGORA Labs in the context of the OpenLLM France initiative. CFDD is a corpus containing roughly 160 million words from transcripts and stage plays in French that we have assembled and publicly released in an effort to further the development of multilingual, open source language models. This paper describes the 24 individual corpora of which CFDD is composed and provides links and citations to their original sources. It also provides our proposed breakdown of the full CFDD dataset into eight categories of subcorpora and describes the process we followed to standardize the format of the final dataset. We conclude with a discussion of similar work and future directions.