On the Usability of Transformers-based models for a French Question-Answering task

TL;DR

This paper evaluates the usability of Transformer-based models for French question-answering tasks, focusing on resource efficiency, data scarcity challenges, and proposing a new compact model for low-resource French NLP applications.

Contribution

It provides a comprehensive assessment of Transformer models' performance on French QA and introduces a new compact French model optimized for low-resource scenarios.

Findings

Data augmentation improves French QA performance.

Hyperparameter tuning enhances model stability.

The new FrALBERT model is competitive in low-resource settings.

Abstract

For many tasks, state-of-the-art results have been achieved with Transformer-based architectures, resulting in a paradigmatic shift in practices from the use of task-specific architectures to the fine-tuning of pre-trained language models. The ongoing trend consists in training models with an ever-increasing amount of data and parameters, which requires considerable resources. It leads to a strong search to improve resource efficiency based on algorithmic and hardware improvements evaluated only for English. This raises questions about their usability when applied to small-scale learning problems, for which a limited amount of training data is available, especially for under-resourced languages tasks. The lack of appropriately sized corpora is a hindrance to applying data-driven and transfer learning-based approaches with strong instability cases. In this paper, we establish a state-of-the-art of the efforts dedicated to the usability of Transformer-based models and propose to evaluate these improvements on the question-answering performances of French language which have few resources. We address the instability relating to data scarcity by investigating various training strategies with data augmentation, hyperparameters optimization and cross-lingual transfer. We also introduce a new compact model for French FrALBERT which proves to be competitive in low-resource settings.