Language Model Knowledge Distillation for Efficient Question Answering in Spanish

TL;DR

This paper introduces SpanishTinyRoBERTa, a compressed Spanish language model created through knowledge distillation, enabling efficient question answering with minimal performance loss, suitable for resource-limited environments.

Contribution

The paper presents a novel distilled Spanish language model based on RoBERTa, optimized for question answering and resource efficiency.

Findings

The distilled model maintains performance comparable to larger models.

Inference speed is significantly increased with minimal accuracy loss.

The approach facilitates NLP tasks in resource-constrained settings.

Abstract

Recent advances in the development of pre-trained Spanish language models has led to significant progress in many Natural Language Processing (NLP) tasks, such as question answering. However, the lack of efficient models imposes a barrier for the adoption of such models in resource-constrained environments. Therefore, smaller distilled models for the Spanish language could be proven to be highly scalable and facilitate their further adoption on a variety of tasks and scenarios. In this work, we take one step in this direction by developing SpanishTinyRoBERTa, a compressed language model based on RoBERTa for efficient question answering in Spanish. To achieve this, we employ knowledge distillation from a large model onto a lighter model that allows for a wider implementation, even in areas with limited computational resources, whilst attaining negligible performance sacrifice. Our experiments show that the dense distilled model can still preserve the performance of its larger counterpart, while significantly increasing inference speedup. This work serves as a starting point for further research and investigation of model compression efforts for Spanish language models across various NLP tasks.