To Adapt or to Fine-tune: A Case Study on Abstractive Summarization

TL;DR

This paper compares fine-tuning and adapter-based methods for abstractive summarization, finding fine-tuning generally performs better with more data, but adapters excel in low-resource scenarios, providing insights into their practical use.

Contribution

It offers a comprehensive investigation into the effectiveness of fine-tuning versus adapters for summarization across various transfer scenarios, highlighting conditions favoring each approach.

Findings

Fine-tuning outperforms adapters with sufficient data.

Adapters are more effective in extremely low-resource settings.

Insights on multilinguality, convergence, and robustness are provided.

Abstract

Recent advances in the field of abstractive summarization leverage pre-trained language models rather than train a model from scratch. However, such models are sluggish to train and accompanied by a massive overhead. Researchers have proposed a few lightweight alternatives such as smaller adapters to mitigate the drawbacks. Nonetheless, it remains uncertain whether using adapters benefits the task of summarization, in terms of improved efficiency without an unpleasant sacrifice in performance. In this work, we carry out multifaceted investigations on fine-tuning and adapters for summarization tasks with varying complexity: language, domain, and task transfer. In our experiments, fine-tuning a pre-trained language model generally attains a better performance than using adapters; the performance gap positively correlates with the amount of training data used. Notably, adapters exceed fine-tuning under extremely low-resource conditions. We further provide insights on multilinguality, model convergence, and robustness, hoping to shed light on the pragmatic choice of fine-tuning or adapters in abstractive summarization.