# To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks

## TL;DR

This paper investigates whether to fine-tune or use feature extraction when adapting pretrained NLP models to new tasks, showing that the best approach depends on task similarity.

## Contribution

It provides empirical insights and guidelines on choosing between fine-tuning and feature extraction based on task similarity for NLP transfer learning.

## Key findings

- Fine-tuning outperforms feature extraction on similar tasks.
- Feature extraction is preferable for dissimilar tasks.
- Guidelines help practitioners select adaptation methods.

## Abstract

While most previous work has focused on different pretraining objectives and architectures for transfer learning, we ask how to best adapt the pretrained model to a given target task. We focus on the two most common forms of adaptation, feature extraction (where the pretrained weights are frozen), and directly fine-tuning the pretrained model. Our empirical results across diverse NLP tasks with two state-of-the-art models show that the relative performance of fine-tuning vs. feature extraction depends on the similarity of the pretraining and target tasks. We explore possible explanations for this finding and provide a set of adaptation guidelines for the NLP practitioner.