A Multi-Encoder Frozen-Decoder Approach for Fine-Tuning Large Language Models

TL;DR

This paper explores freezing the decoder in large language models during multi-task fine-tuning, showing it improves efficiency and performance, especially for natural language output tasks and multilingual settings.

Contribution

It demonstrates the effectiveness of decoder freezing in multi-task fine-tuning, expanding its applicability to structured and QA tasks with larger models.

Findings

Freezing decoders benefits natural language output tasks.

Mitigates catastrophic forgetting in multilingual tasks.

Pairing frozen decoders with larger models maintains or improves performance.

Abstract

Among parameter-efficient fine-tuning methods, freezing has emerged as a popular strategy for speeding up training, reducing catastrophic forgetting, and improving downstream performance. We investigate the impact of freezing the decoder in a multi-task setup comprising diverse natural language tasks, aiming to reduce deployment overhead and enhance portability to novel tasks. Our experiments, conducted by fine-tuning both individual and multi-task setups on the AlexaTM model, reveal that freezing decoders is highly effective for tasks with natural language outputs and mitigates catastrophic forgetting in multilingual tasks. However, we find that pairing frozen decoders with a larger model can effectively maintain or even enhance performance in structured and QA tasks, making it a viable strategy for a broader range of task types.