Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT

TL;DR

This paper investigates how increasing trainable parameters and their placement during PEFT improves language adaptation in smaller LLMs, especially for medium-resourced languages like Icelandic, while addressing challenges with longer context handling.

Contribution

It demonstrates that more trainable parameters and strategic placement, such as in feed-forward layers, enhance language adaptation without compromising context length capabilities.

Findings

More trainable parameters improve robustness in language adaptation.

LoRAs in feed-forward layers outperform other placements.

Adapting only final layers mitigates longer context issues.

Abstract

Smaller LLMs still face significant challenges even in medium-resourced languages, particularly when it comes to language-specific knowledge -- a problem not easily resolved with machine-translated data. In this case study on Icelandic, we aim to enhance the generation performance of an LLM by specialising it using unstructured text corpora. A key focus is on preventing interference with the models' capabilities of handling longer context during this adaptation. Through ablation studies using various parameter-efficient fine-tuning (PEFT) methods and setups, we find that increasing the number of trainable parameters leads to better and more robust language adaptation. LoRAs placed in the feed-forward layers and bottleneck adapters show promising results with sufficient parameters, while prefix tuning and (IA)3 are not suitable. Although improvements are consistent in 0-shot summarisation, some adapted models struggle with longer context lengths, an issue that can be mitigated by adapting only the final layers.