Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMs

TL;DR

This paper introduces PAPEFT, a novel fine-tuning method that reduces positional bias in large language models, thereby improving their performance on long-context tasks involving external knowledge retrieval.

Contribution

The study presents a new position-aware fine-tuning approach combining data augmentation and adapters to mitigate positional bias in LLMs, which is more effective than prompt-based methods.

Findings

PAPEFT reduces positional bias in LLMs.

Improves performance on long-context tasks.

Enhances knowledge retrieval accuracy.

Abstract

Recent advances in large language models (LLMs) have enhanced their ability to process long input contexts. This development is particularly crucial for tasks that involve retrieving knowledge from an external datastore, which can result in long inputs. However, recent studies show a positional bias in LLMs, demonstrating varying performance depending on the location of useful information within the input sequence. In this study, we conduct extensive experiments to investigate the root causes of positional bias. Our findings indicate that the primary contributor to LLM positional bias stems from the inherent positional preferences of different models. We demonstrate that merely employing prompt-based solutions is inadequate for overcoming the positional preferences. To address this positional bias issue of a pre-trained LLM, we developed a Position-Aware Parameter Efficient Fine-Tuning (PAPEFT) approach which is composed of a data augmentation technique and a parameter efficient adapter, enhancing a uniform attention distribution across the input context. Our experiments demonstrate that the proposed approach effectively reduces positional bias, improving LLMs' effectiveness in handling long context sequences for various tasks that require externally retrieved knowledge.