Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction

TL;DR

This paper investigates positional bias in language models, revealing that models struggle to extract information from later parts of documents due to auto-regressive training, and proposes regularization methods to improve extraction across positions.

Contribution

It uncovers the positional bias in LLMs' knowledge extraction and demonstrates how regularization can mitigate this issue, advancing understanding of model training effects.

Findings

Models answer first sentence questions accurately.

Extraction from middle/end of documents is challenging.

Regularization improves extraction from diverse positions.

Abstract

Large language models require updates to remain up-to-date or adapt to new domains by fine-tuning them with new documents. One key is memorizing the latest information in a way that the memorized information is extractable with a query prompt. However, LLMs suffer from a phenomenon called perplexity curse; despite minimizing document perplexity during fine-tuning, LLMs struggle to extract information through a prompt sentence. In this new knowledge acquisition and extraction, we find a very intriguing fact that LLMs can accurately answer questions about the first sentence, but they struggle to extract information described in the middle or end of the documents used for fine-tuning. Our study suggests that the auto-regressive training causes this issue; each token is prompted by reliance on all previous tokens, which hinders the model from recalling information from training documents by question prompts. To conduct the in-depth study, we publish both synthetic and real datasets, enabling the evaluation of the QA performance w.r.t. the position of the corresponding answer in a document. Our investigation shows that even a large model suffers from the perplexity curse, but regularization such as denoising auto-regressive loss can enhance the information extraction from diverse positions. These findings will be (i) a key to improving knowledge extraction from LLMs and (ii) new elements to discuss the trade-off between RAG and fine-tuning in adapting LLMs to a new domain.