Power-Law Decay Loss for Large Language Model Finetuning: A Theory Perspective

TL;DR

This paper proposes Power-Law Decay Loss (PDL), a novel loss function for finetuning large language models that re-weights tokens based on their frequency, emphasizing low-frequency, informative tokens to improve text generation quality.

Contribution

It introduces PDL, a theoretically motivated loss function that adjusts token weights according to a power-law decay to enhance finetuning outcomes for text generation tasks.

Findings

PDL improves diversity and informativeness in generated text.

Theoretically grounded in information theory and linguistics.

Applicable to various text generation tasks like summarization and dialogue.

Abstract

During the finetuning stage of text generation tasks, standard cross-entropy loss treats all tokens equally. This can lead models to overemphasize high-frequency, low-information tokens, neglecting lower-frequency tokens crucial for specificity and informativeness in generated content. This paper introduces a novel loss function, Power-Law Decay Loss (PDL), specifically designed to optimize the finetuning process for text generation. The core motivation for PDL stems from observations in information theory and linguistics: the informativeness of a token is often inversely proportional to its frequency of occurrence. PDL re-weights the contribution of each token in the standard cross-entropy loss based on its frequency in the training corpus, following a power-law decay. Specifically, the weights for high-frequency tokens are reduced, while low-frequency, information-dense tokens are assigned higher weights. This mechanism guides the model during finetuning to focus more on learning and generating tokens that convey specific and unique information, thereby enhancing the quality, diversity, and informativeness of the generated text. We theoretically elaborate on the motivation and construction of PDL and discuss its potential applications and advantages across various text generation finetuning tasks, such as abstractive summarization, dialogue systems, and style transfer.