Hansel: Output Length Controlling Framework for Large Language Models

TL;DR

Hansel is a versatile framework that enables precise length control in large language models during finetuning, improving accuracy and extrapolation without sacrificing text quality.

Contribution

It introduces a simple, effective method using hidden tokens for length control applicable to any pre-trained LLMs during finetuning, enhancing output length accuracy and generalization.

Findings

Significant reduction in mean absolute error of output length across models and datasets.

Improved ability to extrapolate to unseen target lengths.

Maintains coherency and fluency of generated text.

Abstract

Despite the great success of large language models (LLMs), efficiently controlling the length of the output sequence still remains a challenge. In this paper, we propose Hansel, an efficient framework for length control in LLMs without affecting its generation ability. Hansel utilizes periodically outputted hidden special tokens to keep track of the remaining target length of the output sequence. Together with techniques to avoid abrupt termination of the output, this seemingly simple method proved to be efficient and versatile, while not harming the coherency and fluency of the generated text. The framework can be applied to any pre-trained LLMs during the finetuning stage of the model, regardless of its original positional encoding method. We demonstrate this by finetuning four different LLMs with Hansel and show that the mean absolute error of the output sequence decreases significantly in every model and dataset compared to the prompt-based length control finetuning. Moreover, the framework showed a substantially improved ability to extrapolate to target lengths unseen during finetuning, such as long dialog responses or extremely short summaries. This indicates that the model learns the general means of length control, rather than learning to match output lengths to those seen during training.