PositionID: LLMs can Control Lengths, Copy and Paste with Explicit Positional Awareness

TL;DR

This paper introduces PositionID techniques that enhance LLMs' ability to control text length and perform copy-paste operations by incorporating explicit positional awareness, supported by new benchmarks and experimental validation.

Contribution

It proposes novel PositionID Prompting and Fine-Tuning methods to improve length control and copy-paste accuracy in LLMs, addressing a key limitation due to lack of positional awareness.

Findings

Significant improvement in length adherence during text generation

Enhanced accuracy in copy-paste tasks

Maintained response quality with new methods

Abstract

Large Language Models (LLMs) demonstrate impressive capabilities across various domains, including role-playing, creative writing, mathematical reasoning, and coding. Despite these advancements, LLMs still encounter challenges with length control, frequently failing to adhere to specific length constraints due to their token-level operations and insufficient training on data with strict length limitations. We identify this issue as stemming from a lack of positional awareness and propose novel approaches--PositionID Prompting and PositionID Fine-Tuning--to address it. These methods enhance the model's ability to continuously monitor and manage text length during generation. Additionally, we introduce PositionID CP Prompting to enable LLMs to perform copy and paste operations accurately. Furthermore, we develop two benchmarks for evaluating length control and copy-paste abilities. Our experiments demonstrate that our methods significantly improve the model's adherence to length constraints and copy-paste accuracy without compromising response quality.