Instruction Position Matters in Sequence Generation with Large Language Models

TL;DR

This paper shows that repositioning task instructions after input sentences in training data enhances large language models' ability to follow instructions, especially in long sequences, improving zero-shot translation and summarization performance.

Contribution

The paper introduces a simple method of shifting instructions after input sentences to improve instruction-following in LLMs, backed by theoretical analysis and extensive experiments.

Findings

Improved zero-shot translation performance, up to 9.7 BLEU points.

Consistent gains across model scales and tasks.

No additional data or annotation required.

Abstract

Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization, through instruction fine-tuning. The fine-tuning data is generally sequentially concatenated from a specific task instruction, an input sentence, and the corresponding response. Considering the locality modeled by the self-attention mechanism of LLMs, these models face the risk of instruction forgetting when generating responses for long input sentences. To mitigate this issue, we propose enhancing the instruction-following capability of LLMs by shifting the position of task instructions after the input sentences. Theoretical analysis suggests that our straightforward method can alter the model's learning focus, thereby emphasizing the training of instruction-following capabilities. Concurrently, experimental results demonstrate that our approach consistently outperforms traditional settings across various model scales (1B / 7B / 13B) and different sequence generation tasks (translation and summarization), without any additional data or annotation costs. Notably, our method significantly improves the zero-shot performance on conditional sequence generation, e.g., up to 9.7 BLEU points on WMT zero-shot translation tasks.