Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering
Praveen Venkateswaran, Danish Contractor

TL;DR
This paper introduces a dynamic attention steering method that allows users to emphasize specific parts of prompts during inference, improving instruction following in large language models without performance loss.
Contribution
It presents a novel inference-time technique for dynamically steering model attention to user-specified prompt parts, enhancing instruction adherence over static methods.
Findings
Improves instruction following across diverse tasks
Generalizes across different model scales
Does not degrade model performance
Abstract
In many real-world applications, users rely on natural language instructions to guide large language models (LLMs) across a wide range of tasks. These instructions are often complex, diverse, and subject to frequent change. However, LLMs do not always attend to these instructions reliably, and users lack simple mechanisms to emphasize their importance beyond modifying prompt wording or structure. To address this, we present an inference-time method that enables users to emphasize specific parts of their prompt by steering the model's attention toward them, aligning the model's perceived importance of different prompt tokens with user intent. Unlike prior approaches that are limited to static instructions, require significant offline profiling, or rely on fixed biases, we dynamically update the proportion of model attention given to the user-specified parts--ensuring improved instruction…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsMultimodal Machine Learning Applications · Topic Modeling · Natural Language Processing Techniques
