Pay Attention to What Matters
Pedro Luiz Silva, Antonio de Domenico, Ali Maatouk, Fadhel Ayed
TL;DR
This paper introduces GUIDE, a simple method to enhance instruction-following in LLMs by increasing attention to instruction tokens, supported by a new influence metric that traces instruction propagation through transformer layers.
Contribution
The paper presents GUIDE, a novel technique to improve instruction alignment in LLMs, and Influence, a metric to analyze instruction propagation within transformer models.
Findings
GUIDE improves instruction-following accuracy by 29.4% to 60.4%.
GUIDE outperforms natural prompts and fine-tuning with up to 1 million tokens.
Influence effectively highlights how instructions propagate through transformer layers.
Abstract
Despite the remarkable success of Large Language Models (LLMs), they still exhibit a limited capability to align their outputs to the user instructions. In this work, we introduce a simple and effective method, which we name GUIDE, that mechanistically increases attention scores in instruction tokens. To support this operation, we present Influence, a novel metric that highlights how the user's instructions propagate through the transformer layers and impact the LLM output. Our results show that GUIDE improves the accuracy of following instructions 29.4 % to 60.4%, outperforming natural prompting alternatives and Supervised Fine-Tuning up to 1M tokens.
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
1) The presentation of the methods are fairly straightforward and clean. 2) The proposed method is very intuitive and simple. 3) Without any extra resources, a substantial improvement has been seen over all the selected datasets, making it potentially an appealing plug-and-play mechanism to be added to deployed model to highlight crucial part of the input.
Sec 3: 1) The concept of 'influence' as an enhancement to attention rollout is intriguing and initially seems valuable. However, the exact role of influence within the implementation remains somewhat ambiguous. At first the glance, the narrative suggests that influence is computed on a per-example basis to adjust the attention dynamically (delta). Yet, the inclusion of plots and tables depicting a consistent delta value across examples implies that this parameter could potentially be determined
(1) The proposed method is very simple and intuitive. The experiments clearly showed that the methods improved "instruction following" at a coarse granularity. Instruction following is an important topic in modern language model development and the proposed method could have a significant impact and adoption in real applications. (2) The proposed influence score is an intuitive way to guide the setting of the attention bias.
(1) Since this is a training-free method, it would be even better if the authors conducted the experiments on more models with different scales. Given that the experimented models are not very big (<7B) and strong (there are better models now like Llama-3), the lack of comprehensive experiments with different models may undermine the credibility of the experiments. (2) The tasks the authors selected are rather synthetic. Moreover, the metrics are too simple: for example, the correctness of the
- The paper is tackling an important problem in LLM. For instance, as the user prompt gets longer, we observe the LLM not following every part of the user instruction. A method that can enhance instruction following is a great contribution to the field. - The paper is clear and generally well written. - Influence score seems to give a good insight into evaluating the impact of tokens on the entire sequence.
- It is missing a very relevant work from ICLR 2024, "Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs" by Zhang et al. - While it is good to know that the llm is now producing French more often, but this may be at the cost of degraded output accuracy. I suspect that if the attention weights are manually changed, the quality of the outputs may be affected. I think it is very important to measure this as well. - I think Influence metric shouldn't be called "metric", because a
- This study introduces a prompt engineering method for large language models which increases attention scores in instruction tokens by simply enclosing important text within tags like <!-> <-!>. - A new metric is proposed to clarify that the proposed method focuses on the instruction text that are highlighted by the proposed method.
- There is already a prior research on prompt engineering method that shows that enclosing the tags improves performance, but this study does not describe the relationship. e.g. https://arxiv.org/abs/2309.13078 - Assessing the impact of attentions from the lens of vector norms is already a hot topic, but there is no description about the relationships. https://arxiv.org/pdf/2004.10102 - Lack of relationships between sections, mathematical symbols, and descriptions of key points in each section m
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Multimodal Machine Learning Applications