Enhancing Robustness in Large Language Models: Prompting for Mitigating the Impact of Irrelevant Information

TL;DR

This paper introduces a new method called ATF that improves large language models' ability to identify and mitigate irrelevant information, thereby enhancing their reasoning performance on complex tasks with distracting data.

Contribution

The paper proposes the ATF automatic construction method to help LLMs better handle irrelevant information, addressing a key limitation in current prompting techniques.

Findings

ATF significantly improves LLM reasoning accuracy with irrelevant info

LLMs can identify irrelevant info but struggle to mitigate its impact

Experimental results show enhanced performance on GSMIR dataset

Abstract

In recent years, Large language models (LLMs) have garnered significant attention due to their superior performance in complex reasoning tasks. However, recent studies may diminish their reasoning capabilities markedly when problem descriptions contain irrelevant information, even with the use of advanced prompting techniques. To further investigate this issue, a dataset of primary school mathematics problems containing irrelevant information, named GSMIR, was constructed. Testing prominent LLMs and prompting techniques on this dataset revealed that while LLMs can identify irrelevant information, they do not effectively mitigate the interference it causes once identified. A novel automatic construction method, ATF, which enhances the ability of LLMs to identify and self-mitigate the influence of irrelevant information, is proposed to address this shortcoming. This method operates in two steps: first, analysis of irrelevant information, followed by its filtering. The ATF method, as demonstrated by experimental results, significantly improves the reasoning performance of LLMs and prompting techniques, even in the presence of irrelevant information on the GSMIR dataset.