Making Long-Context Language Models Better Multi-Hop Reasoners

TL;DR

This paper introduces a novel attribution-based prompting method to improve multi-hop reasoning in long-context language models, demonstrating enhanced accuracy and robustness across multiple datasets and models.

Contribution

The paper proposes Reasoning with Attributions, a new prompting approach, along with an attribution-annotated dataset and training strategy to boost multi-hop reasoning in language models.

Findings

The approach improves reasoning accuracy on multiple datasets.

Fine-tuned models achieve performance close to proprietary LMs.

Attribution prompts increase model robustness to noisy contexts.

Abstract

Recent advancements in long-context modeling have enhanced language models (LMs) for complex tasks across multiple NLP applications. Despite this progress, we find that these models struggle with multi-hop reasoning and exhibit decreased performance in the presence of noisy contexts. In this paper, we introduce Reasoning with Attributions, a novel approach that prompts LMs to supply attributions for each assertion during their reasoning. We validate our approach through experiments on three multi-hop datasets, employing both proprietary and open-source models, and demonstrate its efficacy and resilience. Furthermore, we explore methods to augment reasoning capabilities via fine-tuning and offer an attribution-annotated dataset and a specialized training strategy. Our fine-tuned model achieves competitive performance on multi-hop reasoning benchmarks, closely paralleling proprietary LMs such as ChatGPT and Claude-instant.