DetectBench: Can Large Language Model Detect and Piece Together Implicit Evidence?

TL;DR

This paper introduces DetectBench, a comprehensive benchmark for evaluating large language models' ability to detect and assemble implicit evidence in long contexts, and proposes methods to improve their performance.

Contribution

It presents DetectBench, a new benchmark with nearly 4,000 questions, and introduces Detective Reasoning Prompt and Finetuning techniques to enhance LLM evidence detection capabilities.

Findings

Existing LLMs perform poorly compared to humans in evidence detection.

Detective Reasoning Prompt improves detection in powerful LLMs.

Finetuning significantly boosts performance of weaker LLMs.

Abstract

Detecting evidence within the context is a key step in the process of reasoning task. Evaluating and enhancing the capabilities of LLMs in evidence detection will strengthen context-based reasoning performance. This paper proposes a benchmark called DetectBench for verifying the ability to detect and piece together implicit evidence within a long context. DetectBench contains 3,928 multiple-choice questions, with an average of 994 tokens per question. Each question contains an average of 4.55 pieces of implicit evidence, and solving the problem typically requires 7.62 logical jumps to find the correct answer. To enhance the performance of LLMs in evidence detection, this paper proposes Detective Reasoning Prompt and Finetune. Experiments demonstrate that the existing LLMs' abilities to detect evidence in long contexts are far inferior to humans. However, the Detective Reasoning Prompt effectively enhances the capability of powerful LLMs in evidence detection, while the Finetuning method shows significant effects in enhancing the performance of weaker LLMs. Moreover, when the abilities of LLMs in evidence detection are improved, their final reasoning performance is also enhanced accordingly.