Piecing Together Clues: A Benchmark for Evaluating the Detective Skills of Large Language Models

TL;DR

This paper introduces DetectBench, a benchmark dataset for evaluating large language models' abilities in information detection and multi-hop reasoning, and proposes a framework to improve their detective skills.

Contribution

The paper presents a new dataset and a reasoning framework to better assess and enhance LLMs' detective-like information processing capabilities.

Findings

Existing models perform poorly on detection and reasoning tasks.

The Detective Thinking Framework improves model performance.

DetectBench provides a challenging benchmark for complex reasoning.

Abstract

Detectives frequently engage in information detection and reasoning simultaneously when making decisions across various cases, especially when confronted with a vast amount of information. With the rapid development of large language models~(LLMs), evaluating how these models identify key information and reason to solve questions becomes increasingly relevant. We introduces the DetectBench, a reading comprehension dataset designed to assess a model's ability to jointly ability in key information detection and multi-hop reasoning when facing complex and implicit information. The DetectBench comprises 3,928 questions, each paired with a paragraph averaging 190 tokens in length. To enhance model's detective skills, we propose the Detective Thinking Framework. These methods encourage models to identify all possible clues within the context before reasoning. Our experiments reveal that existing models perform poorly in both information detection and multi-hop reasoning. However, the Detective Thinking Framework approach alleviates this issue.