Benchmarking Critical Questions Generation: A Challenging Reasoning Task for Large Language Models

TL;DR

This paper introduces a large-scale dataset and evaluation methods for Critical Questions Generation, a task that enhances reasoning by generating questions that challenge assumptions, with benchmarks for large language models.

Contribution

It provides the first extensive dataset and evaluation framework for CQs-Gen, enabling systematic benchmarking of models on this reasoning task.

Findings

Automatic evaluation correlates well with human judgments.

Zero-shot LLM performance highlights the task's difficulty.

Benchmark results establish a baseline for future research.

Abstract

The task of Critical Questions Generation (CQs-Gen) aims to foster critical thinking by enabling systems to generate questions that expose underlying assumptions and challenge the validity of argumentative reasoning structures. Despite growing interest in this area, progress has been hindered by the lack of suitable datasets and automatic evaluation standards. This paper presents a comprehensive approach to support the development and benchmarking of systems for this task. We construct the first large-scale dataset including ~5K manually annotated questions. We also investigate automatic evaluation methods and propose reference-based techniques as the strategy that best correlates with human judgments. Our zero-shot evaluation of 11 LLMs establishes a strong baseline while showcasing the difficulty of the task. Data and code plus a public leaderboard are provided to encourage further research, not only in terms of model performance, but also to explore the practical benefits of CQs-Gen for both automated reasoning and human critical thinking.