Critical Questions Generation: Motivation and Challenges

TL;DR

This paper introduces the task of Critical Questions Generation to identify missing information in arguments, proposing methods to create datasets for training LLMs, and analyzing their effectiveness in this new role.

Contribution

It presents a novel task of generating critical questions from arguments, and proposes two methods for creating datasets to train and evaluate LLMs on this task.

Findings

LLMs can reasonably generate critical questions but need further improvement.

A procedure to determine valid critical questions from arguments is established.

Using LLMs for CQs generation shows promise but has significant room for enhancement.

Abstract

The development of Large Language Models (LLMs) has brought impressive performances on mitigation strategies against misinformation, such as counterargument generation. However, LLMs are still seriously hindered by outdated knowledge and by their tendency to generate hallucinated content. In order to circumvent these issues, we propose a new task, namely, Critical Questions Generation, consisting of processing an argumentative text to generate the critical questions (CQs) raised by it. In argumentation theory CQs are tools designed to lay bare the blind spots of an argument by pointing at the information it could be missing. Thus, instead of trying to deploy LLMs to produce knowledgeable and relevant counterarguments, we use them to question arguments, without requiring any external knowledge. Research on CQs Generation using LLMs requires a reference dataset for large scale experimentation. Thus, in this work we investigate two complementary methods to create such a resource: (i) instantiating CQs templates as defined by Walton's argumentation theory and (ii), using LLMs as CQs generators. By doing so, we contribute with a procedure to establish what is a valid CQ and conclude that, while LLMs are reasonable CQ generators, they still have a wide margin for improvement in this task.