ELLIS Alicante at CQs-Gen 2025: Winning the critical thinking questions shared task: LLM-based question generation and selection

TL;DR

This paper presents a novel LLM-based system for automatic critical question generation and selection, aimed at fostering deeper reasoning and critical thinking in argumentative texts, achieving top performance in a shared task.

Contribution

It introduces a two-step framework with open source models for generating and selecting critical questions, advancing automatic critical question generation methods.

Findings

System ranked first in the shared task competition.

Demonstrated effectiveness in encouraging critical engagement.

Proved the viability of small-scale open source models for this task.

Abstract

The widespread adoption of chat interfaces based on Large Language Models (LLMs) raises concerns about promoting superficial learning and undermining the development of critical thinking skills. Instead of relying on LLMs purely for retrieving factual information, this work explores their potential to foster deeper reasoning by generating critical questions that challenge unsupported or vague claims in debate interventions. This study is part of a shared task of the 12th Workshop on Argument Mining, co-located with ACL 2025, focused on automatic critical question generation. We propose a two-step framework involving two small-scale open source language models: a Questioner that generates multiple candidate questions and a Judge that selects the most relevant ones. Our system ranked first in the shared task competition, demonstrating the potential of the proposed LLM-based approach to encourage critical engagement with argumentative texts.