TurQUaz at CheckThat! 2025: Debating Large Language Models for Scientific Web Discourse Detection

TL;DR

This paper introduces a novel council debate approach among large language models for scientific web discourse detection, focusing on identifying scientific claims, references, and entities in tweets, with the council debate method showing superior performance in detecting scientific study references.

Contribution

The paper proposes a new council debate method among LLMs for scientific discourse detection, demonstrating its effectiveness in identifying references to scientific studies.

Findings

Outperforms other debate methods in detecting scientific study references

Ranks first in identifying references to scientific studies

Has lower performance in detecting scientific claims and entities

Abstract

In this paper, we present our work developed for the scientific web discourse detection task (Task 4a) of CheckThat! 2025. We propose a novel council debate method that simulates structured academic discussions among multiple large language models (LLMs) to identify whether a given tweet contains (i) a scientific claim, (ii) a reference to a scientific study, or (iii) mentions of scientific entities. We explore three debating methods: i) single debate, where two LLMs argue for opposing positions while a third acts as a judge; ii) team debate, in which multiple models collaborate within each side of the debate; and iii) council debate, where multiple expert models deliberate together to reach a consensus, moderated by a chairperson model. We choose council debate as our primary model as it outperforms others in the development test set. Although our proposed method did not rank highly for identifying scientific claims (8th out of 10) or mentions of scientific entities (9th out of 10), it ranked first in detecting references to scientific studies.