Fraunhofer SIT at CheckThat! 2023: Tackling Classification Uncertainty Using Model Souping on the Example of Check-Worthiness Classification

TL;DR

This paper presents a model soup ensemble approach for check-worthiness classification in political debates, achieving high accuracy and ranking second in the CLEF-2023 CheckThat! challenge.

Contribution

It introduces a novel ensemble method based on Model Souping for check-worthiness detection, improving performance over previous approaches.

Findings

Achieved an F1 score of 0.878 on the English dataset.

Ranked second in the CLEF-2023 CheckThat! competition.

Demonstrated effectiveness of Model Souping ensemble technique.

Abstract

This paper describes the second-placed approach developed by the Fraunhofer SIT team in the CLEF-2023 CheckThat! lab Task 1B for English. Given a text snippet from a political debate, the aim of this task is to determine whether it should be assessed for check-worthiness. Detecting check-worthy statements aims to facilitate manual fact-checking efforts by prioritizing the claims that fact-checkers should consider first. It can also be considered as primary step of a fact-checking system. Our best-performing method took advantage of an ensemble classification scheme centered on Model Souping. When applied to the English data set, our submitted model achieved an overall F1 score of 0.878 and was ranked as the second-best model in the competition.