Annotation Tool and Dataset for Fact-Checking Podcasts

TL;DR

This paper introduces a real-time podcast annotation tool that combines transcription, crowdsourcing, and machine learning to facilitate fact-checking of multilingual spoken content, and releases a high-quality dataset for further research.

Contribution

It presents a novel real-time annotation system for podcasts, integrating transcription, crowdsourcing, and transformer models, and releases a new dataset for fact-checking tasks.

Findings

High-quality annotated podcast dataset created

Effective fine-tuning of multilingual models demonstrated

Preliminary experiments show promising fact-checking performance

Abstract

Podcasts are a popular medium on the web, featuring diverse and multilingual content that often includes unverified claims. Fact-checking podcasts is a challenging task, requiring transcription, annotation, and claim verification, all while preserving the contextual details of spoken content. Our tool offers a novel approach to tackle these challenges by enabling real-time annotation of podcasts during playback. This unique capability allows users to listen to the podcast and annotate key elements, such as check-worthy claims, claim spans, and contextual errors, simultaneously. By integrating advanced transcription models like OpenAI's Whisper and leveraging crowdsourced annotations, we create high-quality datasets to fine-tune multilingual transformer models such as XLM-RoBERTa for tasks like claim detection and stance classification. Furthermore, we release the annotated podcast transcripts and sample annotations with preliminary experiments.