Clash of the models: Comparing performance of BERT-based variants for generic news frame detection

TL;DR

This paper compares five BERT-based models for generic news frame detection, introduces fine-tuned models, and provides a Swiss electoral dataset to evaluate their performance in political communication analysis.

Contribution

It offers a comparative analysis of BERT variants, introduces robust fine-tuned models, and supplies a new Swiss electoral news framing dataset.

Findings

BERT variants outperform bag-of-words models in frame detection.

Fine-tuned models show high robustness across contexts.

Swiss dataset enables testing of model generalizability.

Abstract

Framing continues to remain one of the most extensively applied theories in political communication. Developments in computation, particularly with the introduction of transformer architecture and more so with large language models (LLMs), have naturally prompted scholars to explore various novel computational approaches, especially for deductive frame detection, in recent years. While many studies have shown that different transformer models outperform their preceding models that use bag-of-words features, the debate continues to evolve regarding how these models compare with each other on classification tasks. By placing itself at this juncture, this study makes three key contributions: First, it comparatively performs generic news frame detection and compares the performance of five BERT-based variants (BERT, RoBERTa, DeBERTa, DistilBERT and ALBERT) to add to the debate on best practices around employing computational text analysis for political communication studies. Second, it introduces various fine-tuned models capable of robustly performing generic news frame detection. Third, building upon numerous previous studies that work with US-centric data, this study provides the scholarly community with a labelled generic news frames dataset based on the Swiss electoral context that aids in testing the contextual robustness of these computational approaches to framing analysis.