Capturing Topic Framing via Masked Language Modeling

TL;DR

This paper introduces a scalable framework using masked language models to measure how different media outlets frame issues, capturing subtle language differences across polarized topics.

Contribution

It proposes a novel approach combining prompt design, normalization, and robustness analysis for modeling issue framing with large-scale language models.

Findings

Framework reliably captures differential framing across media outlets.

Effective in analyzing five politically polarized topics.

Demonstrates robustness to different pre-trained language models.

Abstract

Differential framing of issues can lead to divergent world views on important issues. This is especially true in domains where the information presented can reach a large audience, such as traditional and social media. Scalable and reliable measurement of such differential framing is an important first step in addressing them. In this work, based on the intuition that framing affects the tone and word choices in written language, we propose a framework for modeling the differential framing of issues through masked token prediction via large-scale fine-tuned language models (LMs). Specifically, we explore three key factors for our framework: 1) prompt generation methods for the masked token prediction; 2) methods for normalizing the output of fine-tuned LMs; 3) robustness to the choice of pre-trained LMs used for fine-tuning. Through experiments on a dataset of articles from traditional media outlets covering five diverse and politically polarized topics, we show that our framework can capture differential framing of these topics with high reliability.