Alignment Helps Make the Most of Multimodal Data

TL;DR

This paper emphasizes the importance of aligning multimodal data in political science research, introduces a decision tree framework to guide alignment choices, and demonstrates its value through practical applications.

Contribution

It introduces a systematic decision tree framework for aligning multimodal data, addressing a gap in current political science research practices.

Findings

Most political science studies do not align multimodal data.

The decision tree improves research design and modeling decisions.

Alignment enhances analysis in campaign ads and parliamentary speech studies.

Abstract

Political scientists increasingly analyze multimodal data. However, the effective analysis of such data requires aligning information across different modalities. In our paper, we demonstrate the significance of such alignment. Informed by a systematic review of 2,703 papers, we find that political scientists typically do not align their multimodal data. Introducing a decision tree that guides alignment choices, our framework highlights alignment's untapped potential and provides concrete advice in research design and modeling decisions. We illustrate alignment's analytical value through two applications: predicting tonality in U.S. presidential campaign ads and cross-modal querying of German parliamentary speeches to examine responses to the far-right AfD.