I Spy With My Little Eye: A Minimum Cost Multicut Investigation of Dataset Frames

TL;DR

This paper presents a novel approach to visual framing analysis by formulating image clustering as a Minimum Cost Multicut Problem, leveraging different embedding spaces to improve automated detection of visual frames in social science datasets.

Contribution

It introduces the use of the Minimum Cost Multicut formulation for image clustering in visual frame analysis and compares embedding spaces for optimal clustering performance.

Findings

DINOv2 is effective for broad visual frames

ConvNeXt V2 detects fine-grain differences

Multicut clustering improves visual frame detection

Abstract

Visual framing analysis is a key method in social sciences for determining common themes and concepts in a given discourse. To reduce manual effort, image clustering can significantly speed up the annotation process. In this work, we phrase the clustering task as a Minimum Cost Multicut Problem [MP]. Solutions to the MP have been shown to provide clusterings that maximize the posterior probability, solely from provided local, pairwise probabilities of two images belonging to the same cluster. We discuss the efficacy of numerous embedding spaces to detect visual frames and show its superiority over other clustering methods. To this end, we employ the climate change dataset \textit{ClimateTV} which contains images commonly used for visual frame analysis. For broad visual frames, DINOv2 is a suitable embedding space, while ConvNeXt V2 returns a larger number of clusters which contain fine-grain differences, i.e. speech and protest. Our insights into embedding space differences in combination with the optimal clustering - by definition - advances automated visual frame detection. Our code can be found at https://github.com/KathPra/MP4VisualFrameDetection.