Social Caption: Evaluating Social Understanding in Multimodal Models

TL;DR

This paper introduces Social Caption, a framework for evaluating social understanding in multimodal large language models across inference, analysis, and extraction tasks, providing insights into model performance factors.

Contribution

It presents a novel evaluation framework grounded in interaction theory, addressing the gap in assessing social understanding in multimodal models.

Findings

Model performance improves with scale and architectural enhancements.

Automated evaluation methods provide reliable insights into social understanding.

Factors like spoken context significantly influence model capabilities.

Abstract

Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce Social Caption, a framework grounded in interaction theory to evaluate social understanding abilities of MLLMs along three dimensions: Social Inference (SI), the ability to make accurate inferences about interactions; Holistic Social Analysis (HSA), the ability to generate comprehensive descriptions of interactions; Directed Social Analysis (DSA), the ability to extract relevant social information from interactions. We analyze factors influencing model performance in social understanding, such as scale, architectural design, and spoken context. Experiments with MLLM judges contribute insights about scaling automated evaluation of multimodal social understanding.