Can Vision Language Models Understand Mimed Actions?

TL;DR

This paper introduces MIME, a new benchmark for evaluating vision-language models' understanding of mimed actions, revealing current models' significant performance gap compared to humans and highlighting the need for improved gesture recognition capabilities.

Contribution

The paper presents MIME, a novel video-based question answering benchmark for mimed actions, with diverse variations and perturbations to assess recognition robustness in vision-language models.

Findings

Models perform worse than humans on MIME.

Open-weight and API-based models show significant recognition gaps.

MIME highlights the need for more robust gesture understanding in models.

Abstract

Nonverbal communication (NVC) plays an integral role in human language, but studying NVC in general is challenging because of its broad scope and high variance in interpretation among individuals and cultures. However, mime -- the theatrical technique of suggesting intent using only gesture, expression, and movement -- is a subset of NVC that consists of explicit and embodied actions with much lower human interpretation variance. We argue that a solid understanding of mimed actions is a crucial prerequisite for vision-language models capable of interpreting and commanding more subtle aspects of NVC. Hence, we propose Mime Identification Multimodal Evaluation (MIME), a novel video-based question answering benchmark comprising of 86 mimed actions. Constructed with motion capture data, MIME consists of variations of each action with perturbations applied to the character, background, and viewpoint for evaluating recognition robustness. We find that both open-weight and API-based vision-language models perform significantly worse than humans on MIME, motivating the need for increased research for instilling more robust understanding of human gestures.