Action Without Interaction: Probing the Physical Foundations of Video LMMs via Contact-Release Detection

TL;DR

This paper investigates whether large multi-modal models truly understand physical interactions in videos by testing their ability to detect contact and release events, revealing a gap between semantic recognition and physical grounding.

Contribution

Introduces a novel large-scale dataset with annotated contact-release events and evaluates state-of-the-art models' physical reasoning capabilities in videos.

Findings

Models reliably identify objects and actions but fail to locate interaction start/end frames.

Models exhibit shortcut learning, recognizing patterns without understanding physical primitives.

There is a disconnect between semantic success and physical grounding in current LMMs.

Abstract

Large multi-modal models (LMMs) show increasing performance in realistic visual tasks for images and, more recently, for videos. For example, given a video sequence, such models are able to describe in detail objects, the surroundings and dynamic actions. In this study, we explored the extent to which these models ground their semantic understanding in the actual visual input. Specifically, given sequences of hands interacting with objects, we asked models when and where the interaction begins or ends. For this purpose, we introduce a first of its kind, large-scale dataset with more than 20K annotated interactions on videos from the Something-Something-V2 dataset. 250 AMTurk human annotators labeled core interaction events, particularly when and where objects and agents become attached (`contact') or detached (`release'). We asked SoTA LMMs, including GPT, Gemini and Qwen to locate these events in short videos, each with a single event. The results show that while models reliably name target objects and identify actions, they exhibit a form of `shortcut learning' where semantic success masks a failure in physical grounding. Specifically, they consistently fail to identify the frame where the interaction begins or ends and poorly localize the physical event within the scene. This disconnect suggests that while LMMs excel at System 1 intuitive pattern recognition (naming the action and objects), they lack the System 2 cognitive foundations required to reason about physical primitives like `contact' and `release', hence truly ground dynamic scenes in physical reality.