What can human minimal videos tell us about dynamic recognition models?

TL;DR

This paper investigates how humans recognize objects and actions from minimal videos combining spatial and temporal cues, revealing gaps in current deep models' ability to replicate human dynamic vision.

Contribution

It introduces the concept of minimal videos to analyze human recognition and highlights the limitations of current deep networks in these scenarios.

Findings

Humans can recognize objects and actions from minimal videos.

Deep networks fail to replicate human recognition in these minimal configurations.

Current models lack mechanisms for integrating spatial and temporal cues effectively.

Abstract

In human vision objects and their parts can be visually recognized from purely spatial or purely temporal information but the mechanisms integrating space and time are poorly understood. Here we show that human visual recognition of objects and actions can be achieved by efficiently combining spatial and motion cues in configurations where each source on its own is insufficient for recognition. This analysis is obtained by identifying minimal videos: these are short and tiny video clips in which objects, parts, and actions can be reliably recognized, but any reduction in either space or time makes them unrecognizable. State-of-the-art deep networks for dynamic visual recognition cannot replicate human behavior in these configurations. This gap between humans and machines points to critical mechanisms in human dynamic vision that are lacking in current models.