Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception

TL;DR

This paper introduces the Motion Perceiver, a neural network that generalizes biological motion perception to unseen stimuli, outperforming existing AI models and aligning with human recognition patterns.

Contribution

The paper presents the Motion Perceiver, a novel AI model that effectively generalizes biological motion perception across diverse stimuli, surpassing prior models and aligning with human perception.

Findings

MP outperforms all existing AI models by up to 29% in accuracy.

MP generalizes well across 62,656 stimuli in neuroscience tests.

Psychophysics experiments show MP's recognition aligns with human behavior.

Abstract

Biological motion perception (BMP) refers to humans' ability to perceive and recognize the actions of living beings solely from their motion patterns, sometimes as minimal as those depicted on point-light displays. While humans excel at these tasks without any prior training, current AI models struggle with poor generalization performance. To close this research gap, we propose the Motion Perceiver (MP). MP solely relies on patch-level optical flows from video clips as inputs. During training, it learns prototypical flow snapshots through a competitive binding mechanism and integrates invariant motion representations to predict action labels for the given video. During inference, we evaluate the generalization ability of all AI models and humans on 62,656 video stimuli spanning 24 BMP conditions using point-light displays in neuroscience. Remarkably, MP outperforms all existing AI models with a maximum improvement of 29% in top-1 action recognition accuracy on these conditions. Moreover, we benchmark all AI models in point-light displays of two standard video datasets in computer vision. MP also demonstrates superior performance in these cases. More interestingly, via psychophysics experiments, we found that MP recognizes biological movements in a way that aligns with human behaviors. Our data and code are available at https://github.com/ZhangLab-DeepNeuroCogLab/MotionPerceiver.