Scaling may be all you need for achieving human-level object recognition capacity with human-like visual experience

TL;DR

Scaling up data, model size, and image resolution simultaneously enables self-supervised vision models to reach human-level object recognition, demonstrating that human-like visual experience and simple architectures suffice for high-level perception.

Contribution

This work shows that large-scale self-supervised vision transformers trained on extensive human-like video data can achieve human-level object recognition without complex inductive biases.

Findings

Human-level accuracy achievable with 2.5B parameters trained on 20K hours of video

Scaling data, model size, and resolution together is effective

Masked autoencoders enable efficient large-scale training

Abstract

This paper asks whether current self-supervised learning methods, if sufficiently scaled up, would be able to reach human-level visual object recognition capabilities with the same type and amount of visual experience humans learn from. Previous work on this question only considered the scaling of data size. Here, we consider the simultaneous scaling of data size, model size, and image resolution. We perform a scaling experiment with vision transformers up to 633M parameters in size (ViT-H/14) trained with up to 5K hours of human-like video data (long, continuous, mostly egocentric videos) with image resolutions of up to 476x476 pixels. The efficiency of masked autoencoders (MAEs) as a self-supervised learning algorithm makes it possible to run this scaling experiment on an unassuming academic budget. We find that it is feasible to reach human-level object recognition capacity at sub-human scales of model size, data size, and image size, if these factors are scaled up simultaneously. To give a concrete example, we estimate that a 2.5B parameter ViT model trained with 20K hours (2.3 years) of human-like video data with a spatial resolution of 952x952 pixels should be able to reach roughly human-level accuracy on ImageNet. Human-level competence is thus achievable for a fundamental perceptual capability from human-like perceptual experience (human-like in both amount and type) with extremely generic learning algorithms and architectures and without any substantive inductive biases.