Data-efficient Large Vision Models through Sequential Autoregression

TL;DR

This paper introduces an efficient autoregressive vision model that performs well on diverse visual tasks with fewer parameters and less training data, advancing sustainable generalist vision modeling.

Contribution

The paper presents a novel autoregressive vision model designed to operate effectively with limited data and reduced model size, improving versatility and efficiency.

Findings

Achieves high performance across multiple visual tasks

Reduces parameter count compared to large models

Requires significantly less training data

Abstract

Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding. These models are intended to not only comprehend but also seamlessly transit to out-of-domain tasks. However, current endeavors are hamstrung by an over-reliance on colossal models, exemplified by models with upwards of 3B parameters, and the necessity for an extensive corpus of visual data, often comprising a staggering 400B tokens. In this paper, we delve into the development of an efficient, autoregression-based vision model, innovatively architected to operate on a limited dataset. We meticulously demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding during the testing phase. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint, and a marked decrease in training data requirements, thereby paving the way for more sustainable and accessible advancements in the field of generalist vision models. The code is available at https://github.com/ggjy/DeLVM.