Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization

TL;DR

This paper introduces Explanatory Instructions and a large dataset to enable vision-language models to understand and generalize across diverse computer vision tasks in a zero-shot setting, addressing a key barrier in CV.

Contribution

It proposes a novel approach using explanatory instructions and a large-scale dataset to improve zero-shot generalization in computer vision models.

Findings

Achieves instruction-level zero-shot capabilities for seen tasks

Demonstrates strong zero-shot generalization to unseen CV tasks

Provides a large dataset of 12 million triplets for training and evaluation

Abstract

Computer Vision (CV) has yet to fully achieve the zero-shot task generalization observed in Natural Language Processing (NLP), despite following many of the milestones established in NLP, such as large transformer models, extensive pre-training, and the auto-regression paradigm, among others. In this paper, we explore the idea that CV adopts discrete and terminological task definitions (\eg, ``image segmentation''), which may be a key barrier to zero-shot task generalization. Our hypothesis is that without truly understanding previously-seen tasks--due to these terminological definitions--deep models struggle to generalize to novel tasks. To verify this, we introduce Explanatory Instructions, which provide an intuitive way to define CV task objectives through detailed linguistic transformations from input images to outputs. We create a large-scale dataset comprising 12 million ``image input $\to$ explanatory instruction $\to$ output'' triplets, and train an auto-regressive-based vision-language model (AR-based VLM) that takes both images and explanatory instructions as input. By learning to follow these instructions, the AR-based VLM achieves instruction-level zero-shot capabilities for previously-seen tasks and demonstrates strong zero-shot generalization for unseen CV tasks. Code and dataset will be openly available on our GitHub repository.