A Unified Sequence Interface for Vision Tasks

TL;DR

This paper introduces a unified sequence-based framework for multiple vision tasks, enabling a single model architecture to perform object detection, segmentation, keypoint detection, and captioning without task-specific modifications.

Contribution

The authors propose a shared pixel-to-sequence interface that unifies diverse vision tasks, allowing a single model to handle multiple tasks with prompts and sequence outputs.

Findings

Achieves competitive performance across tasks

Uses a single architecture and loss function

No task-specific customization needed

Abstract

While language tasks are naturally expressed in a single, unified, modeling framework, i.e., generating sequences of tokens, this has not been the case in computer vision. As a result, there is a proliferation of distinct architectures and loss functions for different vision tasks. In this work we show that a diverse set of "core" computer vision tasks can also be unified if formulated in terms of a shared pixel-to-sequence interface. We focus on four tasks, namely, object detection, instance segmentation, keypoint detection, and image captioning, all with diverse types of outputs, e.g., bounding boxes or dense masks. Despite that, by formulating the output of each task as a sequence of discrete tokens with a unified interface, we show that one can train a neural network with a single model architecture and loss function on all these tasks, with no task-specific customization. To solve a specific task, we use a short prompt as task description, and the sequence output adapts to the prompt so it can produce task-specific output. We show that such a model can achieve competitive performance compared to well-established task-specific models.