Unified Open-Vocabulary Dense Visual Prediction

TL;DR

This paper introduces a unified network for open-vocabulary dense visual prediction tasks, leveraging multi-modal data and a specialized training mechanism to improve performance across multiple dense prediction tasks.

Contribution

It proposes a novel unified network architecture with multi-modal, multi-scale, and multi-task decoding, and a training mechanism to bridge domain and task gaps, enabling joint training for multiple dense prediction tasks.

Findings

Effective on four datasets, outperforming task-specific models.

Leverages diverse training data to enhance individual task performance.

Addresses multi-modal data integration and domain gaps in unified models.

Abstract

In recent years, open-vocabulary (OV) dense visual prediction (such as OV object detection, semantic, instance and panoptic segmentations) has attracted increasing research attention. However, most of existing approaches are task-specific and individually tackle each task. In this paper, we propose a Unified Open-Vocabulary Network (UOVN) to jointly address four common dense prediction tasks. Compared with separate models, a unified network is more desirable for diverse industrial applications. Moreover, OV dense prediction training data is relatively less. Separate networks can only leverage task-relevant training data, while a unified approach can integrate diverse training data to boost individual tasks. We address two major challenges in unified OV prediction. Firstly, unlike unified methods for fixed-set predictions, OV networks are usually trained with multi-modal data. Therefore, we propose a multi-modal, multi-scale and multi-task (MMM) decoding mechanism to better leverage multi-modal data. Secondly, because UOVN uses data from different tasks for training, there are significant domain and task gaps. We present a UOVN training mechanism to reduce such gaps. Experiments on four datasets demonstrate the effectiveness of our UOVN.