TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection

TL;DR

TaskCLIP introduces a two-stage, vision-language model-based approach for task-oriented object detection, leveraging large VLMs and a transformer aligner to improve accuracy and generalizability over existing methods.

Contribution

The paper presents a novel two-stage framework using large vision-language models and a transformer aligner for improved task-oriented object detection.

Findings

Outperforms state-of-the-art DETR-based model TOIST by 3.5%

Requires only a single NVIDIA RTX 4090 for training and inference

Effectively aligns embeddings for better object selection

Abstract

Task-oriented object detection aims to find objects suitable for accomplishing specific tasks. As a challenging task, it requires simultaneous visual data processing and reasoning under ambiguous semantics. Recent solutions are mainly all-in-one models. However, the object detection backbones are pre-trained without text supervision. Thus, to incorporate task requirements, their intricate models undergo extensive learning on a highly imbalanced and scarce dataset, resulting in capped performance, laborious training, and poor generalizability. In contrast, we propose TaskCLIP, a more natural two-stage design composed of general object detection and task-guided object selection. Particularly for the latter, we resort to the recently successful large Vision-Language Models (VLMs) as our backbone, which provides rich semantic knowledge and a uniform embedding space for images and texts. Nevertheless, the naive application of VLMs leads to sub-optimal quality, due to the misalignment between embeddings of object images and their visual attributes, which are mainly adjective phrases. To this end, we design a transformer-based aligner after the pre-trained VLMs to re-calibrate both embeddings. Finally, we employ a trainable score function to post-process the VLM matching results for object selection. Experimental results demonstrate that our TaskCLIP outperforms the state-of-the-art DETR-based model TOIST by 3.5% and only requires a single NVIDIA RTX 4090 for both training and inference.