ART: Adaptive Relation Tuning for Generalized Relation Prediction

TL;DR

ART introduces an instruction tuning framework with adaptive sampling to enhance visual relation detection, enabling models to generalize to unseen relations and improve scene understanding.

Contribution

The paper presents ART, a novel adaptive relation tuning method that leverages instruction tuning and strategic sampling to improve VRD generalization and unseen relation inference.

Findings

Significant improvement over baselines in relation prediction accuracy.

Ability to infer unseen relation concepts.

Enhanced scene segmentation using predicted relations.

Abstract

Visual relation detection (VRD) is the task of identifying the relationships between objects in a scene. VRD models trained solely on relation detection data struggle to generalize beyond the relations on which they are trained. While prompt tuning has been used to adapt vision-language models (VLMs) for VRD, it uses handcrafted prompts and struggles with novel or complex relations. We argue that instruction tuning offers a more effective solution by fine-tuning VLMs on diverse instructional data. We thus introduce ART, an Adaptive Relation Tuning framework that adapts VLMs for VRD through instruction tuning and strategic instance selection. By converting VRD datasets into an instruction tuning format and employing an adaptive sampling algorithm, ART directs the VLM to focus on informative relations while maintaining generalizability. Specifically, we focus on the relation classification, where subject-object boxes are given and the model predicts the predicate between them. We tune on a held-in set and evaluate across multiple held-out datasets of varying complexity. Our approach strongly improves over its baselines and can infer unseen relation concepts, a capability absent in mainstream VRD methods. We demonstrate ART's practical value by using the predicted relations for segmenting complex scenes.