Unified Visual Relationship Detection with Vision and Language Models

TL;DR

This paper introduces UniVRD, a unified approach leveraging vision and language models to detect visual relationships across multiple datasets, improving performance and generalization in scene understanding tasks.

Contribution

The paper proposes a novel bottom-up method, UniVRD, that unifies visual relationship detection across datasets using VLMs, addressing label inconsistency and enhancing performance.

Findings

Achieves 38.07 mAP on HICO-DET, surpassing previous methods by 14.26 mAP.

Performs comparably to dataset-specific models, with further improvements when scaled.

Demonstrates effectiveness in human-object interaction detection and scene-graph generation.

Abstract

This work focuses on training a single visual relationship detector predicting over the union of label spaces from multiple datasets. Merging labels spanning different datasets could be challenging due to inconsistent taxonomies. The issue is exacerbated in visual relationship detection when second-order visual semantics are introduced between pairs of objects. To address this challenge, we propose UniVRD, a novel bottom-up method for Unified Visual Relationship Detection by leveraging vision and language models (VLMs). VLMs provide well-aligned image and text embeddings, where similar relationships are optimized to be close to each other for semantic unification. Our bottom-up design enables the model to enjoy the benefit of training with both object detection and visual relationship datasets. Empirical results on both human-object interaction detection and scene-graph generation demonstrate the competitive performance of our model. UniVRD achieves 38.07 mAP on HICO-DET, outperforming the current best bottom-up HOI detector by 14.26 mAP. More importantly, we show that our unified detector performs as well as dataset-specific models in mAP, and achieves further improvements when we scale up the model. Our code will be made publicly available on GitHub.