Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation

TL;DR

This paper introduces RAHP, a novel framework that enhances open-vocabulary scene graph generation by integrating relation-aware prompts and dynamic prompt selection, significantly improving alignment and diversity in visual relationship detection.

Contribution

The paper proposes a relation-aware hierarchical prompting framework that leverages entity clustering, large language models, and dynamic prompt selection to improve open-vocabulary scene graph generation.

Findings

Achieves state-of-the-art performance on Visual Genome and Open Images v6 datasets.

Effectively captures fine-grained visual interactions and diverse relationships.

Reduces noise and improves accuracy through adaptive prompt selection.

Abstract

Open-vocabulary Scene Graph Generation (OV-SGG) overcomes the limitations of the closed-set assumption by aligning visual relationship representations with open-vocabulary textual representations. This enables the identification of novel visual relationships, making it applicable to real-world scenarios with diverse relationships. However, existing OV-SGG methods are constrained by fixed text representations, limiting diversity and accuracy in image-text alignment. To address these challenges, we propose the Relation-Aware Hierarchical Prompting (RAHP) framework, which enhances text representation by integrating subject-object and region-specific relation information. Our approach utilizes entity clustering to address the complexity of relation triplet categories, enabling the effective integration of subject-object information. Additionally, we utilize a large language model (LLM) to generate detailed region-aware prompts, capturing fine-grained visual interactions and improving alignment between visual and textual modalities. RAHP also introduces a dynamic selection mechanism within Vision-Language Models (VLMs), which adaptively selects relevant text prompts based on the visual content, reducing noise from irrelevant prompts. Extensive experiments on the Visual Genome and Open Images v6 datasets demonstrate that our framework consistently achieves state-of-the-art performance, demonstrating its effectiveness in addressing the challenges of open-vocabulary scene graph generation. The code is available at: https://github.com/Leon022/RAHP