Hallucinate, Ground, Repeat: A Framework for Generalized Visual Relationship Detection

TL;DR

This paper presents an iterative framework that combines large language models and visual grounding to improve generalization in visual relationship detection, especially for unseen predicates, by bootstrapping relational understanding from external knowledge.

Contribution

It introduces a novel EM-inspired method leveraging LLMs as relational priors for open-world VRD and provides a new benchmark with unseen predicates for evaluation.

Findings

Outperforms baselines on open-world VRD benchmark

Achieves higher mean recall on predicate classification

Enables generalization to unseen predicates

Abstract

Understanding relationships between objects is central to visual intelligence, with applications in embodied AI, assistive systems, and scene understanding. Yet, most visual relationship detection (VRD) models rely on a fixed predicate set, limiting their generalization to novel interactions. A key challenge is the inability to visually ground semantically plausible, but unannotated, relationships hypothesized from external knowledge. This work introduces an iterative visual grounding framework that leverages large language models (LLMs) as structured relational priors. Inspired by expectation-maximization (EM), our method alternates between generating candidate scene graphs from detected objects using an LLM (expectation) and training a visual model to align these hypotheses with perceptual evidence (maximization). This process bootstraps relational understanding beyond annotated data and enables generalization to unseen predicates. Additionally, we introduce a new benchmark for open-world VRD on Visual Genome with 21 held-out predicates and evaluate under three settings: seen, unseen, and mixed. Our model outperforms LLM-only, few-shot, and debiased baselines, achieving mean recall (mR@50) of 15.9, 13.1, and 11.7 on predicate classification on these three sets. These results highlight the promise of grounded LLM priors for scalable open-world visual understanding.