Synthetic Visual Genome

TL;DR

This paper introduces ROBIN, a densely annotated instruction-tuned multimodal language model trained on a synthetic scene graph dataset, achieving state-of-the-art results in visual relationship reasoning and comprehension tasks.

Contribution

The paper presents ROBIN, a novel MLM trained on SVG, a synthetic dataset, and SG-EDIT, a self-distillation framework, to improve dense scene graph generation and visual reasoning.

Findings

ROBIN-3B outperforms larger models on relationship understanding benchmarks.

ROBIN achieves 88.9 in referring expression comprehension, surpassing previous best.

Training on refined scene graph data is key to high performance in visual reasoning.

Abstract

Reasoning over visual relationships-spatial, functional, interactional, social, etc.-is considered to be a fundamental component of human cognition. Yet, despite the major advances in visual comprehension in multimodal language models (MLMs), precise reasoning over relationships and their generations remains a challenge. We introduce ROBIN: an MLM instruction-tuned with densely annotated relationships capable of constructing high-quality dense scene graphs at scale. To train ROBIN, we curate SVG, a synthetic scene graph dataset by completing the missing relations of selected objects in existing scene graphs using a teacher MLM and a carefully designed filtering process to ensure high-quality. To generate more accurate and rich scene graphs at scale for any image, we introduce SG-EDIT: a self-distillation framework where GPT-4o further refines ROBIN's predicted scene graphs by removing unlikely relations and/or suggesting relevant ones. In total, our dataset contains 146K images and 5.6M relationships for 2.6M objects. Results show that our ROBIN-3B model, despite being trained on less than 3 million instances, outperforms similar-size models trained on over 300 million instances on relationship understanding benchmarks, and even surpasses larger models up to 13B parameters. Notably, it achieves state-of-the-art performance in referring expression comprehension with a score of 88.9, surpassing the previous best of 87.4. Our results suggest that training on the refined scene graph data is crucial to maintaining high performance across diverse visual reasoning task.