DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement

TL;DR

This paper introduces DiscoSG, a new discourse-level text scene graph parsing task and dataset, along with a lightweight iterative graph refinement method that significantly improves parsing accuracy and efficiency for complex multi-sentence visual descriptions.

Contribution

The paper presents DiscoSG-DS dataset, a novel task for discourse-level scene graph parsing, and proposes DiscoSG-Refiner, an open-source iterative refinement model that outperforms baselines in accuracy and speed.

Findings

Fine-tuning GPT-4o improves SPICE by over 40% but has high inference costs.

Smaller models perform well on simple graphs but struggle with complex ones.

DiscoSG-Refiner achieves 30% higher SPICE and 86x faster inference than GPT-4o.

Abstract

Vision-Language Models (VLMs) generate discourse-level, multi-sentence visual descriptions, challenging text scene graph parsers built for single-sentence caption-to-graph mapping. Current approaches typically merge sentence-level parsing outputs for discourse input, often missing phenomena like cross-sentence coreference, resulting in fragmented graphs and degraded downstream VLM task performance. We introduce a new task, Discourse-level text Scene Graph parsing (DiscoSG), and release DiscoSG-DS, a dataset of 400 expert-annotated and 8,430 synthesised multi-sentence caption-graph pairs. Each caption averages 9 sentences, and each graph contains at least 3 times more triples than those in existing datasets. Fine-tuning GPT-4o on DiscoSG-DS yields over 40% higher SPICE metric than the best sentence-merging baseline. However, its high inference cost and licensing restrict open-source use. Smaller fine-tuned open-source models (e.g., Flan-T5) perform well on simpler graphs yet degrade on denser, more complex graphs. To bridge this gap, we introduce DiscoSG-Refiner, a lightweight open-source parser that drafts a seed graph and iteratively refines it with a novel learned graph-editing model, achieving 30% higher SPICE than the baseline while delivering 86 times faster inference than GPT-4o. It generalises from simple to dense graphs, thereby consistently improving downstream VLM tasks, including discourse-level caption evaluation and hallucination detection, outperforming alternative open-source parsers. Code and data are available at https://github.com/ShaoqLin/DiscoSG .