SceneProp: Combining Neural Network and Markov Random Field for Scene-Graph Grounding

TL;DR

SceneProp introduces a novel approach to scene-graph grounding by formulating it as a MAP inference problem in an MRF, leveraging global inference and differentiable belief propagation to improve accuracy with complex relational queries.

Contribution

It presents SceneProp, a new method that enhances scene-graph grounding by integrating neural networks with Markov Random Fields and end-to-end differentiable inference.

Findings

Outperforms prior methods on four benchmarks.

Accuracy improves with larger and more complex query graphs.

Effectively leverages relational context for better grounding.

Abstract

Grounding complex, compositional visual queries with multiple objects and relationships is a fundamental challenge for vision-language models. While standard phrase grounding methods excel at localizing single objects, they lack the structural inductive bias to parse intricate relational descriptions, often failing as queries become more descriptive. To address this structural deficit, we focus on scene-graph grounding, a powerful but less-explored formulation where the query is an explicit graph of objects and their relationships. However, existing methods for this task also struggle, paradoxically showing decreased performance as the query graph grows -- failing to leverage the very information that should make grounding easier. We introduce SceneProp, a novel method that resolves this issue by reformulating scene-graph grounding as a Maximum a Posteriori (MAP) inference problem in a Markov Random Field (MRF). By performing global inference over the entire query graph, SceneProp finds the optimal assignment of image regions to nodes that jointly satisfies all constraints. This is achieved within an end-to-end framework via a differentiable implementation of the Belief Propagation algorithm. Experiments on four benchmarks show that our dedicated focus on the scene-graph grounding formulation allows SceneProp to significantly outperform prior work. Critically, its accuracy consistently improves with the size and complexity of the query graph, demonstrating for the first time that more relational context can, and should, lead to better grounding. Codes are available at https://github.com/keitaotani/SceneProp.