Improving Scene Graph Classification by Exploiting Knowledge from Texts

TL;DR

This paper demonstrates that leveraging textual scene descriptions and knowledge extraction significantly improves scene graph classification accuracy, reducing the need for extensive annotated image data.

Contribution

It introduces a framework that integrates symbolic textual knowledge into scene graph classification, achieving substantial accuracy gains with minimal annotated images.

Findings

8x improvement in scene graph classification accuracy

3x improvement in object classification

1.5x improvement in predicate classification

Abstract

Training scene graph classification models requires a large amount of annotated image data. Meanwhile, scene graphs represent relational knowledge that can be modeled with symbolic data from texts or knowledge graphs. While image annotation demands extensive labor, collecting textual descriptions of natural scenes requires less effort. In this work, we investigate whether textual scene descriptions can substitute for annotated image data. To this end, we employ a scene graph classification framework that is trained not only from annotated images but also from symbolic data. In our architecture, the symbolic entities are first mapped to their correspondent image-grounded representations and then fed into the relational reasoning pipeline. Even though a structured form of knowledge, such as the form in knowledge graphs, is not always available, we can generate it from unstructured texts using a transformer-based language model. We show that by fine-tuning the classification pipeline with the extracted knowledge from texts, we can achieve ~8x more accurate results in scene graph classification, ~3x in object classification, and ~1.5x in predicate classification, compared to the supervised baselines with only 1% of the annotated images.