HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation

TL;DR

This paper introduces HiKER-SGG, a hierarchical knowledge-based approach for robust scene graph generation that performs well on both corrupted and clean images, and a new benchmark with weather and transformation corruptions.

Contribution

The paper presents a novel benchmark with weather corruptions and transformations, and proposes HiKER-SGG, a hierarchical knowledge graph method that enhances robustness in scene graph generation.

Findings

HiKER-SGG outperforms state-of-the-art methods on corrupted images in zero-shot settings.

HiKER-SGG achieves superior performance on uncorrupted scene graph generation tasks.

The benchmark provides a challenging testbed for robustness in scene understanding.

Abstract

Being able to understand visual scenes is a precursor for many downstream tasks, including autonomous driving, robotics, and other vision-based approaches. A common approach enabling the ability to reason over visual data is Scene Graph Generation (SGG); however, many existing approaches assume undisturbed vision, i.e., the absence of real-world corruptions such as fog, snow, smoke, as well as non-uniform perturbations like sun glare or water drops. In this work, we propose a novel SGG benchmark containing procedurally generated weather corruptions and other transformations over the Visual Genome dataset. Further, we introduce a corresponding approach, Hierarchical Knowledge Enhanced Robust Scene Graph Generation (HiKER-SGG), providing a strong baseline for scene graph generation under such challenging setting. At its core, HiKER-SGG utilizes a hierarchical knowledge graph in order to refine its predictions from coarse initial estimates to detailed predictions. In our extensive experiments, we show that HiKER-SGG does not only demonstrate superior performance on corrupted images in a zero-shot manner, but also outperforms current state-of-the-art methods on uncorrupted SGG tasks. Code is available at https://github.com/zhangce01/HiKER-SGG.