Knowledge-augmented Few-shot Visual Relation Detection

TL;DR

This paper introduces a knowledge-augmented few-shot visual relation detection framework that leverages textual and visual relation knowledge to significantly improve generalization and outperform existing models on standard benchmarks.

Contribution

The paper proposes a novel framework combining textual and visual relation knowledge to enhance few-shot VRD performance, addressing generalization issues of prior methods.

Findings

Outperforms state-of-the-art models on three Visual Genome benchmarks.

Significant improvement in generalization ability for few-shot VRD.

Effectiveness validated through extensive experiments.

Abstract

Visual Relation Detection (VRD) aims to detect relationships between objects for image understanding. Most existing VRD methods rely on thousands of training samples of each relationship to achieve satisfactory performance. Some recent papers tackle this problem by few-shot learning with elaborately designed pipelines and pre-trained word vectors. However, the performance of existing few-shot VRD models is severely hampered by the poor generalization capability, as they struggle to handle the vast semantic diversity of visual relationships. Nonetheless, humans have the ability to learn new relationships with just few examples based on their knowledge. Inspired by this, we devise a knowledge-augmented, few-shot VRD framework leveraging both textual knowledge and visual relation knowledge to improve the generalization ability of few-shot VRD. The textual knowledge and visual relation knowledge are acquired from a pre-trained language model and an automatically constructed visual relation knowledge graph, respectively. We extensively validate the effectiveness of our framework. Experiments conducted on three benchmarks from the commonly used Visual Genome dataset show that our performance surpasses existing state-of-the-art models with a large improvement.