Few-shot Visual Relationship Co-localization

TL;DR

This paper introduces a novel few-shot visual relationship co-localization method that learns to localize subject-object pairs connected by common predicates across images with minimal supervision, using a meta-learning based optimization framework.

Contribution

It proposes a new optimization framework for VRC that employs relationship embedding and meta-learning to handle few-shot scenarios and unseen predicates.

Findings

Achieves impressive co-localization performance on VrR-VG and VG-150 datasets.

Effectively learns relationship embeddings as translation vectors in a shared space.

Utilizes a greedy approximation algorithm for efficient solution inference.

Abstract

In this paper, given a small bag of images, each containing a common but latent predicate, we are interested in localizing visual subject-object pairs connected via the common predicate in each of the images. We refer to this novel problem as visual relationship co-localization or VRC as an abbreviation. VRC is a challenging task, even more so than the well-studied object co-localization task. This becomes further challenging when using just a few images, the model has to learn to co-localize visual subject-object pairs connected via unseen predicates. To solve VRC, we propose an optimization framework to select a common visual relationship in each image of the bag. The goal of the optimization framework is to find the optimal solution by learning visual relationship similarity across images in a few-shot setting. To obtain robust visual relationship representation, we utilize a simple yet effective technique that learns relationship embedding as a translation vector from visual subject to visual object in a shared space. Further, to learn visual relationship similarity, we utilize a proven meta-learning technique commonly used for few-shot classification tasks. Finally, to tackle the combinatorial complexity challenge arising from an exponential number of feasible solutions, we use a greedy approximation inference algorithm that selects approximately the best solution. We extensively evaluate our proposed framework on variations of bag sizes obtained from two challenging public datasets, namely VrR-VG and VG-150, and achieve impressive visual co-localization performance.