Self-supervised Multi-view Disentanglement for Expansion of Visual Collections

TL;DR

This paper introduces a self-supervised learning approach to disentangle multi-view image representations, enabling more effective image retrieval and collection expansion by understanding the intent across different visual axes.

Contribution

It proposes a novel self-supervised method for learning disentangled view-specific features to improve multi-view image retrieval and collection expansion.

Findings

Disentangled representations reduce inter-view overlap.

Collection intent can be modeled as a distribution over views.

Enhanced retrieval performance with multi-collection composition.

Abstract

Image search engines enable the retrieval of images relevant to a query image. In this work, we consider the setting where a query for similar images is derived from a collection of images. For visual search, the similarity measurements may be made along multiple axes, or views, such as style and color. We assume access to a set of feature extractors, each of which computes representations for a specific view. Our objective is to design a retrieval algorithm that effectively combines similarities computed over representations from multiple views. To this end, we propose a self-supervised learning method for extracting disentangled view-specific representations for images such that the inter-view overlap is minimized. We show how this allows us to compute the intent of a collection as a distribution over views. We show how effective retrieval can be performed by prioritizing candidate expansion images that match the intent of a query collection. Finally, we present a new querying mechanism for image search enabled by composing multiple collections and perform retrieval under this setting using the techniques presented in this paper.