Compositional Mixture Representations for Vision and Text

TL;DR

This paper introduces a model that learns a shared, compositional Gaussian mixture representation for vision and language, enabling weakly supervised object detection and generalization to unseen object combinations.

Contribution

The model uniquely combines spatial transformers with Gaussian mixture representations to encode images into interpretable patches aligned with textual semantics without explicit location supervision.

Findings

Successfully performs weakly supervised object detection.

Generalizes to unseen object combinations.

Effective on MNIST and CIFAR10 variations.

Abstract

Learning a common representation space between vision and language allows deep networks to relate objects in the image to the corresponding semantic meaning. We present a model that learns a shared Gaussian mixture representation imposing the compositionality of the text onto the visual domain without having explicit location supervision. By combining the spatial transformer with a representation learning approach we learn to split images into separately encoded patches to associate visual and textual representations in an interpretable manner. On variations of MNIST and CIFAR10, our model is able to perform weakly supervised object detection and demonstrates its ability to extrapolate to unseen combination of objects.