Fully Distributed, Flexible Compositional Visual Representations via Soft Tensor Products

TL;DR

This paper introduces Soft TPR, a novel distributed approach to compositional visual representations that outperforms traditional disentanglement methods, improving learning efficiency and sample performance.

Contribution

It extends Smolensky's Tensor Product Representation to a soft, distributed form and develops an autoencoder architecture for learning these representations.

Findings

Achieves state-of-the-art disentanglement performance

Enhances convergence and sample efficiency in representation learning

Outperforms conventional methods in downstream visual tasks

Abstract

Since the inception of the classicalist vs. connectionist debate, it has been argued that the ability to systematically combine symbol-like entities into compositional representations is crucial for human intelligence. In connectionist systems, the field of disentanglement has gained prominence for its ability to produce explicitly compositional representations; however, it relies on a fundamentally symbolic, concatenative representation of compositional structure that clashes with the continuous, distributed foundations of deep learning. To resolve this tension, we extend Smolensky's Tensor Product Representation (TPR) and introduce Soft TPR, a representational form that encodes compositional structure in an inherently distributed, flexible manner, along with Soft TPR Autoencoder, a theoretically-principled architecture designed specifically to learn Soft TPRs. Comprehensive evaluations in the visual representation learning domain demonstrate that the Soft TPR framework consistently outperforms conventional disentanglement alternatives -- achieving state-of-the-art disentanglement, boosting representation learner convergence, and delivering superior sample efficiency and low-sample regime performance in downstream tasks. These findings highlight the promise of a distributed and flexible approach to representing compositional structure by potentially enhancing alignment with the core principles of deep learning over the conventional symbolic approach.