Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddings

TL;DR

This paper introduces a novel approach combining iterated learning and simplicial embeddings to enhance the compositional generalization of deep neural networks, inspired by cognitive science theories and analyzed through Kolmogorov complexity.

Contribution

It proposes a new method that leverages iterated learning and simplicial embeddings to improve compositional generalization in neural networks, with theoretical motivation and empirical validation.

Findings

Improved compositional generalization on vision tasks

Enhanced performance on molecular graph prediction

Outperforms existing approaches in experiments

Abstract

Compositional generalization, the ability of an agent to generalize to unseen combinations of latent factors, is easy for humans but hard for deep neural networks. A line of research in cognitive science has hypothesized a process, ``iterated learning,'' to help explain how human language developed this ability; the theory rests on simultaneous pressures towards compressibility (when an ignorant agent learns from an informed one) and expressivity (when it uses the representation for downstream tasks). Inspired by this process, we propose to improve the compositional generalization of deep networks by using iterated learning on models with simplicial embeddings, which can approximately discretize representations. This approach is further motivated by an analysis of compositionality based on Kolmogorov complexity. We show that this combination of changes improves compositional generalization over other approaches, demonstrating these improvements both on vision tasks with well-understood latent factors and on real molecular graph prediction tasks where the latent structure is unknown.