Block-Operations: Using Modular Routing to Improve Compositional Generalization

TL;DR

This paper introduces block-operations and a new architectural component called the Multiplexer, which significantly improves compositional generalization in neural networks by promoting modular routing and effective learning of underlying processes.

Contribution

The paper proposes block-operations and the Multiplexer as novel methods to enhance compositional generalization in neural networks, addressing routing difficulties.

Findings

Multiplexers exhibit strong compositional generalization.

Models with Multiplexers learn underlying processes rather than heuristics.

The approach outperforms standard FNNs and Transformers on synthetic and real tasks.

Abstract

We explore the hypothesis that poor compositional generalization in neural networks is caused by difficulties with learning effective routing. To solve this problem, we propose the concept of block-operations, which is based on splitting all activation tensors in the network into uniformly sized blocks and using an inductive bias to encourage modular routing and modification of these blocks. Based on this concept we introduce the Multiplexer, a new architectural component that enhances the Feed Forward Neural Network (FNN). We experimentally confirm that Multiplexers exhibit strong compositional generalization. On both a synthetic and a realistic task our model was able to learn the underlying process behind the task, whereas both FNNs and Transformers were only able to learn heuristic approximations. We propose as future work to use the principles of block-operations to improve other existing architectures.