# On the Realization of Compositionality in Neural Networks

## TL;DR

This paper compares two sequence-to-sequence neural network models trained for a compositional task, demonstrating that models with attentive guidance develop more modular and compositional solutions than baseline models.

## Contribution

It provides an in-depth analysis of how attentive guidance influences the internal structure and compositionality of neural networks.

## Key findings

- Guided models infer more compositional solutions.
- Guided models develop more modular and specialized neuron groups.
- Structural differences are evident in parameter space and activations.

## Abstract

We present a detailed comparison of two types of sequence to sequence models trained to conduct a compositional task. The models are architecturally identical at inference time, but differ in the way that they are trained: our baseline model is trained with a task-success signal only, while the other model receives additional supervision on its attention mechanism (Attentive Guidance), which has shown to be an effective method for encouraging more compositional solutions (Hupkes et al.,2019). We first confirm that the models with attentive guidance indeed infer more compositional solutions than the baseline, by training them on the lookup table task presented by Li\v{s}ka et al. (2019). We then do an in-depth analysis of the structural differences between the two model types, focusing in particular on the organisation of the parameter space and the hidden layer activations and find noticeable differences in both these aspects. Guided networks focus more on the components of the input rather than the sequence as a whole and develop small functional groups of neurons with specific purposes that use their gates more selectively. Results from parameter heat maps, component swapping and graph analysis also indicate that guided networks exhibit a more modular structure with a small number of specialized, strongly connected neurons.