Layer-wise Representation Fusion for Compositional Generalization

TL;DR

This paper introduces LRF, a layer-wise fusion framework with fuse-attention modules to improve compositional generalization in neural models by effectively integrating multi-layer information.

Contribution

It identifies the representation entanglement problem in Transformers and proposes a novel fusion method to address it, enhancing CG performance.

Findings

LRF outperforms baseline models on benchmark datasets.

Fuse-attention modules effectively integrate multi-layer information.

Analysis shows improved syntactic and semantic disentanglement.

Abstract

Existing neural models are demonstrated to struggle with compositional generalization (CG), i.e., the ability to systematically generalize to unseen compositions of seen components. A key reason for failure on CG is that the syntactic and semantic representations of sequences in both the uppermost layer of the encoder and decoder are entangled. However, previous work concentrates on separating the learning of syntax and semantics instead of exploring the reasons behind the representation entanglement (RE) problem to solve it. We explain why it exists by analyzing the representation evolving mechanism from the bottom to the top of the Transformer layers. We find that the ``shallow'' residual connections within each layer fail to fuse previous layers' information effectively, leading to information forgetting between layers and further the RE problems. Inspired by this, we propose LRF, a novel \textbf{L}ayer-wise \textbf{R}epresentation \textbf{F}usion framework for CG, which learns to fuse previous layers' information back into the encoding and decoding process effectively through introducing a \emph{fuse-attention module} at each encoder and decoder layer. LRF achieves promising results on two realistic benchmarks, empirically demonstrating the effectiveness of our proposal.