Understanding Transformer Encoder-Decoder Representations through Bernoulli Dropout

TL;DR

This paper investigates how Bernoulli dropout affects Transformer encoder-decoder models, revealing a sparsity threshold for maintaining prediction accuracy and demonstrating this through theoretical analysis and translation experiments.

Contribution

It introduces a novel analysis of Transformer overparameterization using Bernoulli dropout and establishes a sparsity-dependent threshold for preserving model performance.

Findings

Prediction accuracy remains stable below a certain dropout threshold.

Validation accuracy and BLEU scores decline sharply beyond the threshold.

Theoretical proof links sparsity levels to decoder stability.

Abstract

We study Transformer overparameterization through the lens of angular similarity in high-dimensional encoder-decoder embeddings. We apply Bernoulli dropout between the encoder and the decoder, varying the keep probability $p$ to identify a sparsity-dependent threshold above which the Top-1 prediction is preserved. Theoretically, we prove that, if the effective sparsity embeddings is sufficiently large, and thus decoder performance, remain stable under moderate coordinate dropout. Empirically, we implement the Bernoulli dropout by constructing a new Transformer model augmented with Binary Erasure Channel (BEC) and test its performance on an English-French translation task. Experimental results visualize the trends for validation accuracies and BLEU scores, both decline sharply at some threshold.