Differential Gated Self-Attention

TL;DR

This paper introduces Multihead Differential Gated Self-Attention (M-DGSA), a biologically inspired mechanism that enhances Transformer robustness to noisy inputs by dynamically gating attention through excitatory and inhibitory branches, with minimal computational cost.

Contribution

It proposes a novel input-dependent gating mechanism for self-attention inspired by biological lateral inhibition, integrating contrast enhancement into Transformer models.

Findings

Demonstrates improved robustness on vision and language benchmarks.

Achieves consistent performance gains over baseline Transformers.

Shows effective noise suppression with minimal computational overhead.

Abstract

Transformers excel across a large variety of tasks but remain susceptible to corrupted inputs, since standard self-attention treats all query-key interactions uniformly. Inspired by lateral inhibition in biological neural circuits and building on the recent use by the Differential Transformer's use of two parallel softmax subtraction for noise cancellation, we propose Multihead Differential Gated Self-Attention (M-DGSA) that learns per-head input-dependent gating to dynamically suppress attention noise. Each head splits into excitatory and inhibitory branches whose dual softmax maps are fused by a sigmoid gate predicted from the token embedding, yielding a context-aware contrast enhancement. M-DGSA integrates seamlessly into existing Transformer stacks with minimal computational overhead. We evaluate on both vision and language benchmarks, demonstrating consistent robustness gains over vanilla Transformer, Vision Transformer, and Differential Transformer baselines. Our contributions are (i) a novel input-dependent gating mechanism for self-attention grounded in lateral inhibition, (ii) a principled synthesis of biological contrast-enhancement and self-attention theory, and (iii) comprehensive experiments demonstrating noise resilience and cross-domain applicability.