Differentiable Window for Dynamic Local Attention

TL;DR

This paper introduces Differentiable Window, a neural module for dynamic window selection that enhances attention mechanisms, leading to improved performance across various NLP tasks.

Contribution

It presents a novel Differentiable Window module and integrates it into Transformer models, demonstrating its effectiveness across multiple NLP benchmarks.

Findings

Consistent improvements in machine translation, sentiment analysis, and language modeling.

Enhanced focus in attention mechanisms improves task performance.

Two variants of Differentiable Window outperform baseline models.

Abstract

We propose Differentiable Window, a new neural module and general purpose component for dynamic window selection. While universally applicable, we demonstrate a compelling use case of utilizing Differentiable Window to improve standard attention modules by enabling more focused attentions over the input regions. We propose two variants of Differentiable Window, and integrate them within the Transformer architecture in two novel ways. We evaluate our proposed approach on a myriad of NLP tasks, including machine translation, sentiment analysis, subject-verb agreement and language modeling. Our experimental results demonstrate consistent and sizable improvements across all tasks.