Inceptive Transformers: Enhancing Contextual Representations through Multi-Scale Feature Learning Across Domains and Languages

TL;DR

Inceptive Transformers introduce a multi-scale convolution module to enhance transformer models by preserving local features, leading to improved performance across various tasks and languages without significant efficiency loss.

Contribution

The paper presents a novel inception-style convolution module integrated into transformer models to better capture local features, improving their effectiveness across multiple domains and languages.

Findings

Consistent performance improvements across five diverse tasks.

Multi-scale convolution outperforms single kernel approaches.

Self-attention layer is crucial for the enhanced performance.

Abstract

Encoder transformer models compress information from all tokens in a sequence into a single [CLS] token to represent global context. This approach risks diluting fine-grained or hierarchical features, leading to information loss in downstream tasks where local patterns are important. To remedy this, we propose a lightweight architectural enhancement: an inception-style 1-D convolution module that sits on top of the transformer layer and augments token representations with multi-scale local features. This enriched feature space is then processed by a self-attention layer that dynamically weights tokens based on their task relevance. Experiments on five diverse tasks show that our framework consistently improves general-purpose, domain-specific, and multilingual models, outperforming baselines by 1% to 14% while maintaining efficiency. Ablation studies show that multi-scale convolution performs better than any single kernel and that the self-attention layer is critical for performance.