Neural Attention: Enhancing QKV Calculation in Self-Attention Mechanism with Neural Networks

TL;DR

This paper introduces a neural network-based approach to compute QKV in self-attention, leading to improved translation quality and language modeling performance over traditional linear methods.

Contribution

It proposes a novel neural network structure for QKV calculation in self-attention, demonstrating significant performance gains in translation and language modeling tasks.

Findings

Enhanced BLEU scores in translation tasks

Reduced perplexity in language modeling

Validated effectiveness across multiple datasets

Abstract

In the realm of deep learning, the self-attention mechanism has substantiated its pivotal role across a myriad of tasks, encompassing natural language processing and computer vision. Despite achieving success across diverse applications, the traditional self-attention mechanism primarily leverages linear transformations for the computation of query, key, and value (QKV), which may not invariably be the optimal choice under specific circumstances. This paper probes into a novel methodology for QKV computation-implementing a specially-designed neural network structure for the calculation. Utilizing a modified Marian model, we conducted experiments on the IWSLT 2017 German-English translation task dataset and juxtaposed our method with the conventional approach. The experimental results unveil a significant enhancement in BLEU scores with our method. Furthermore, our approach also manifested superiority when training the Roberta model with the Wikitext-103 dataset, reflecting a notable reduction in model perplexity compared to its original counterpart. These experimental outcomes not only validate the efficacy of our method but also reveal the immense potential in optimizing the self-attention mechanism through neural network-based QKV computation, paving the way for future research and practical applications. The source code and implementation details for our proposed method can be accessed at https://github.com/ocislyjrti/NeuralAttention.