Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks

TL;DR

This paper introduces Wonderful Matrices, an innovative architecture for language modeling that combines inner product position encoding, dynamic masking attention, and cross domain mixture of experts to enhance efficiency and effectiveness.

Contribution

It presents a novel architecture integrating new position encoding, attention mechanisms, and expert models, improving language modeling performance and efficiency.

Findings

Enhanced language modeling accuracy

Improved computational efficiency

Better handling of complex language tasks

Abstract

We prove the availability of inner product form position encoding in the state space dual algorithm and study the effectiveness of different position embeddings in the hybrid quadratic causal self-attention and state space dual algorithms. We propose inner function attention with dynamic mask, which can improve the expressiveness of the attention algorithm and avoid the sequence noise significantly affecting the accuracy of the attention score. We also design cross domain mixture of experts, which can improve the granularity of the sparse activation feedforward network while maintaining the efficiency of parameter utilization and retrieval. The combination of these methods constitutes our foundation model architecture: Wonderful Matrices. We conduct experiments on the language modeling task and find that Wonderful Matrices are more efficient and effective in handling complex language tasks.