Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecture

TL;DR

This paper introduces Wonderful Matrices, a novel foundation model architecture that combines sequence and state transformations to improve efficiency and effectiveness, demonstrating significant performance gains across multiple tasks.

Contribution

It proposes a new matrix-based foundation model architecture that unifies position encoding, enhances multi-query recall, and accelerates expert retrieval, outperforming existing methods.

Findings

Rotary position embedding reduces perplexity by over 4%.

Dynamic mask attention achieves 100% accuracy in complex recall tasks.

Expert retrieval speed increases 8-10 times with over 1024 experts.

Abstract

In order to make the foundation model more efficient and effective, our idea is combining sequence transformation and state transformation. First, we prove the availability of rotary position embedding in the state space duality algorithm, which reduces the perplexity of the hybrid quadratic causal self-attention and state space duality by more than 4%, to ensure that the combining sequence transformation unifies position encoding. Second, we propose dynamic mask attention, which maintains 100% accuracy in the more challenging multi-query associative recall task, improving by more than 150% compared to quadratic causal self-attention and state space duality, to ensure that the combining sequence transformation selectively filters relevant information. Third, we design cross domain mixture of experts, which makes the computational speed of expert retrieval with more than 1024 experts 8 to 10 times faster than the mixture of experts, to ensure that the combining state transformation quickly retrieval mixture. Finally, we summarize these matrix algorithms that can form the foundation model: Wonderful Matrices, which can be a competitor to popular model architectures.