The Backpropagation of the Wave Network

TL;DR

This paper introduces Wave Network, a novel token representation method using complex vectors to capture global and local semantics, with analysis showing efficiency and reduced resource usage compared to BERT.

Contribution

The paper presents a new wave-inspired token representation method, Token2Wave, with analysis of its convergence, backpropagation, and efficiency advantages over existing models.

Findings

Reduces video memory usage compared to BERT

Maintains semantic richness through complex vector representations

Demonstrates efficient training and convergence behavior

Abstract

This paper provides an in-depth analysis of Wave Network, a novel token representation method derived from the Wave Network, designed to capture both global and local semantics of input text through wave-inspired complex vectors. In complex vector token representation, each token is represented with a magnitude component, capturing the global semantics of the entire input text, and a phase component, encoding the relationships between individual tokens and the global semantics. Building on prior research that demonstrated the effectiveness of wave-like operations, such as interference and modulation, during forward propagation, this study investigates the convergence behavior, backpropagation characteristics, and embedding independence within the Token2Wave framework. A detailed computational complexity analysis shows that Token2Wave can significantly reduce video memory usage and training time compared to BERT. Gradient comparisons for the [CLS] token, total input text, and classifier parameters further highlight Token2Wave's unique characteristics. This research offers new insights into wave-based token representations, demonstrating their potential to enable efficient and computationally friendly language model architectures.