Abelian Neural Networks

TL;DR

This paper introduces neural network architectures for modeling Abelian group and semigroup operations, leveraging algebraic properties to improve multiset function representation and demonstrate benefits in word analogy tasks.

Contribution

It presents novel neural network models that incorporate algebraic structures of Abelian groups and semigroups, enabling size-generalization and improved performance in word analogy tasks.

Findings

Models achieve size-generalization in multiset functions

Improved performance over word2vec in word analogy tasks

Demonstrates the utility of algebraic structures in neural network design

Abstract

We study the problem of modeling a binary operation that satisfies some algebraic requirements. We first construct a neural network architecture for Abelian group operations and derive a universal approximation property. Then, we extend it to Abelian semigroup operations using the characterization of associative symmetric polynomials. Both models take advantage of the analytic invertibility of invertible neural networks. For each case, by repeating the binary operations, we can represent a function for multiset input thanks to the algebraic structure. Naturally, our multiset architecture has size-generalization ability, which has not been obtained in existing methods. Further, we present modeling the Abelian group operation itself is useful in a word analogy task. We train our models over fixed word embeddings and demonstrate improved performance over the original word2vec and another naive learning method.