Representing Unordered Data Using Complex-Weighted Multiset Automata

TL;DR

This paper introduces complex-weighted multiset automata to effectively represent unordered, variable-sized data, providing new theoretical insights and extending existing neural models like Transformers and DeepSets for improved performance.

Contribution

It presents a novel automaton-based framework for multisets, unifies existing neural architectures under this framework, and enhances DeepSets with complex numbers for better results.

Findings

Automaton-based multiset representations unify existing models.

Complex extension of DeepSets outperforms previous models.

Theoretical justification for sinusoidal positional encoding in Transformers.

Abstract

Unordered, variable-sized inputs arise in many settings across multiple fields. The ability for set- and multiset-oriented neural networks to handle this type of input has been the focus of much work in recent years. We propose to represent multisets using complex-weighted multiset automata and show how the multiset representations of certain existing neural architectures can be viewed as special cases of ours. Namely, (1) we provide a new theoretical and intuitive justification for the Transformer model's representation of positions using sinusoidal functions, and (2) we extend the DeepSets model to use complex numbers, enabling it to outperform the existing model on an extension of one of their tasks.