Learning Set-equivariant Functions with SWARM Mappings

TL;DR

This paper introduces SWARM mappings, a neural network architecture that efficiently learns set-equivariant functions, achieving state-of-the-art results by mimicking natural swarm behaviors and outperforming existing methods like Set Transformer.

Contribution

The paper presents a novel SWARM mapping architecture based on gated recurrent networks for learning set-equivariant functions, demonstrating superior performance over existing models.

Findings

SWARM mappings achieve state-of-the-art results.

Models based on SWARM layers outperform Set Transformer.

The approach effectively learns permutation-invariant functions.

Abstract

In this work we propose a new neural network architecture that efficiently implements and learns general purpose set-equivariant functions. Such a function f maps a set of entities x = {x1, . . . , xn} from one domain to a set of same cardinality y = f (x) = {y1, . . . , yn} in another domain regardless of the ordering of the entities. The architecture is based on a gated recurrent network which is iteratively applied to all entities individually and at the same time syncs with the progression of the whole population. In reminiscence to this pattern, which can be frequently observed in nature, we call our approach SWARM mapping. Set-equivariant and generally permutation invariant functions are important building blocks for many state of the art machine learning approaches. Even in applications where the permutation invariance is not of primary interest, as to be seen in the recent success of attention based transformer models (Vaswani et. al. 2017). Accordingly, we demonstrate the power and usefulness of SWARM mappings in different applications. We compare the performance of our approach with another recently proposed set-equivariant function, the Set Transformer (Lee et.al. 2018) and we demonstrate that models solely based on SWARM layers gives state of the art results.