Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding

TL;DR

This paper introduces a scalable set encoding method that maintains permutation invariance and equivariance while being efficient for large or streaming data, addressing limitations of existing algorithms.

Contribution

The authors propose Mini-Batch Consistency (MBC) and an attention-based set encoder suitable for large-scale and streaming set data, ensuring symmetry properties and efficiency.

Findings

Method is computationally efficient.

Produces rich set representations.

Works effectively on large-scale and streaming data.

Abstract

Most existing set encoding algorithms operate under the implicit assumption that all the set elements are accessible, and that there are ample computational and memory resources to load the set into memory during training and inference. However, both assumptions fail when the set is excessively large such that it is impossible to load all set elements into memory, or when data arrives in a stream. To tackle such practical challenges in large-scale set encoding, the general set-function constraints of permutation invariance and equivariance are not sufficient. We introduce a new property termed Mini-Batch Consistency (MBC) that is required for large scale mini-batch set encoding. Additionally, we present a scalable and efficient attention-based set encoding mechanism that is amenable to mini-batch processing of sets, and capable of updating set representations as data arrives. The proposed method adheres to the required symmetries of invariance and equivariance as well as maintaining MBC for any partition of the input set. We perform extensive experiments and show that our method is computationally efficient and results in rich set encoding representations for set-structured data.