Learning Functions over Sets via Permutation Adversarial Networks

TL;DR

This paper introduces SPAN, a novel neural network architecture that learns permutation-invariant functions over sets by adversarially optimizing permutations, outperforming existing methods on various set-learning tasks and improving recommendation system features.

Contribution

The paper proposes SPAN, a new permutation-invariant neural network architecture that uses adversarial permutation learning to efficiently model set functions.

Findings

SPAN achieves near permutation-invariance while maintaining accuracy.

Outperforms state-of-the-art methods like DeepSets and Janossy Pooling.

Improves recommendation system feature extraction by up to 2% accuracy.

Abstract

In this paper, we consider the problem of learning functions over sets, i.e., functions that are invariant to permutations of input set items. Recent approaches of pooling individual element embeddings can necessitate extremely large embedding sizes for challenging functions. We address this challenge by allowing standard neural networks like LSTMs to succinctly capture the function over the set. However, to ensure invariance with respect to permutations of set elements, we propose a novel architecture called SPAN that simultaneously learns the function as well as adversarial or worst-case permutations for each input set. The learning problem reduces to a min-max optimization problem that is solved via a simple alternating block coordinate descent technique. We conduct extensive experiments on a variety of set-learning tasks and demonstrate that SPAN learns nearly permutation-invariant functions while still ensuring accuracy on test data. On a variety of tasks sampled from the domains of statistics, graph functions and linear algebra, we show that our method can significantly outperform state-of-the-art methods such as DeepSets and Janossy Pooling. Finally, we present a case study of how learning set-functions can help extract powerful features for recommendation systems, and show that such a method can be as much as 2% more accurate than carefully hand-tuned features on a real-world recommendation system.