Set-to-Sequence Methods in Machine Learning: a Review

TL;DR

This paper reviews set-to-sequence machine learning methods, focusing on permutation-invariant set representations and their use in generating complex output sequences across various applications.

Contribution

It offers a comprehensive overview and qualitative comparison of key models addressing the challenges of set representation and sequence generation.

Findings

Different model architectures have unique strengths in set representation and sequence output.

Permutation invariance is crucial for effective set-to-sequence learning.

The review highlights gaps and future directions in the field.

Abstract

Machine learning on sets towards sequential output is an important and ubiquitous task, with applications ranging from language modeling and meta-learning to multi-agent strategy games and power grid optimization. Combining elements of representation learning and structured prediction, its two primary challenges include obtaining a meaningful, permutation invariant set representation and subsequently utilizing this representation to output a complex target permutation. This paper provides a comprehensive introduction to the field as well as an overview of important machine learning methods tackling both of these key challenges, with a detailed qualitative comparison of selected model architectures.