Set Prediction without Imposing Structure as Conditional Density Estimation

TL;DR

This paper introduces a novel set prediction method that models the problem as conditional density estimation using deep energy-based models, enabling multi-modal predictions without imposing strict set structures.

Contribution

It proposes a new learning framework that avoids set losses, utilizing energy-based models and gradient-guided sampling for flexible, multi-modal set predictions.

Findings

Capable of learning multi-modal densities

Produces diverse plausible predictions

Competitive performance on standard benchmarks

Abstract

Set prediction is about learning to predict a collection of unordered variables with unknown interrelations. Training such models with set losses imposes the structure of a metric space over sets. We focus on stochastic and underdefined cases, where an incorrectly chosen loss function leads to implausible predictions. Example tasks include conditional point-cloud reconstruction and predicting future states of molecules. In this paper, we propose an alternative to training via set losses by viewing learning as conditional density estimation. Our learning framework fits deep energy-based models and approximates the intractable likelihood with gradient-guided sampling. Furthermore, we propose a stochastically augmented prediction algorithm that enables multiple predictions, reflecting the possible variations in the target set. We empirically demonstrate on a variety of datasets the capability to learn multi-modal densities and produce different plausible predictions. Our approach is competitive with previous set prediction models on standard benchmarks. More importantly, it extends the family of addressable tasks beyond those that have unambiguous predictions.