State Combinatorial Generalization In Decision Making With Conditional Diffusion Models

TL;DR

This paper introduces a new approach using conditioned diffusion models for zero-shot generalization in combinatorial decision-making problems, outperforming traditional RL methods in unseen state combinations.

Contribution

The work formalizes the zero-shot generalization problem in combinatorial decision-making and demonstrates the effectiveness of conditioned diffusion models over traditional RL algorithms.

Findings

Diffusion models generalize better to unseen state combinations.

Behavior cloning with diffusion models outperforms RL in various environments.

The approach is broadly applicable across maze, driving, and multiagent tasks.

Abstract

Many real-world decision-making problems are combinatorial in nature, where states (e.g., surrounding traffic of a self-driving car) can be seen as a combination of basic elements (e.g., pedestrians, trees, and other cars). Due to combinatorial complexity, observing all combinations of basic elements in the training set is infeasible, which leads to an essential yet understudied problem of zero-shot generalization to states that are unseen combinations of previously seen elements. In this work, we first formalize this problem and then demonstrate how existing value-based reinforcement learning (RL) algorithms struggle due to unreliable value predictions in unseen states. We argue that this problem cannot be addressed with exploration alone, but requires more expressive and generalizable models. We demonstrate that behavior cloning with a conditioned diffusion model trained on successful trajectory generalizes better to states formed by new combinations of seen elements than traditional RL methods. Through experiments in maze, driving, and multiagent environments, we show that conditioned diffusion models outperform traditional RL techniques and highlight the broad applicability of our problem formulation.