Generating Diverse Programs with Instruction Conditioned Reinforced Adversarial Learning

TL;DR

This paper introduces a reinforcement learning method that enables agents to generate diverse programs conditioned on instructions, effectively capturing goal diversity in visual scene generation tasks.

Contribution

It proposes a modified reinforced adversarial learning approach for instruction-conditioned policies that produce diverse, goal-aligned programs, outperforming fixed reward baselines.

Findings

The method successfully generates diverse MNIST digit drawings based on instructions.

It constructs 3D scenes satisfying specific instructions with high diversity.

The stochastic policy better captures goal distribution diversity than fixed reward methods.

Abstract

Advances in Deep Reinforcement Learning have led to agents that perform well across a variety of sensory-motor domains. In this work, we study the setting in which an agent must learn to generate programs for diverse scenes conditioned on a given symbolic instruction. Final goals are specified to our agent via images of the scenes. A symbolic instruction consistent with the goal images is used as the conditioning input for our policies. Since a single instruction corresponds to a diverse set of different but still consistent end-goal images, the agent needs to learn to generate a distribution over programs given an instruction. We demonstrate that with simple changes to the reinforced adversarial learning objective, we can learn instruction conditioned policies to achieve the corresponding diverse set of goals. Most importantly, our agent's stochastic policy is shown to more accurately capture the diversity in the goal distribution than a fixed pixel-based reward function baseline. We demonstrate the efficacy of our approach on two domains: (1) drawing MNIST digits with a paint software conditioned on instructions and (2) constructing scenes in a 3D editor that satisfies a certain instruction.