Insertion Based Sequence Generation with Learnable Order Dynamics

TL;DR

This paper introduces a learnable order dynamics approach for insertion-based sequence generation, enhancing flexibility and quality in tasks like molecule generation by jointly training order and generator models.

Contribution

It proposes a novel method to incorporate trainable order dynamics into insertion models, enabling joint training without numerical simulation and improving generation outcomes.

Findings

Learned order dynamics increase valid molecule generation.

Joint training improves flexibility and stability.

Trade-offs between flexibility, stability, and quality are demonstrated.

Abstract

In many domains generating variable length sequences through insertions provides greater flexibility over autoregressive models. However, the action space of insertion models is much larger than that of autoregressive models (ARMs) making the learning challenging. To address this, we incorporate trainable order dynamics into the target rates for discrete flow matching, and show that with suitable choices of parameterizations, joint training of the target order dynamics and the generator is tractable without the need for numerical simulation. As the generative insertion model, we use a variable length masked diffusion model, which generates by inserting and filling mask tokens. On graph traversal tasks for which a locally optimal insertion order is known, we explore the choices of parameterization empirically and demonstrate the trade-offs between flexibility, training stability and generation quality. On de novo small molecule generation, we find that the learned order dynamics leads to an increase in the number of valid molecules generated and improved quality, when compared to uniform order dynamics.