AGDC: Autoregressive Generation of Variable-Length Sequences with Joint Discrete and Continuous Spaces

TL;DR

AGDC introduces a unified autoregressive framework that jointly models discrete and continuous data, enabling high-precision sequence generation in complex domains like semiconductor design, surpassing existing discretization-based methods.

Contribution

The paper proposes AGDC, a novel hybrid model combining categorical and diffusion-based approaches with dynamic EOS adjustment and length regularization for scalable high-precision sequence generation.

Findings

AGDC outperforms baselines in semiconductor layout generation.

The model achieves high fidelity in diverse vector data.

Experiments demonstrate scalability and precision in complex domains.

Abstract

Transformer-based autoregressive models excel in data generation but are inherently constrained by their reliance on discretized tokens, which limits their ability to represent continuous values with high precision. We analyze the scalability limitations of existing discretization-based approaches for generating hybrid discrete-continuous sequences, particularly in high-precision domains such as semiconductor circuit designs, where precision loss can lead to functional failure. To address the challenge, we propose AGDC, a novel unified framework that jointly models discrete and continuous values for variable-length sequences. AGDC employs a hybrid approach that combines categorical prediction for discrete values with diffusion-based modeling for continuous values, incorporating two key technical components: an end-of-sequence (EOS) logit adjustment mechanism that uses an MLP to dynamically adjust EOS token logits based on sequence context, and a length regularization term integrated into the loss function. Additionally, we present ContLayNet, a large-scale benchmark comprising 334K high-precision semiconductor layout samples with specialized evaluation metrics that capture functional correctness where precision errors significantly impact performance. Experiments on semiconductor layouts (ContLayNet), graphic layouts, and SVGs demonstrate AGDC's superior performance in generating high-fidelity hybrid vector representations compared to discretization-based and fixed-schema baselines, achieving scalable high-precision generation across diverse domains.