Syntax-Directed Variational Autoencoder for Structured Data

TL;DR

This paper introduces a syntax-directed variational autoencoder that enforces both syntactic and semantic correctness in generating structured data like code and molecules, improving over existing methods.

Contribution

It proposes a novel SD-VAE with stochastic lazy attributes that guides the decoder on-the-fly, ensuring syntactic and semantic validity in generated data.

Findings

Outperforms state-of-the-art methods in generating valid structured data.

Effectively incorporates syntactic and semantic constraints.

Demonstrates success in programming language and molecular applications.

Abstract

Deep generative models have been enjoying success in modeling continuous data. However it remains challenging to capture the representations for discrete structures with formal grammars and semantics, e.g., computer programs and molecular structures. How to generate both syntactically and semantically correct data still remains largely an open problem. Inspired by the theory of compiler where the syntax and semantics check is done via syntax-directed translation (SDT), we propose a novel syntax-directed variational autoencoder (SD-VAE) by introducing stochastic lazy attributes. This approach converts the offline SDT check into on-the-fly generated guidance for constraining the decoder. Comparing to the state-of-the-art methods, our approach enforces constraints on the output space so that the output will be not only syntactically valid, but also semantically reasonable. We evaluate the proposed model with applications in programming language and molecules, including reconstruction and program/molecule optimization. The results demonstrate the effectiveness in incorporating syntactic and semantic constraints in discrete generative models, which is significantly better than current state-of-the-art approaches.