Language Generation via Combinatorial Constraint Satisfaction: A Tree Search Enhanced Monte-Carlo Approach

TL;DR

This paper introduces TSMH, a flexible and efficient tree search-enhanced Monte Carlo method for constrained natural language generation, improving sampling quality without task-specific training.

Contribution

The paper presents a novel tree search integrated into MCMC for better constraint satisfaction in language generation, without requiring task-specific training.

Findings

TSMH outperforms existing MCMC methods in mixing performance.

The approach achieves significant improvements across multiple language tasks.

It effectively satisfies complex combinatorial constraints in generated sentences.

Abstract

Generating natural language under complex constraints is a principled formulation towards controllable text generation. We present a framework to allow specification of combinatorial constraints for sentence generation. We propose TSMH, an efficient method to generate high likelihood sentences with respect to a pre-trained language model while satisfying the constraints. Our approach is highly flexible, requires no task-specific training, and leverages efficient constraint satisfaction solving techniques. To better handle the combinatorial constraints, a tree search algorithm is embedded into the proposal process of the Markov chain Monte Carlo (MCMC) to explore candidates that satisfy more constraints. Compared to existing MCMC approaches, our sampling approach has a better mixing performance. Experiments show that TSMH achieves consistent and significant improvement on multiple language generation tasks.