DISCO : efficient unsupervised decoding for discrete natural language problems via convex relaxation

TL;DR

This paper introduces Disco, a convex relaxation-based decoding algorithm for NLP tasks that efficiently approximates solutions to NP-hard problems, demonstrating superior performance in adversarial text generation.

Contribution

Develops a convex relaxation framework and an efficient gradient-based algorithm for decoding in NLP, with theoretical convergence guarantees and improved empirical results.

Findings

Disco converges linearly to near-optimal solutions.

Outperforms existing decoding methods in adversarial text generation.

Provides theoretical analysis of the relaxation approach.

Abstract

In this paper we study test time decoding; an ubiquitous step in almost all sequential text generation task spanning across a wide array of natural language processing (NLP) problems. Our main contribution is to develop a continuous relaxation framework for the combinatorial NP-hard decoding problem and propose Disco - an efficient algorithm based on standard first order gradient based. We provide tight analysis and show that our proposed algorithm linearly converges to within $\epsilon$ neighborhood of the optima. Finally, we perform preliminary experiments on the task of adversarial text generation and show superior performance of Disco over several popular decoding approaches.