IDL-Expressions: A Formalism for Representing and Parsing Finite Languages in Natural Language Processing

TL;DR

This paper introduces IDL-expressions, a new formalism for representing finite languages in NLP, enabling efficient generation and filtering of candidate sentences in tasks like translation and generation.

Contribution

It presents the IDL-expressions formalism, analyzes its properties, compares it with existing formalisms, and introduces a novel parsing algorithm with proven complexity bounds.

Findings

IDL-expressions effectively represent large sets of candidate sentences.

The proposed parsing algorithm has a non-trivial upper bound on its time complexity.

IDL-expressions outperform traditional formalisms in certain NLP applications.

Abstract

We propose a formalism for representation of finite languages, referred to as the class of IDL-expressions, which combines concepts that were only considered in isolation in existing formalisms. The suggested applications are in natural language processing, more specifically in surface natural language generation and in machine translation, where a sentence is obtained by first generating a large set of candidate sentences, represented in a compact way, and then by filtering such a set through a parser. We study several formal properties of IDL-expressions and compare this new formalism with more standard ones. We also present a novel parsing algorithm for IDL-expressions and prove a non-trivial upper bound on its time complexity.