Optimal word order for non-causal text generation with Large Language Models: the Spanish case

TL;DR

This paper investigates the optimal word order for non-causal language models in Spanish NLG, revealing that the best order differs from causal generation and is influenced by syntactic structure.

Contribution

It introduces a Viterbi algorithm-based method for estimating maximum likelihood word order in non-causal LLMs and compares it to causal order in Spanish.

Findings

Optimal order differs from causal order in Spanish.

Causal NLG prefers English-like SVO structures.

Optimal order is influenced by syntactic structure.

Abstract

Natural Language Generation (NLG) popularity has increased owing to the progress in Large Language Models (LLMs), with zero-shot inference capabilities. However, most neural systems utilize decoder-only causal (unidirectional) transformer models, which are effective for English but may reduce the richness of languages with less strict word order, subject omission, or different relative clause attachment preferences. This is the first work that analytically addresses optimal text generation order for non-causal language models. We present a novel Viterbi algorithm-based methodology for maximum likelihood word order estimation. We analyze the non-causal most-likelihood order probability for NLG in Spanish and, then, the probability of generating the same phrases with Spanish causal NLG. This comparative analysis reveals that causal NLG prefers English-like SVO structures. We also analyze the relationship between optimal generation order and causal left-to-right generation order using Spearman's rank correlation. Our results demonstrate that the ideal order predicted by the maximum likelihood estimator is not closely related to the causal order and may be influenced by the syntactic structure of the target sentence.