In Tree Structure Should Sentence Be Generated

TL;DR

This paper proposes a novel tree-structured approach to sentence generation, aiming to address issues like hallucinations and logical loops in autoregressive models, by introducing the SenTree module and a joint training framework.

Contribution

It introduces a new tree-traversing method for sequence generation, including the SenTree module and a GAN-based joint training framework, enhancing language generation performance.

Findings

The tree-based approach improves coherence in generated sentences.

SenTree effectively approximates binary trees for sentence structure.

The joint training framework enhances generation quality.

Abstract

Generative models reliant on sequential autoregression have been at the forefront of language generation for an extensive period, particularly following the introduction of widely acclaimed transformers. Despite its excellent performance, there are always some issues that we face today. For example, problems such as hallucinations and getting trapped in a logic loop may occur. To enhance the performance of existing systems, this paper introduces a new method for generating sequences in natural language, which involves generating the targeted sentence in a tree-traversing order. The paper includes an illustration of the theoretical basis and validity of the approach, as well as a comparison of its fundamentals with the diffusion model in graphic generation. Finally, a module called SenTree is introduced for generating an approximating binary tree. It is already available at https://github.com/arklyg/sentree. Additionally, a joint training framework based on this approach is proposed, incorporating the intrinsics of generative adversarial networks.