Hierarchical GPT with Congruent Transformers for Multi-Sentence Language Models

TL;DR

This paper introduces a hierarchical GPT model with congruent transformers designed for multi-sentence dialogue generation and document understanding, improving performance by aligning similarity measures and structuring sentence processing.

Contribution

The paper proposes a novel hierarchical GPT architecture with congruent transformers and a new similarity measure alignment, enhancing multi-sentence language modeling capabilities.

Findings

Improved performance on multi-sentence tasks

Effective hierarchical structure for dialogue and document understanding

Enhanced similarity comparison in transformers

Abstract

We report a GPT-based multi-sentence language model for dialogue generation and document understanding. First, we propose a hierarchical GPT which consists of three blocks, i.e., a sentence encoding block, a sentence generating block, and a sentence decoding block. The sentence encoding and decoding blocks are basically the encoder-decoder blocks of the standard Transformers, which work on each sentence independently. The sentence generating block is inserted between the encoding and decoding blocks, and generates the next sentence embedding vector from the previous sentence embedding vectors. We believe it is the way human make conversation and understand paragraphs and documents. Since each sentence may consist of fewer words, the sentence encoding and decoding Transformers can use much smaller dimensional embedding vectors. Secondly, we note the attention in the Transformers utilizes the inner-product similarity measure. Therefore, to compare the two vectors in the same space, we set the transform matrices for queries and keys to be the same. Otherwise, the similarity concept is incongruent. We report experimental results to show that these two modifications increase the language model performance for tasks with multiple sentences.