Look Ahead Text Understanding and LLM Stitching

TL;DR

This paper introduces the look ahead section identification (LASI) problem, demonstrating how combining bidirectional and unidirectional transformer models improves understanding of developing texts, especially under noisy conditions.

Contribution

It proposes novel methods to stitch BERT and GPT models for look ahead text understanding, outperforming existing models in noisy text scenarios.

Findings

Our approach outperforms established models in noisy text conditions.

Combining bidirectional and unidirectional models benefits look ahead understanding.

The methods have potential applications in social media sentiment analysis.

Abstract

This paper proposes a look ahead text understanding problem with look ahead section identification (LASI) as an example. This problem may appear in generative AI as well as human interactions, where we want to understand the direction of a developing text or conversation. We tackle the problem using transformer-based LLMs. We show that LASI is more challenging than classic section identification (SI). We argue that both bidirectional contextual information (e.g., BERT) and unidirectional predictive ability (e.g., GPT) will benefit the task. We propose two approaches to stitch together BERT and GPT. Experiments show that our approach outperforms the established models, especially when there is noise in the text (which is often the case for developing text in generative AI). Our paper sheds light on other look ahead text understanding tasks that are important to social media, such as look ahead sentiment classification, and points out the opportunities to leverage pre-trained LLMs through stitching.