Enhancing LLM's Cognition via Structurization

TL;DR

This paper introduces context structurization, transforming unorganized text into hierarchical structures to improve LLM understanding, leading to significant performance improvements across various models and tasks.

Contribution

It proposes a novel context structurization method that enhances LLM cognition by organizing input data hierarchically, which is validated through extensive experiments.

Findings

Performance gains across multiple NLP tasks

LLaMA2-70B matches GPT-3.5-Turbo in hallucination evaluation

Effective distillation into smaller models like StruXGPT-7B

Abstract

When reading long-form text, human cognition is complex and structurized. While large language models (LLMs) process input contexts through a causal and sequential perspective, this approach can potentially limit their ability to handle intricate and complex inputs effectively. To enhance LLM's cognition capability, this paper presents a novel concept of context structurization. Specifically, we transform the plain, unordered contextual sentences into well-ordered and hierarchically structurized elements. By doing so, LLMs can better grasp intricate and extended contexts through precise attention and information-seeking along the organized structures. Extensive evaluations are conducted across various model architectures and sizes (including a series of auto-regressive LLMs as well as BERT-like masking models) on a diverse set of NLP tasks (e.g., context-based question-answering, exhaustive hallucination evaluation, and passage-level dense retrieval). Empirical results show consistent and significant performance gains afforded by a single-round structurization. In particular, we boost the open-sourced LLaMA2-70B model to achieve comparable performance against GPT-3.5-Turbo as the hallucination evaluator. Besides, we show the feasibility of distilling advanced LLMs' language processing abilities to a smaller yet effective StruXGPT-7B to execute structurization, addressing the practicality of our approach. Code is available at https://github.com/alibaba/struxgpt.