OmniStruct: Universal Text-to-Structure Generation across Diverse Schemas

TL;DR

OmniStruct introduces a benchmark and approach for training smaller models to generate diverse structured outputs from text, matching GPT-4's performance without supervised data.

Contribution

The paper presents OmniStruct, a unified benchmark for text-to-structure tasks, and demonstrates training smaller models with synthetic data to achieve competitive results.

Findings

Smaller models can rival GPT-4 performance on structured tasks.

Synthetic data enables effective training without supervised datasets.

OmniStruct covers diverse real-world structured generation tasks.

Abstract

The ability of Large Language Models (LLMs) to generate structured outputs that follow arbitrary schemas is crucial to a wide range of downstream tasks that require diverse structured representations of results such as information extraction, table generation, and function calling. While modern LLMs excel in generating unstructured responses in natural language, whether this advancement translates to a strong performance on text-to-structure tasks remains unclear. To bridge this gap, we first introduce OmniStruct, a comprehensive benchmark for assessing LLMs' capabilities on diverse text-to-structure tasks such as information extraction, table generation, and function calling. We build OmniStruct by identifying existing datasets across a wide range of tasks that are suitable for a structured answer format, and adapting them under a unified text-to-structure problem setting. To facilitate the development of efficient text-to-structure models, we collect high-quality training data via synthetic task generation. Without using any supervised data for OmniStruct tasks, our experiments demonstrate the possibility of fine-tuning much smaller models on synthetic data into universal structured generation models that can rival the performance of GPT-4o.