Intent-Aware Schema Generation And Refinement For Literature Review Tables

TL;DR

This paper introduces a novel approach for generating and refining schemas for literature review tables using LLMs, addressing ambiguity and editing challenges, and demonstrates improved performance through synthesized intents and refinement techniques.

Contribution

It is the first to augment unannotated tables with synthesized intents and develop schema refinement methods, enhancing schema generation for literature reviews.

Findings

Synthesized intents reduce ambiguity in schema generation.

Fine-tuned small models can match large prompted LLMs.

Refinement techniques further improve schema quality.

Abstract

The increasing volume of academic literature makes it essential for researchers to organize, compare, and contrast collections of documents. Large language models (LLMs) can support this process by generating schemas defining shared aspects along which to compare papers. However, progress on schema generation has been slow due to: (i) ambiguity in reference-based evaluations, and (ii) lack of editing/refinement methods. Our work is the first to address both issues. First, we present an approach for augmenting unannotated table corpora with \emph{synthesized intents}, and apply it to create a dataset for studying schema generation conditioned on a given information need, thus reducing ambiguity. With this dataset, we show how incorporating table intents significantly improves baseline performance in reconstructing reference schemas. We start by comprehensively benchmarking several single-shot schema generation methods, including prompted LLM workflows and fine-tuned models, showing that smaller, open-weight models can be fine-tuned to be competitive with state-of-the-art prompted LLMs. Next, we propose several LLM-based schema refinement techniques and show that these can further improve schemas generated by these methods.