Towards More Effective Table-to-Text Generation: Assessing In-Context Learning and Self-Evaluation with Open-Source Models

TL;DR

This paper investigates how in-context learning and self-evaluation strategies in open-source language models can improve table-to-text generation, highlighting the importance of examples and the potential of self-assessment methods.

Contribution

It provides an in-depth analysis of in-context learning effects and introduces a self-evaluation approach for open-source models in table-to-text tasks.

Findings

Examples significantly improve generation quality

Self-evaluation shows potential but needs better alignment with human judgment

Open-source models can be effective with proper prompting

Abstract

Table processing, a key task in natural language processing, has significantly benefited from recent advancements in language models (LMs). However, the capabilities of LMs in table-to-text generation, which transforms structured data into coherent narrative text, require an in-depth investigation, especially with current open-source models. This study explores the effectiveness of various in-context learning strategies in LMs across benchmark datasets, focusing on the impact of providing examples to the model. More importantly, we examine a real-world use case, offering valuable insights into practical applications. To complement traditional evaluation metrics, we employ a large language model (LLM) self-evaluation approach using chain-of-thought reasoning and assess its correlation with human-aligned metrics like BERTScore. Our findings highlight the significant impact of examples in improving table-to-text generation and suggest that, while LLM self-evaluation has potential, its current alignment with human judgment could be enhanced. This points to the need for more reliable evaluation methods.