Is This a Bad Table? A Closer Look at the Evaluation of Table Generation from Text

TL;DR

This paper introduces TabEval, a new semantic evaluation method for generated tables that correlates better with human judgments by translating tables into natural language statements and assessing their entailment.

Contribution

The paper proposes TabEval, a novel semantic evaluation strategy for tables that improves over existing metrics by capturing overall table semantics through natural language statements.

Findings

TabEval shows stronger correlation with human judgments than existing metrics.

A new dataset of 1,250 Wikipedia tables with descriptions was curated for validation.

TabEval outperforms traditional metrics across multiple datasets.

Abstract

Understanding whether a generated table is of good quality is important to be able to use it in creating or editing documents using automatic methods. In this work, we underline that existing measures for table quality evaluation fail to capture the overall semantics of the tables, and sometimes unfairly penalize good tables and reward bad ones. We propose TabEval, a novel table evaluation strategy that captures table semantics by first breaking down a table into a list of natural language atomic statements and then compares them with ground truth statements using entailment-based measures. To validate our approach, we curate a dataset comprising of text descriptions for 1,250 diverse Wikipedia tables, covering a range of topics and structures, in contrast to the limited scope of existing datasets. We compare TabEval with existing metrics using unsupervised and supervised text-to-table generation methods, demonstrating its stronger correlation with human judgments of table quality across four datasets.