2D-TPE: Two-Dimensional Positional Encoding Enhances Table Understanding for Large Language Models

TL;DR

This paper introduces 2D-TPE, a novel positional encoding method for large language models that preserves 2D table structure, significantly improving their ability to understand and reason over tabular data.

Contribution

The paper proposes 2D-TPE, a simple and effective 2D positional encoding technique that enhances LLMs' capacity to process structured tables without losing spatial information.

Findings

2D-TPE outperforms baseline methods on five benchmarks.

Preserves spatial relationships in tables more effectively.

Scales better to large tables than existing approaches.

Abstract

Tables are ubiquitous across various domains for concisely representing structured information. Empowering large language models (LLMs) to reason over tabular data represents an actively explored direction. However, since typical LLMs only support one-dimensional~(1D) inputs, existing methods often flatten the two-dimensional~(2D) table structure into a sequence of tokens, which can severely disrupt the spatial relationships and result in an inevitable loss of vital contextual information. In this paper, we first empirically demonstrate the detrimental impact of such flattening operations on the performance of LLMs in capturing the spatial information of tables through two elaborate proxy tasks. Subsequently, we introduce a simple yet effective positional encoding method, termed ``2D-TPE'' (Two-Dimensional Table Positional Encoding), to address this challenge. 2D-TPE enables each attention head to dynamically select a permutation order of tokens within the context for attending to them, where each permutation represents a distinct traversal mode for the table, such as column-wise or row-wise traversal. 2D-TPE effectively mitigates the risk of losing essential spatial information while preserving computational efficiency, thus better preserving the table structure. Extensive experiments across five benchmarks demonstrate that 2D-TPE outperforms strong baselines, underscoring the importance of preserving the table structure for accurate table comprehension. Comprehensive analysis further reveals the substantially better scalability of 2D-TPE to large tables than baselines.