GLEAN: Grounded Lightweight Evaluation Anchors for Contamination-Aware Tabular Reasoning

TL;DR

GLEAN is a lightweight, contamination-aware evaluation protocol for small models on tabular reasoning tasks, providing diagnostics and error attribution under hardware constraints.

Contribution

It introduces GLEAN, a novel evaluation framework that integrates contamination-aware probes, structured error attribution, and diagnostics for small models on tabular reasoning benchmarks.

Findings

GLEAN achieves 95.2% execution accuracy using Squall gold SQL as an anchor.

It reveals distinct error modes: grounding errors (L3) and hallucination/abstention errors (L2/L0).

Retrieval Recall@K can saturate even when end-to-end accuracy remains limited.

Abstract

Tabular reasoning benchmarks mix semantic inference, numerical computation, and brittle table formatting, yet evaluations for small models remain vulnerable to contamination, dataset artifacts, and retrieval failures. We propose GLEAN, a lightweight evaluation protocol that integrates contamination-aware probes, weak-supervision governance, retrieval-reasoning diagnostics, and structured error attribution under tight hardware constraints. We evaluate across TabFact, WTQ via Squall, TableBench, RobuT, and SciTab under a 16GB GPU budget. Using Squall gold SQL as an executable anchor (95.2% execution), GLEAN assigns a deterministic error taxonomy (L0-L4 plus L0.5 context miss) and reveals a stable error-mode separation: TAPEX errors skew toward grounding (L3) while TAPAS errors skew toward hallucination/abstention (L2/L0). We validate evidence-row heuristics against SQL-derived rows on simple queries (0.62 precision / 0.71 recall; hybrid recall 0.81) and show that retrieval Recall@K can saturate even when end-to-end EM/F1 remains limited, motivating attribution beyond raw recall. We release a modular framework with audits and sensitivity checks to make small-model tabular evaluation more contamination-aware and diagnostic.