Something's Fishy In The Data Lake: A Critical Re-evaluation of Table Union Search Benchmarks
Allaa Boutaleb, Bernd Amann, Hubert Naacke, Rafael Angarita

TL;DR
This paper critically re-evaluates current benchmarks for table union search in data lakes, revealing their limitations and proposing criteria for more effective future evaluations of semantic understanding.
Contribution
It identifies key shortcomings in existing TUS benchmarks and offers essential criteria to improve their realism and reliability for future research.
Findings
Simple baselines outperform complex methods on current benchmarks.
Current benchmarks are heavily dataset-specific and do not isolate semantic understanding.
Proposes criteria to enhance future benchmark design.
Abstract
Recent table representation learning and data discovery methods tackle table union search (TUS) within data lakes, which involves identifying tables that can be unioned with a given query table to enrich its content. These methods are commonly evaluated using benchmarks that aim to assess semantic understanding in real-world TUS tasks. However, our analysis of prominent TUS benchmarks reveals several limitations that allow simple baselines to perform surprisingly well, often outperforming more sophisticated approaches. This suggests that current benchmark scores are heavily influenced by dataset-specific characteristics and fail to effectively isolate the gains from semantic understanding. To address this, we propose essential criteria for future benchmarks to enable a more realistic and reliable evaluation of progress in semantic table union search.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsData Quality and Management · Handwritten Text Recognition Techniques · Data-Driven Disease Surveillance
