WarpGate: A Semantic Join Discovery System for Cloud Data Warehouses
Tianji Cong, James Gale, Jason Frantz, H. V. Jagadish and, \c{C}a\u{g}atay Demiralp

TL;DR
WarpGate is a system that enables users to discover semantically joinable tables across cloud data warehouses by encoding columns into a high-dimensional space, improving data discovery and analysis.
Contribution
WarpGate introduces an embedding-based approach for semantic join discovery, effectively capturing relationships across large, diverse tables in cloud data warehouses.
Findings
Effectively captures semantic relationships between tables across databases
Scalable to very large tables with millions of rows
Demonstrates practical application within enterprise analytics
Abstract
Data discovery is a major challenge in enterprise data analysis: users often struggle to find data relevant to their analysis goals or even to navigate through data across data sources, each of which may easily contain thousands of tables. One common user need is to discover tables joinable with a given table. This need is particularly critical because join is a ubiquitous operation in data analysis, and join paths are mostly obscure to users, especially across databases. Furthermore, users are typically interested in finding ``semantically'' joinable tables: with columns that can be transformed to become joinable even if they are not joinable as currently represented in the data store. We present WarpGate, a system prototype for data discovery over cloud data warehouses. WarpGate implements an embedding-based solution to semantic join discovery, which encodes columns into…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsData Quality and Management · Big Data and Business Intelligence · Advanced Database Systems and Queries
