
TL;DR
The paper introduces Twinning, a fast and efficient method for partitioning large datasets into statistically similar subsets, enhancing data splitting tasks like cross-validation and data compression.
Contribution
Twinning is a novel, significantly faster dataset partitioning method based on SPlit, suitable for Big Data applications and multiple data splitting scenarios.
Findings
Twinning is orders of magnitude faster than SPlit.
It effectively generates multiple statistically similar dataset splits.
Applicable to large-scale data partitioning and cross-validation.
Abstract
In this work, we develop a method named Twinning, for partitioning a dataset into statistically similar twin sets. Twinning is based on SPlit, a recently proposed model-independent method for optimally splitting a dataset into training and testing sets. Twinning is orders of magnitude faster than the SPlit algorithm, which makes it applicable to Big Data problems such as data compression. Twinning can also be used for generating multiple splits of a given dataset to aid divide-and-conquer procedures and -fold cross validation.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
