Investigating the Impact of Data Selection Strategies on Language Model Performance

TL;DR

This paper investigates how various data selection strategies, including n-gram and embedding-based features, impact language model performance and alignment with target distributions, providing practical insights for optimizing training data.

Contribution

It systematically compares different data selection methods and feature types, revealing their effects on language model performance and distribution alignment.

Findings

Data selection influences downstream task performance

N-gram features improve distribution alignment

Embedding-based features offer complementary benefits

Abstract

Data selection is critical for enhancing the performance of language models, particularly when aligning training datasets with a desired target distribution. This study explores the effects of different data selection methods and feature types on model performance. We evaluate whether selecting data subsets can influence downstream tasks, whether n-gram features improve alignment with target distributions, and whether embedding-based neural features provide complementary benefits. Through comparative experiments using baseline random selection methods and distribution aligned approaches, we provide insights into the interplay between data selection strategies and model training efficacy. All code for this study can be found on \href{https://github.com/jgu13/HIR-Hybrid-Importance-Resampling-for-Language-Models}{github repository}.