Likelihood Variance as Text Importance for Resampling Texts to Map Language Models

TL;DR

This paper introduces a resampling technique based on likelihood variance to efficiently select important texts, reducing computational costs in constructing language model comparison maps while maintaining accuracy.

Contribution

It proposes a novel likelihood variance-based resampling method that decreases the number of texts needed for accurate KL divergence estimation in language model mapping.

Findings

Reduces required texts by about 50% compared to uniform sampling.

Maintains comparable accuracy in KL divergence estimates.

Enables efficient updating of language model maps with new models.

Abstract

We address the computational cost of constructing a model map, which embeds diverse language models into a common space for comparison via KL divergence. The map relies on log-likelihoods over a large text set, making the cost proportional to the number of texts. To reduce this cost, we propose a resampling method that selects important texts with weights proportional to the variance of log-likelihoods across models for each text. Our method significantly reduces the number of required texts while preserving the accuracy of KL divergence estimates. Experiments show that it achieves comparable performance to uniform sampling with about half as many texts, and also facilitates efficient incorporation of new models into an existing map. These results enable scalable and efficient construction of language model maps.