LMdiff: A Visual Diff Tool to Compare Language Models

TL;DR

LMdiff is a visual tool that compares the output distributions of different language models, helping researchers understand their differences and generate hypotheses about model behavior through detailed, token-by-token analysis.

Contribution

The paper introduces LMdiff, a novel visual comparison tool for language models that facilitates hypothesis generation and analysis of model differences.

Findings

Enables detailed token-by-token comparison of models.

Assists in identifying interesting text instances for analysis.

Supports hypothesis generation across multiple case studies.

Abstract

While different language models are ubiquitous in NLP, it is hard to contrast their outputs and identify which contexts one can handle better than the other. To address this question, we introduce LMdiff, a tool that visually compares probability distributions of two models that differ, e.g., through finetuning, distillation, or simply training with different parameter sizes. LMdiff allows the generation of hypotheses about model behavior by investigating text instances token by token and further assists in choosing these interesting text instances by identifying the most interesting phrases from large corpora. We showcase the applicability of LMdiff for hypothesis generation across multiple case studies. A demo is available at http://lmdiff.net .