IQ Test for LLMs: An Evaluation Framework for Uncovering Core Skills in LLMs

TL;DR

This paper introduces a new evaluation framework for large language models that uses factor analysis to uncover core skills, providing a more interpretable and comprehensive assessment than traditional benchmark scores.

Contribution

It proposes a novel factor analysis-based paradigm to identify latent skills in LLMs and applies it to a large benchmark, offering practical tools for model profiling and task redundancy detection.

Findings

Identified a small set of latent skills explaining most performance variance.

Revealed redundancies among benchmark tasks.

Provided tools for better model comparison and skill profiling.

Abstract

Current evaluations of large language models (LLMs) rely on benchmark scores, but it is difficult to interpret what these individual scores reveal about a model's overall skills. Specifically, as a community we lack understanding of how tasks relate to one another, what they measure in common, how they differ, or which ones are redundant. As a result, models are often assessed via a single score averaged across benchmarks, an approach that fails to capture the models' wholistic strengths and limitations. Here, we propose a new evaluation paradigm that uses factor analysis to identify latent skills driving performance across benchmarks. We apply this method to a comprehensive new leaderboard showcasing the performance of 60 LLMs on 44 tasks, and identify a small set of latent skills that largely explain performance. Finally, we turn these insights into practical tools that identify redundant tasks, aid in model selection, and profile models along each latent skill.