LLMs and the Madness of Crowds

TL;DR

This paper analyzes the error patterns of large language models, revealing systematic correlations and unique behaviors that deepen understanding of their underlying structures and relationships.

Contribution

It introduces a taxonomy of LLM errors based on correlation analysis, highlighting non-random, model-specific error behaviors.

Findings

Errors are systematically correlated across models

A taxonomy categorizes models based on error similarities

Insights into the underlying structures of LLMs

Abstract

We investigate the patterns of incorrect answers produced by large language models (LLMs) during evaluation. These errors exhibit highly non-intuitive behaviors unique to each model. By analyzing these patterns, we measure the similarities between LLMs and construct a taxonomy that categorizes them based on their error correlations. Our findings reveal that the incorrect responses are not randomly distributed but systematically correlated across models, providing new insights into the underlying structures and relationships among LLMs.