A Comprehensive Analysis of Large Language Model Outputs: Similarity, Diversity, and Bias

TL;DR

This study analyzes the output similarity, diversity, and bias of 12 large language models across 5,000 prompts, revealing differences in variability, style, and ethical considerations to inform future AI development.

Contribution

It provides a comprehensive comparison of LLM output similarity, diversity, and bias across multiple models and tasks, highlighting key behavioral differences.

Findings

Outputs from the same LLM are more similar than to human texts.

GPT-4 produces more varied responses than WizardLM-2-8x22b.

Some models show higher gender balance and reduced bias.

Abstract

Large Language Models (LLMs) represent a major step toward artificial general intelligence, significantly advancing our ability to interact with technology. While LLMs perform well on Natural Language Processing tasks -- such as translation, generation, code writing, and summarization -- questions remain about their output similarity, variability, and ethical implications. For instance, how similar are texts generated by the same model? How does this compare across different models? And which models best uphold ethical standards? To investigate, we used 5{,}000 prompts spanning diverse tasks like generation, explanation, and rewriting. This resulted in approximately 3 million texts from 12 LLMs, including proprietary and open-source systems from OpenAI, Google, Microsoft, Meta, and Mistral. Key findings include: (1) outputs from the same LLM are more similar to each other than to human-written texts; (2) models like WizardLM-2-8x22b generate highly similar outputs, while GPT-4 produces more varied responses; (3) LLM writing styles differ significantly, with Llama 3 and Mistral showing higher similarity, and GPT-4 standing out for distinctiveness; (4) differences in vocabulary and tone underscore the linguistic uniqueness of LLM-generated content; (5) some LLMs demonstrate greater gender balance and reduced bias. These results offer new insights into the behavior and diversity of LLM outputs, helping guide future development and ethical evaluation.