The Rosetta Paradox: Domain-Specific Performance Inversions in Large Language Models

TL;DR

This paper investigates the Rosetta Paradox, an unexplored phenomenon where large language models perform inconsistently across domains, excelling in specialized fields but failing in general knowledge tasks, revealing intrinsic model properties.

Contribution

The paper formalizes the Rosetta Paradox, introduces a new analysis framework with DSI and PIM metrics, and provides extensive experimental insights into its manifestations across models and domains.

Findings

The paradox is likely an intrinsic property of neural networks, not just data distribution.

Model architecture and training methods influence the paradox's manifestation.

Standard evaluation metrics may not fully capture domain-specific performance issues.

Abstract

While large language models, such as GPT and BERT, have already demonstrated unprecedented skills in everything from natural language processing to domain-specific applications, there came an unexplored phenomenon we term the Rosetta Paradox. The Rosetta Paradox characterizes the counterintuitive performance inversions across domains of knowledge. This paradox captures how such LLMs can excel in highly specialized fields but do poorly on tasks which require general, everyday knowledge. This paper formalizes the definition of the Rosetta Paradox and introduces a panoramic analysis framework that includes both a Domain Specificity Index (DSI) and a Performance Inversion Metric (PIM) for consistent quantification of domain-specific behavior in LLMs. We adopt this paradox and conduct a series of investigations through extensive experiments across diverse models and knowledge domains, ranging from rich technical areas to common-sense reasoning. Our findings indicate that the Rosetta Paradox is likely not a mere artifact of data distribution but an intrinsic architectural and emergent property of deep neural networks. We present comparative analyses across different model architectures, sizes, and training methodologies that shed light into the peculiar ways this paradox manifests itself and challenge the standard evaluation metrics.