Exploring Human-AI Conceptual Alignment through the Prism of Chess

TL;DR

This study investigates whether a chess-playing AI truly understands human strategic concepts by analyzing layer-wise representations, revealing a divergence between human-aligned concepts and performance-driven representations.

Contribution

It introduces the first Chess960 dataset for testing conceptual understanding and provides a layer-wise analysis showing the divergence between human concepts and AI performance.

Findings

Early layers encode human concepts with high accuracy

Deeper layers drift toward alien representations despite better performance

Removing opening theory reduces concept recognition by 10-20%

Abstract

Do AI systems truly understand human concepts or merely mimic surface patterns? We investigate this through chess, where human creativity meets precise strategic concepts. Analyzing a 270M-parameter transformer that achieves grandmaster-level play, we uncover a striking paradox: while early layers encode human concepts like center control and knight outposts with up to 85\% accuracy, deeper layers, despite driving superior performance, drift toward alien representations, dropping to 50-65\% accuracy. To test conceptual robustness beyond memorization, we introduce the first Chess960 dataset: 240 expert-annotated positions across 6 strategic concepts. When opening theory is eliminated through randomized starting positions, concept recognition drops 10-20\% across all methods, revealing the model's reliance on memorized patterns rather than abstract understanding. Our layer-wise analysis exposes a fundamental tension in current architectures: the representations that win games diverge from those that align with human thinking. These findings suggest that as AI systems optimize for performance, they develop increasingly alien intelligence, a critical challenge for creative AI applications requiring genuine human-AI collaboration. Dataset and code are available at: https://github.com/slomasov/ChessConceptsLLM.