Subjective Evaluation Profile Analysis of Science Fiction Short Stories and its Critical-Theoretical Significance

TL;DR

This study uses large language models as subjective critics to analyze Japanese science fiction stories, revealing diverse evaluation patterns and vocabularies that resemble human critical schools, highlighting the models' implicit value systems.

Contribution

It introduces a novel methodology for assessing LLMs as subjective literary critics and uncovers their individual evaluation characteristics and vocabularies.

Findings

Significant variation in evaluation consistency among models.

Identification of five distinct evaluation patterns.

Distinctive vocabularies for each model confirmed by TF-IDF analysis.

Abstract

This study positions large language models (LLMs) as "subjective literary critics" to explore aesthetic preferences and evaluation patterns in literary assessment. Ten Japanese science fiction short stories were translated into English and evaluated by six state-of-the-art LLMs across seven independent sessions. Principal component analysis and clustering techniques revealed significant variations in evaluation consistency ({\alpha} ranging from 1.00 to 0.35) and five distinct evaluation patterns. Additionally, evaluation variance across stories differed by up to 4.5-fold, with TF-IDF analysis confirming distinctive evaluation vocabularies for each model. Our seven-session within-day protocol using an original Science Fiction corpus strategically minimizes external biases, allowing us to observe implicit value systems shaped by RLHF and their influence on literary judgment. These findings suggest that LLMs may possess individual evaluation characteristics similar to human critical schools, rather than functioning as neutral benchmarkers.