AnimeScore: A Preference-Based Dataset and Framework for Evaluating Anime-Like Speech Style

TL;DR

AnimeScore introduces a preference-based, pairwise ranking framework for objectively evaluating anime-like speech, addressing the lack of standardized metrics and revealing acoustic features that influence perceived anime-likeness.

Contribution

The paper presents AnimeScore, a novel preference-based evaluation framework and dataset for anime-like speech, with insights into acoustic features affecting perceived anime-likeness.

Findings

SSL-based ranking models outperform handcrafted features in AUC.

Perceived anime-likeness is influenced by resonance shaping, prosodic continuity, and articulation.

Handcrafted features reach a 69.3% AUC ceiling, while models achieve 90.8%.

Abstract

Evaluating 'anime-like' voices currently relies on costly subjective judgments, yet no standardized objective metric exists. A key challenge is that anime-likeness, unlike naturalness, lacks a shared absolute scale, making conventional Mean Opinion Score (MOS) protocols unreliable. To address this gap, we propose AnimeScore, a preference-based framework for automatic anime-likeness evaluation via pairwise ranking. We collect 15,000 pairwise judgments from 187 evaluators with free-form descriptions, and acoustic analysis reveals that perceived anime-likeness is driven by controlled resonance shaping, prosodic continuity, and deliberate articulation rather than simple heuristics such as high pitch. We show that handcrafted acoustic features reach a 69.3% AUC ceiling, while SSL-based ranking models achieve up to 90.8% AUC, providing a practical metric that can also serve as a reward signal for preference-based optimization of generative speech models.