When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation

TL;DR

This paper constructs a large dataset for long-form literary review generation based on TTCW scores, revealing that reasoning supervision may hinder performance in fixed-format rubric-based review tasks.

Contribution

It introduces a new dataset of over 260,000 stories with TTCW-based annotations and evaluates fine-tuned Qwen3 models, highlighting the limited benefits of reasoning supervision.

Findings

Non-reasoning fine-tuning outperforms reasoning-supervised models.

Reasoning supervision increases parse failures and irrelevant outputs.

Fixed-format review generation remains challenging despite fine-tuning.

Abstract

Automatic evaluation of long-form literary writing remains challenging, as generic LLM-as-Judge approaches may not fully capture creativity-related dimensions such as originality and flexibility. Although the Torrance Test of Creative Writing (TTCW) provides a structured creativity framework, and prior work has demonstrated reference-based TTCW evaluation at the pairwise level, no large-scale dataset exists for long-form TTCW-based literary review generation. We address this gap by constructing a dataset of 263,911 long-form stories, each annotated with scalar scores and meta-synthesised review comments across 14 TTCW-based dimensions. Using this dataset, we fine-tune Qwen3 models at two scales, 4B and 8B, under two conditions: with and without reasoning content. Results show that non-reasoning fine-tuning achieves stronger and more stable performance, with the best setting reaching an evaluation score of 0.6820. Further analysis shows that reasoning-supervised models are more prone to parse failures, often continuing with irrelevant or repetitive reasoning-style text rather than completing the required 14-metric review report. These results suggest that, for fixed-format rubric-based review generation, reasoning supervision is not straightforwardly beneficial, and precise metric-aligned scoring remains challenging even after task-specific fine-tuning.