Content Rating Classification for Fan Fiction

TL;DR

This paper explores automatic content rating classification for fan fiction using NLP techniques, highlighting challenges with multi-class accuracy and the impact of self-annotation quality.

Contribution

It introduces NLP-based methods for fan fiction content rating and analyzes the limitations of current self-annotated datasets.

Findings

Binary classification outperforms multi-class methods.

Self-annotation errors limit classification accuracy.

Traditional and deep learning methods show poor multi-class results.

Abstract

Content ratings can enable audiences to determine the suitability of various media products. With the recent advent of fan fiction, the critical issue of fan fiction content ratings has emerged. Whether fan fiction content ratings are done voluntarily or required by regulation, there is the need to automate the content rating classification. The problem is to take fan fiction text and determine the appropriate content rating. Methods for other domains, such as online books, have been attempted though none have been applied to fan fiction. We propose natural language processing techniques, including traditional and deep learning methods, to automatically determine the content rating. We show that these methods produce poor accuracy results for multi-classification. We then demonstrate that treating the problem as a binary classification problem produces better accuracy. Finally, we believe and provide some evidence that the current approach of self-annotating has led to incorrect labels limiting classification results.