ManzaiSet: A Multimodal Dataset of Viewer Responses to Japanese Manzai Comedy

TL;DR

ManzaiSet is a large, multimodal dataset capturing viewer responses to Japanese manzai comedy, enabling culturally aware emotion AI and personalized entertainment systems, and revealing viewer heterogeneity and response patterns.

Contribution

First large-scale multimodal dataset of Japanese manzai comedy viewer responses, addressing Western bias in affective computing and enabling culturally specific emotion AI.

Findings

Identified three distinct viewer types with heterogeneity in responses.

Viewer engagement increased over time, contradicting fatigue hypotheses.

Automated humor classification showed no significant differences across viewer types.

Abstract

We present ManzaiSet, the first large scale multimodal dataset of viewer responses to Japanese manzai comedy, capturing facial videos and audio from 241 participants watching up to 10 professional performances in randomized order (94.6 percent watched >= 8; analyses focus on n=228). This addresses the Western centric bias in affective computing. Three key findings emerge: (1) k means clustering identified three distinct viewer types: High and Stable Appreciators (72.8 percent, n=166), Low and Variable Decliners (13.2 percent, n=30), and Variable Improvers (14.0 percent, n=32), with heterogeneity of variance (Brown Forsythe p < 0.001); (2) individual level analysis revealed a positive viewing order effect (mean slope = 0.488, t(227) = 5.42, p < 0.001, permutation p < 0.001), contradicting fatigue hypotheses; (3) automated humor classification (77 instances, 131 labels) plus viewer level response modeling found no type wise differences after FDR correction. The dataset enables culturally aware emotion AI development and personalized entertainment systems tailored to non Western contexts.