Designing a Multimodal Viewer for Piano Performance Analysis -- a Pedagogy-First Approach

TL;DR

This paper presents a web-based multimodal viewer combining video, motion capture, and scores to improve piano pedagogy by providing concrete visual feedback, addressing interpretative variability in verbal instructions.

Contribution

It introduces a novel multimodal dashboard prototype that enhances piano teaching by integrating multiple data modalities for clearer instructor feedback.

Findings

Validated technical feasibility with 109 datasets

Identified seven core pedagogical need groups

Enhanced clarity of instructor feedback through multimodal visualization

Abstract

Abstract instructions in piano education, such as "raise your wrist" and "relax your tension," lead to varying interpretations among learners, preventing instructors from effectively conveying their intended pedagogical guidance. To address this problem, this study conducted systematic interviews with a piano professor with 18 years teaching experience, and two researchers derived seven core need groups through cross-validation. Based on these findings, we developed a web-based dashboard prototype integrating video, motion capture, and musical scores, enabling instructors to provide concrete, visual feedback instead of relying solely on abstract verbal instructions. Technical feasibility was validated through 109 performance datasets.