The Renaissance of Expert Systems: Optical Recognition of Printed Chinese Jianpu Musical Scores with Lyrics

TL;DR

This paper introduces a hybrid expert-system approach for optical recognition of printed Chinese Jianpu musical scores with lyrics, converting them into digital formats without extensive training data, and demonstrating high accuracy on large folk song datasets.

Contribution

It presents a novel modular pipeline combining traditional computer vision and unsupervised deep learning for Chinese Jianpu score recognition, addressing a gap in OMR research.

Findings

Recognizes over 5,000 melody-only songs with high accuracy.

Achieves note-wise F1 score of 0.951 for melody recognition.

Achieves character-wise F1 score of 0.931 for lyrics alignment.

Abstract

Large-scale optical music recognition (OMR) research has focused mainly on Western staff notation, leaving Chinese Jianpu (numbered notation) and its rich lyric resources underexplored. We present a modular expert-system pipeline that converts printed Jianpu scores with lyrics into machine-readable MusicXML and MIDI, without requiring massive annotated training data. Our approach adopts a top-down expert-system design, leveraging traditional computer-vision techniques (e.g., phrase correlation, skeleton analysis) to capitalize on prior knowledge, while integrating unsupervised deep-learning modules for image feature embeddings. This hybrid strategy strikes a balance between interpretability and accuracy. Evaluated on The Anthology of Chinese Folk Songs, our system massively digitizes (i) a melody-only collection of more than 5,000 songs (> 300,000 notes) and (ii) a curated subset with lyrics comprising over 1,400 songs (> 100,000 notes). The system achieves high-precision recognition on both melody (note-wise F1 = 0.951) and aligned lyrics (character-wise F1 = 0.931).