audio2chart: End to End Audio Transcription into playable Guitar Hero charts

TL;DR

audio2chart is a novel framework that automatically generates Guitar Hero charts from raw audio, using sequence prediction models that are improved by audio conditioning, with publicly available code and pretrained models.

Contribution

This work introduces a new end-to-end neural approach for automatic chart generation directly from audio, demonstrating the effectiveness of audio conditioning and providing reproducible resources.

Findings

Unconditional baseline achieves strong performance.

Audio conditioning improves note prediction accuracy.

Public release of code and pretrained models supports reproducibility.

Abstract

This work introduces audio2chart, a framework for the automatic generation of Guitar Hero style charts directly from raw audio. The task is formalized as a sequence prediction problem, where models are trained to generate discrete chart tokens aligned with the audio on discrete time steps. An unconditional baseline demonstrates strong predictive performance, while the addition of audio conditioning yields consistent improvements across accuracy based metrics. This work demonstrates that incorporating audio conditioning is both feasible and effective for improving note prediction in automatic chart generation. The complete codebase for training and inference is publicly available on GitHub supporting reproducible research on neural chart generation. A family of pretrained models is released on Hugging Face.