herakoi: a sonification experiment for astronomical data
Michele Ginolfi, Luca Di Mascolo, Anita Zanella

TL;DR
Herakoi is an open-source software that uses machine learning and hand-tracking to convert astronomical images into sound in real-time, enhancing accessibility and engagement for diverse users, including those with visual impairments.
Contribution
The paper introduces herakoi, a novel real-time image sonification tool leveraging machine learning and hand-tracking for astronomical data interpretation.
Findings
High responsiveness and reliability in real-time sonification
Effective in educational and outreach contexts
Accessible to blind and visually impaired users
Abstract
Recent research is revealing data-sonification as a promising complementary approach to vision, benefiting both data perception and interpretation. We present herakoi, a novel open-source software that uses machine learning to allow real-time image sonification, with a focus on astronomical data. By tracking hand movements via a webcam and mapping them to image coordinates, herakoi translates visual properties into sound, enabling users to "hear" images. Its swift responsiveness allows users to access information in astronomical images with short training, demonstrating high reliability and effectiveness. The software has shown promise in educational and outreach settings, making complex astronomical concepts more engaging and accessible to diverse audiences, including blind and visually impaired individuals. We also discuss future developments, such as the integration of large language…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTime Series Analysis and Forecasting
