Jess+: designing embodied AI for interactive music-making

TL;DR

This paper presents Jess+, an embodied AI system designed for inclusive, interactive music-making that enables disabled and non-disabled musicians to collaborate seamlessly, transforming their musical practice through innovative design features.

Contribution

It introduces novel design features for embodied AI in music, including a modular system, an AI Factory with a musicking dataset, and an embedded belief system, supporting inclusive ensemble collaboration.

Findings

Rich musical experiences for diverse musicians

Transformation of inclusive ensemble practices

Effective support for disabled musicians in live settings

Abstract

In this paper, we discuss the conceptualisation and design of embodied AI within an inclusive music-making project. The central case study is Jess+ an intelligent digital score system for shared creativity with a mixed ensemble of non-disabled and disabled musicians. The overarching aim is that the digital score enables disabled musicians to thrive in a live music conversation with other musicians regardless of the potential barriers of disability and music-making. After defining what we mean by embodied AI and how this approach supports the aims of the Jess+ project, we outline the main design features of the system. This includes several novel approaches such as its modular design, an AI Factory based on an embodied musicking dataset, and an embedded belief system. Our findings showed that the implemented design decisions and embodied-AI approach led to rich experiences for the musicians which in turn transformed their practice as an inclusive ensemble.