Tracking Ensemble Performance on Touch-Screens with Gesture Classification and Transition Matrices

TL;DR

This paper introduces a real-time touch-screen gesture analysis system using Random Forests and transition matrices to evaluate ensemble performances and adapt musical interfaces during live performances.

Contribution

It presents a novel system combining gesture classification and transition analysis for ensemble performance tracking on touch-screens, with real-time adaptive capabilities.

Findings

Effective gesture classification with Random Forests

Successful real-time performance analysis during concerts

Enhanced musical interface responsiveness based on gesture transitions

Abstract

We present and evaluate a novel interface for tracking ensemble performances on touch-screens. The system uses a Random Forest classifier to extract touch-screen gestures and transition matrix statistics. It analyses the resulting gesture-state sequences across an ensemble of performers. A series of specially designed iPad apps respond to this real-time analysis of free-form gestural performances with calculated modifications to their musical interfaces. We describe our system and evaluate it through cross-validation and profiling as well as concert experience.