A novel processing pipeline for optical multi-touch surfaces

TL;DR

This thesis introduces a new optical multi-touch processing pipeline that detects and clusters fingertips using extremal regions, enabling real-time, robust multi-user two-handed interaction on touch surfaces.

Contribution

It presents a novel approach utilizing extremal regions for fingertip detection and hierarchical clustering for hand identification, improving robustness and real-time performance.

Findings

Robust detection despite non-uniform illumination

Reliable hand and finger identification

Real-time multi-user two-handed input handling

Abstract

In this thesis a new approach for touch detection on optical multi-touch devices is proposed that exploits the fact that the camera images reveal not only the actual touch points but also objects above the screen such as the hand or arm of a user. The touch processing relies on the Maximally Stable Extremal Regions algorithm for finding the users' fingertips in the camera image. The hierarchical structure of the generated extremal regions serves as a starting point for agglomerative clustering of the fingertips into hands. Furthermore, a heuristic is suggested that supports the identification of individual fingers as well as the distinction between left hands and right hands if all five fingers of a hand are in contact with the touch surface. The evaluation confirmed that the system is robust against detection errors resulting from non-uniform illumination and reliably assigns touch points to individual hands based on the implicitly tracked context information. The efficient multi-threaded implementation handles two-handed input from multiple users in real-time.