Supporting Out-of-turn Interactions in a Multimodal Web Interface

TL;DR

This paper presents a software framework that integrates hyperlink and speech interactions in multimodal web interfaces, enabling out-of-turn interactions for more adaptive and flexible web browsing experiences.

Contribution

It introduces a novel framework that supports out-of-turn speech interactions in multimodal web interfaces, enhancing user flexibility beyond traditional hyperlink navigation.

Findings

Framework successfully integrates speech and hyperlink modes.

Enables out-of-turn interactions for adaptive web browsing.

Demonstrated through two case studies.

Abstract

Multimodal interfaces are becoming increasingly important with the advent of mobile devices, accessibility considerations, and novel software technologies that combine diverse interaction media. This article investigates systems support for web browsing in a multimodal interface. Specifically, we outline the design and implementation of a software framework that integrates hyperlink and speech modes of interaction. Instead of viewing speech as merely an alternative interaction medium, the framework uses it to support out-of-turn interaction, providing a flexibility of information access not possible with hyperlinks alone. This approach enables the creation of websites that adapt to the needs of users, yet permits the designer fine-grained control over what interactions to support. Design methodology, implementation details, and two case studies are presented.