GEMINI: A Generic Multi-Modal Natural Interface Framework for Videogames
Lu\'is Filipe Te\'ofilo, Pedro Alves Nogueira, Pedro Brand\~ao Silva

TL;DR
GEMINI is a versatile middleware framework that enables natural multi-modal control of video games through body poses and voice commands, enhancing user engagement and accessibility.
Contribution
It introduces a flexible, real-time virtual controller system that maps body and voice inputs to any game, improving interaction naturalness and customization.
Findings
System effectively recognizes Kinect body poses and voice commands in real time.
User reports indicate increased engagement and satisfaction.
Framework demonstrates compatibility with various game controllers like Wiimote.
Abstract
In recent years videogame companies have recognized the role of player engagement as a major factor in user experience and enjoyment. This encouraged a greater investment in new types of game controllers such as the WiiMote, Rock Band instruments and the Kinect. However, the native software of these controllers was not originally designed to be used in other game applications. This work addresses this issue by building a middleware framework, which maps body poses or voice commands to actions in any game. This not only warrants a more natural and customized user-experience but it also defines an interoperable virtual controller. In this version of the framework, body poses and voice commands are respectively recognized through the Kinect's built-in cameras and microphones. The acquired data is then translated into the native interaction scheme in real time using a lightweight method…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsVideo Analysis and Summarization · Interactive and Immersive Displays · Tactile and Sensory Interactions
