Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous Systems

TL;DR

This paper advocates for a neuro-symbolic, user-centered approach to multimodal learning in autonomous systems, emphasizing explicit and implicit human interactions for more human-like AI understanding.

Contribution

It introduces hypotheses and design guidelines for integrating multimodal, human-in-the-loop, and incremental learning techniques in autonomous systems.

Findings

Proposes a framework combining neuro-symbolic and user-centered learning.

Highlights the importance of multimodal inputs for AI understanding.

Suggests design principles for human-in-the-loop autonomous systems.

Abstract

Recent advances in machine learning, particularly deep learning, have enabled autonomous systems to perceive and comprehend objects and their environments in a perceptual subsymbolic manner. These systems can now perform object detection, sensor data fusion, and language understanding tasks. However, there is a growing need to enhance these systems to understand objects and their environments more conceptually and symbolically. It is essential to consider both the explicit teaching provided by humans (e.g., describing a situation or explaining how to act) and the implicit teaching obtained by observing human behavior (e.g., through the system's sensors) to achieve this level of powerful artificial intelligence. Thus, the system must be designed with multimodal input and output capabilities to support implicit and explicit interaction models. In this position paper, we argue for considering both types of inputs, as well as human-in-the-loop and incremental learning techniques, for advancing the field of artificial intelligence and enabling autonomous systems to learn like humans. We propose several hypotheses and design guidelines and highlight a use case from related work to achieve this goal.