VECSR: Virtually Embodied Common Sense Reasoning System

TL;DR

This paper introduces VECSR, a system that combines goal-directed reasoning with virtual environment simulations to improve autonomous agents' trustworthiness and explainability, addressing limitations of deep learning models.

Contribution

It presents a novel integration of s(CASP) reasoning with VirtualHome to enhance decision justification and reduce response times in autonomous agents.

Findings

s(CASP) effectively justifies instruction choices in virtual environments

The framework achieves comparable accuracy to GPT-4o in domestic scenarios

Optimizations reduce response time significantly for complex decision spaces

Abstract

The development of autonomous agents has seen a revival of enthusiasm due to the emergence of LLMs, such as GPT-4o. Deploying these agents in environments where they coexist with humans (e.g., as domestic assistants) requires special attention to trustworthiness and explainability. However, the use of LLMs and other deep learning models still does not resolve these key issues. Deep learning systems may hallucinate, be unable to justify their decisions as black boxes, or perform badly on unseen scenarios. In this work, we propose the use of s(CASP), a goal-directed common sense reasoner based on Answer Set Programming, to break down the high-level tasks of an autonomous agent into mid-level instructions while justifying the selection of these instructions. To validate its use in real applications we present a framework that integrates the reasoner into the VirtualHome simulator and compares its accuracy with GPT-4o, running some of the "real" use cases available in the domestic environments of VirtualHome. Additionally, since experiments with VirtualHome have shown the need to reduce the response time (which increases as the agent's decision space grows), we have proposed and evaluated a series of optimizations based on program analysis that exploit the advantages of the top-down execution of s(CASP).