Human-Centric Autonomous Systems With LLMs for User Command Reasoning

TL;DR

This paper explores using Large Language Models to interpret and reason about user commands in autonomous driving, emphasizing the importance of prompt design and model quality for accurate understanding.

Contribution

It demonstrates the potential of LLMs for understanding user commands in autonomous systems and highlights the impact of prompt design on reasoning accuracy.

Findings

LLMs can understand and reason about user commands in autonomous driving.

Prompt design significantly affects LLM performance.

Model quality influences the accuracy of system requirement inference.

Abstract

The evolution of autonomous driving has made remarkable advancements in recent years, evolving into a tangible reality. However, a human-centric large-scale adoption hinges on meeting a variety of multifaceted requirements. To ensure that the autonomous system meets the user's intent, it is essential to accurately discern and interpret user commands, especially in complex or emergency situations. To this end, we propose to leverage the reasoning capabilities of Large Language Models (LLMs) to infer system requirements from in-cabin users' commands. Through a series of experiments that include different LLM models and prompt designs, we explore the few-shot multivariate binary classification accuracy of system requirements from natural language textual commands. We confirm the general ability of LLMs to understand and reason about prompts but underline that their effectiveness is conditioned on the quality of both the LLM model and the design of appropriate sequential prompts. Code and models are public with the link \url{https://github.com/KTH-RPL/DriveCmd_LLM}.