Foundation Models for Autonomous Robots in Unstructured Environments

TL;DR

This paper reviews how foundation models, especially Large Language Models, are being applied to enable autonomous robots in unstructured environments, highlighting current capabilities, challenges, and future directions.

Contribution

It systematically analyzes the application of foundation models in robotics for unstructured environments and synthesizes findings with deliberative acting theory, establishing a benchmark for progress.

Findings

LLMs enhance perception in human-robot interactions.

LLMs are used in project management and safety in construction.

Current state-of-the-art is at conditional automation level.

Abstract

Automating activities through robots in unstructured environments, such as construction sites, has been a long-standing desire. However, the high degree of unpredictable events in these settings has resulted in far less adoption compared to more structured settings, such as manufacturing, where robots can be hard-coded or trained on narrowly defined datasets. Recently, pretrained foundation models, such as Large Language Models (LLMs), have demonstrated superior generalization capabilities by providing zero-shot solutions for problems do not present in the training data, proposing them as a potential solution for introducing robots to unstructured environments. To this end, this study investigates potential opportunities and challenges of pretrained foundation models from a multi-dimensional perspective. The study systematically reviews application of foundation models in two field of robotic and unstructured environment and then synthesized them with deliberative acting theory. Findings showed that linguistic capabilities of LLMs have been utilized more than other features for improving perception in human-robot interactions. On the other hand, findings showed that the use of LLMs demonstrated more applications in project management and safety in construction, and natural hazard detection in disaster management. Synthesizing these findings, we located the current state-of-the-art in this field on a five-level scale of automation, placing them at conditional automation. This assessment was then used to envision future scenarios, challenges, and solutions toward autonomous safe unstructured environments. Our study can be seen as a benchmark to track our progress toward that future.