Every Software as an Agent: Blueprint and Case Study

TL;DR

This paper proposes a whitebox approach to software agents using LLMs with access to source code and runtime context, enabling more accurate and efficient software understanding and operation.

Contribution

It introduces a novel architecture allowing LLMs to interact directly with software internals and inject code dynamically, advancing beyond traditional API or GUI-based agents.

Findings

Demonstrated case studies on web-based desktop applications

Highlighted advantages of whitebox LLM integration over blackbox methods

Discussed challenges and future directions for software agent development

Abstract

The rise of (multimodal) large language models (LLMs) has shed light on software agent -- where software can understand and follow user instructions in natural language. However, existing approaches such as API-based and GUI-based agents are far from satisfactory at accuracy and efficiency aspects. Instead, we advocate to endow LLMs with access to the software internals (source code and runtime context) and the permission to dynamically inject generated code into software for execution. In such a whitebox setting, one may better leverage the software context and the coding ability of LLMs. We then present an overall design architecture and case studies on two popular web-based desktop applications. We also give in-depth discussion of the challenges and future directions. We deem that such a new paradigm has the potential to fundamentally overturn the existing software agent design, and finally creating a digital world in which software can comprehend, operate, collaborate, and even think to meet complex user needs.