# KG-RAG: Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generation

## TL;DR

KG-RAG enhances GUI agent decision-making by transforming UI Transition Graphs into knowledge graphs for efficient retrieval, leading to higher success rates and decision accuracy in mobile app tasks.

## Contribution

Introduces KG-RAG, a novel framework that converts UTGs into structured vector databases for improved real-time retrieval and decision-making in GUI agents.

## Key findings

- Achieves 75.8% success rate, 8.9% improvement over AutoDroid.
- Reaches 84.6% decision accuracy, 8.1% higher than previous methods.
- Reduces average task steps from 4.5 to 4.1.

## Abstract

Despite recent progress, Graphic User Interface (GUI) agents powered by Large Language Models (LLMs) struggle with complex mobile tasks due to limited app-specific knowledge. While UI Transition Graphs (UTGs) offer structured navigation representations, they are underutilized due to poor extraction and inefficient integration. We introduce KG-RAG, a Knowledge Graph-driven Retrieval-Augmented Generation framework that transforms fragmented UTGs into structured vector databases for efficient real-time retrieval. By leveraging an intent-guided LLM search method, KG-RAG generates actionable navigation paths, enhancing agent decision-making. Experiments across diverse mobile apps show that KG-RAG outperforms existing methods, achieving a 75.8% success rate (8.9% improvement over AutoDroid), 84.6% decision accuracy (8.1% improvement), and reducing average task steps from 4.5 to 4.1. Additionally, we present KG-Android-Bench and KG-Harmony-Bench, two benchmarks tailored to the Chinese mobile ecosystem for future research. Finally, KG-RAG transfers to web/desktop (+40% SR on Weibo-web; +20% on QQ Music-desktop), and a UTG cost ablation shows accuracy saturates at ~4h per complex app, enabling practical deployment trade-offs.