MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution

TL;DR

MAGNET is an adaptive GUI agent framework that uses dual-level memory to maintain robustness and generalization despite frequent UI changes, by focusing on stable semantics and task intents.

Contribution

The paper introduces a memory-driven framework with dynamic memory evolution for GUI agents, enhancing robustness against interface changes.

Findings

Significant performance improvements over baselines.

Robustness under distribution shifts.

Effective knowledge prioritization through memory evolution.

Abstract

Mobile GUI agents powered by large foundation models enable autonomous task execution, but frequent updates altering UI appearance and reorganizing workflows cause agents trained on historical data to fail. Despite surface changes, functional semantics and task intents remain fundamentally stable. Building on this insight, we introduce MAGNET, a memory-driven adaptive agent framework with dual-level memory: stationary memory linking diverse visual features to stable functional semantics for robust action grounding and procedural memory capturing stable task intents across varying workflows. We propose a dynamic memory evolution mechanism that continuously refines both memories by prioritizing frequently accessed knowledge. Online benchmark AndroidWorld evaluations show substantial improvements over baselines, while offline benchmarks confirm consistent gains under distribution shifts. These results validate that leveraging stable structures across interface changes improves agent performance and generalization in evolving software environments.