Yanyun-3: Enabling Cross-Platform Strategy Game Operation with Vision-Language Models

TL;DR

Yanyun-3 is a vision-language model-based agent that enables cross-platform strategy game automation by integrating visual reasoning and interface execution, achieving significant performance improvements and generalization without platform-specific tuning.

Contribution

We introduce Yanyun-3, a novel framework combining structured multimodal data organization and fine-tuning techniques to enhance VLM performance in cross-platform game automation.

Findings

12.98x BLEU-4 score improvement

63% reduction in inference time

Effective cross-platform task execution

Abstract

Cross-platform strategy game automation remains a challenge due to diverse user interfaces and dynamic battlefield environments. Existing Vision--Language Models (VLMs) struggle with generalization across heterogeneous platforms and lack precision in interface understanding and action execution. We introduce Yanyun-3, a VLM-based agent that integrates Qwen2.5-VL for visual reasoning and UI-TARS for interface execution. We propose a novel data organization principle -- combination granularity -- to distinguish intra-sample fusion and inter-sample mixing of multimodal data (static images, multi-image sequences, and videos). The model is fine-tuned using QLoRA on a curated dataset across three strategy game platforms. The optimal strategy (M*V+S) achieves a 12.98x improvement in BLEU-4 score and a 63% reduction in inference time compared to full fusion. Yanyun-3 successfully executes core tasks (e.g., target selection, resource allocation) across platforms without platform-specific tuning. Our findings demonstrate that structured multimodal data organization significantly enhances VLM performance in embodied tasks. Yanyun-3 offers a generalizable framework for GUI automation, with broader implications for robotics and autonomous systems.