An Automated Multi-modal Evaluation Framework for Mobile Intelligent Assistants Based on Large Language Models and Multi-Agent Collaboration

TL;DR

This paper introduces an automated multi-modal evaluation framework for mobile intelligent assistants that leverages large language models and multi-agent collaboration to improve accuracy and reduce manual effort.

Contribution

It presents a novel three-tier agent architecture and fine-tunes a large language model to automate evaluation, outperforming traditional manual methods.

Findings

Achieves high evaluation accuracy comparable to human experts.

Effectively predicts user satisfaction and detects generation defects.

Validated on eight major intelligent agents.

Abstract

With the rapid development of mobile intelligent assistant technologies, multi-modal AI assistants have become essential interfaces for daily user interactions. However, current evaluation methods face challenges including high manual costs, inconsistent standards, and subjective bias. This paper proposes an automated multi-modal evaluation framework based on large language models and multi-agent collaboration. The framework employs a three-tier agent architecture consisting of interaction evaluation agents, semantic verification agents, and experience decision agents. Through supervised fine-tuning on the Qwen3-8B model, we achieve a significant evaluation matching accuracy with human experts. Experimental results on eight major intelligent agents demonstrate the framework's effectiveness in predicting users' satisfaction and identifying generation defects.