FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents

TL;DR

FormFactory introduces a comprehensive benchmark suite for evaluating multimodal large language models on the complex task of automated form filling, highlighting current models' limitations and guiding future research.

Contribution

We define the form-filling task formally and develop a benchmark with a web interface, dataset, and evaluation tools to assess model performance on real-world scenarios.

Findings

No model exceeds 5% accuracy on the benchmark

Current models struggle with visual layout reasoning

Field-value alignment remains a significant challenge

Abstract

Online form filling is a common yet labor-intensive task involving extensive keyboard and mouse interactions. Despite the long-standing vision of automating this process with "one click", existing tools remain largely rule-based and lack generalizable, generative capabilities. Recent advances in Multimodal Large Language Models (MLLMs) have enabled promising agents for GUI-related tasks in general-purpose scenarios. However, they struggle with the unique challenges of form filling, such as flexible layouts and the difficulty of aligning textual instructions with on-screen fields. To bridge this gap, we formally define the form-filling task and propose FormFactory, an interactive benchmarking suite comprising a web-based interface, backend evaluation module, and carefully constructed dataset. Our benchmark covers diverse real-world scenarios, incorporates various field formats, and simulates high-fidelity form interactions. We conduct a comprehensive evaluation of state-of-the-art MLLMs and observe that no model surpasses 5% accuracy, underscoring the inherent difficulty of the task. These findings also reveal significant limitations in current models' visual layout reasoning and field-value alignment abilities. We hope our benchmark can serve as a stepping stone for further research into robust, practical form-filling agents.