FM SO.P: A Progressive Task Mixture Framework with Automatic Evaluation for Cross-Domain SOP Understanding

TL;DR

This paper introduces FM SO.P, a progressive framework with automatic multi-agent evaluation that enhances cross-domain SOP understanding by staged learning of reasoning skills and adaptive testing, achieving high performance with fewer parameters.

Contribution

The paper presents a novel progressive task mixture approach and an automatic multi-agent evaluation system for improved SOP understanding across domains, with significant parameter efficiency.

Findings

Achieves 48.3% pass rate with 32B model on SOPBench

Outperforms baseline with a 7B model matching larger models' performance

Introduces adaptive evaluation for domain-specific SOP tasks

Abstract

Standard Operating Procedures (SOPs) are critical for enterprise operations, yet existing language models struggle with SOP understanding and cross-domain generalization. Current methods fail because joint training cannot differentiate between reasoning capabilities that SOP requires: terminology precision, sequential ordering, and constraint reasoning. We propose FM SO.P, solving these challenges through two novelties. First, we introduce progressive task mixtures that build capabilities by stages across three task types with cumulative data: concept disambiguation for terminology precision, action sequence understanding for procedural correctness, and scenario-aware graph reasoning for conditional logic. Second, we propose an automatic multi-agent evaluation system consisting of three agents that adaptively generate rubrics, stratified test sets, and rubric scoring, adapting to domains (e.g., temporal constraints for DMV, regulatory compliance for banking). Evaluated on SOPBench across seven domains (Bank, DMV, Healthcare, Market, University, Library, Hotel), FM SO.P achieves 48.3\% pass rate with our 32B model and 34.3\% with our opensource 7B model, matching Qwen-2.5-72B-Instruct baseline (34.4\%) with 10x fewer parameters.