CataractSurg-80K: Knowledge-Driven Benchmarking for Structured Reasoning in Ophthalmic Surgery Planning

TL;DR

This paper introduces CataractSurg-80K, a large-scale ophthalmic surgery planning benchmark and a domain-specific language model, Qwen-CSP, designed to improve structured reasoning and decision support in cataract surgery.

Contribution

It presents the first comprehensive cataract surgery planning benchmark with structured reasoning annotations and a specialized LLM, Qwen-CSP, tailored for ophthalmic clinical decision-making.

Findings

Qwen-CSP outperforms general LLMs on multiple metrics.

The benchmark enables rigorous evaluation of surgical reasoning.

The dataset facilitates future research in medical AI reasoning.

Abstract

Cataract surgery remains one of the most widely performed and effective procedures for vision restoration. Effective surgical planning requires integrating diverse clinical examinations for patient assessment, intraocular lens (IOL) selection, and risk evaluation. Large language models (LLMs) have shown promise in supporting clinical decision-making. However, existing LLMs often lack the domain-specific expertise to interpret heterogeneous ophthalmic data and provide actionable surgical plans. To enhance the model's ability to interpret heterogeneous ophthalmic reports, we propose a knowledge-driven Multi-Agent System (MAS), where each agent simulates the reasoning process of specialist ophthalmologists, converting raw clinical inputs into structured, actionable summaries in both training and deployment stages. Building on MAS, we introduce CataractSurg-80K, the first large-scale benchmark for cataract surgery planning that incorporates structured clinical reasoning. Each case is annotated with diagnostic questions, expert reasoning chains, and structured surgical recommendations. We further introduce Qwen-CSP, a domain-specialized model built on Qwen-4B, fine-tuned through a multi-stage process tailored for surgical planning. Comprehensive experiments show that Qwen-CSP outperforms strong general-purpose LLMs across multiple metrics. Our work delivers a high-quality dataset, a rigorous benchmark, and a domain-adapted LLM to facilitate future research in medical AI reasoning and decision support.

Figures

Tables

Table 1. Table 1: Performance of Baseline Models on the Cataract Surgery Benchmark Dataset (BLEU, ROUGE, BERTScore, Keypoint-level Metrics)

Model	Task	BLEU	ROUGE-L	BERTScoreF1	k-F1	k-Prec.	k-Rec.
GPT4.1	COT	0.039$\pm$0.018	0.197$\pm$0.029	0.865$\pm$0.013	0.225$\pm$0.038	0.230$\pm$0.039	0.230$\pm$0.051
GPT4.1	Response	0.068$\pm$0.025	0.224$\pm$0.040	0.867$\pm$0.014	0.259$\pm$0.050	0.224$\pm$0.038	0.325$\pm$0.078
Claude-3.7	COT	0.034$\pm$0.014	0.186$\pm$0.032	0.853$\pm$0.014	0.204$\pm$0.042	0.225$\pm$0.042	0.197$\pm$0.050
Claude-3.7	Response	0.040$\pm$0.018	0.206$\pm$0.038	0.854$\pm$0.016	0.231$\pm$0.054	0.243$\pm$0.041	0.226$\pm$0.063
Gemini-2	COT	0.023$\pm$0.012	0.160$\pm$0.028	0.846$\pm$0.017	0.196$\pm$0.042	0.197$\pm$0.042	0.201$\pm$0.051
Gemini-2	Response	0.020$\pm$0.022	0.141$\pm$0.064	0.830$\pm$0.021	0.169$\pm$0.096	0.216$\pm$0.077	0.194$\pm$0.141
DeepSeek	COT	0.054$\pm$0.018	0.212$\pm$0.023	0.876$\pm$0.011	0.207$\pm$0.030	0.224$\pm$0.034	0.194$\pm$0.033
DeepSeek	Response	0.019$\pm$0.013	0.182$\pm$0.025	0.856$\pm$0.010	0.170$\pm$0.034	0.162$\pm$0.035	0.182$\pm$0.039
Qwen-Max	COT	0.059$\pm$0.017	0.207$\pm$0.024	0.863$\pm$0.012	0.220$\pm$0.033	0.256$\pm$0.043	0.196$\pm$0.034
Qwen-Max	Response	0.098$\pm$0.038	0.249$\pm$0.049	0.873$\pm$0.015	0.270$\pm$0.054	0.246$\pm$0.050	0.312$\pm$0.074
Qwen-8B	COT	0.047$\pm$0.015	0.153$\pm$0.015	0.831$\pm$0.011	0.214$\pm$0.031	0.219$\pm$0.034	0.212$\pm$0.034
Qwen-8B	Response	0.041$\pm$0.012	0.173$\pm$0.020	0.837$\pm$0.012	0.218$\pm$0.037	0.170$\pm$0.028	0.311$\pm$0.066
Qwen-4B	COT	0.048$\pm$0.016	0.153$\pm$0.016	0.829$\pm$0.013	0.216$\pm$0.030	0.230$\pm$0.034	0.205$\pm$0.031
Qwen-4B	Response	0.039$\pm$0.012	0.166$\pm$0.023	0.836$\pm$0.013	0.222$\pm$0.039	0.179$\pm$0.030	0.302$\pm$0.064
Qwen-CSP	COT	0.131$\pm$0.029	0.261$\pm$0.034	0.886$\pm$0.015	0.292$\pm$0.043	0.317$\pm$0.046	0.275$\pm$0.045
Qwen-CSP	Response	0.170$\pm$0.041	0.330$\pm$0.044	0.896$\pm$0.013	0.367$\pm$0.054	0.377$\pm$0.062	0.361$\pm$0.051
Improvement	COT	122%	23.1%	1.14%	15.8%	23.8%	19.6%
Improvement	Response	73.4%	32.5%	2.63%	35.9%	53.3%	11.1%

Table 2. Table 2: Performance of Baseline Models on the Cataract Surgery Benchmark Dataset (Entity-level and NLI Consistency)

Model	Task	Entity F1	Entity Prec.	Entity Rec.	NLI Cons.
GPT4.1	COT	0.247$\pm$0.058	0.244$\pm$0.067	0.279$\pm$0.071	0.907$\pm$0.276
Claude-3.7	COT	0.208$\pm$0.062	0.246$\pm$0.076	0.215$\pm$0.078	0.619$\pm$0.466
Gemini-2	COT	0.214$\pm$0.059	0.211$\pm$0.064	0.252$\pm$0.077	0.916$\pm$0.264
DeepSeek	COT	0.201$\pm$0.065	0.245$\pm$0.077	0.204$\pm$0.069	0.201$\pm$0.362
Qwen-Max	COT	0.096$\pm$0.079	0.099$\pm$0.098	0.167$\pm$0.106	0.297$\pm$0.321
Qwen-8B	COT	0.221$\pm$0.057	0.233$\pm$0.063	0.248$\pm$0.080	0.823$\pm$0.350
Qwen-4B	COT	0.226$\pm$0.056	0.243$\pm$0.064	0.245$\pm$0.076	0.859$\pm$0.328
Qwen-CSP	COT	0.258$\pm$0.083	0.294$\pm$0.096	0.270$\pm$0.082	0.158$\pm$0.324
Improvement	COT	4.45%	19.5%	-3.3%	–

Table 3. Table 3: Ablation results on the Cataract Surgery Benchmark (BLEU, ROUGE, BERTScore, Keypoint-level Metrics. The term "Base" refers to the base model Qwen3-4B.

Model	Task	BLEU	ROUGE-L	BERTScoreF1	k-F1	k-Prec.	k-Rec.
Base	COT	0.048$\pm$0.016	0.153$\pm$0.016	0.829$\pm$0.013	0.216$\pm$0.030	0.230$\pm$0.034	0.205$\pm$0.031
Base	Response	0.039$\pm$0.012	0.166$\pm$0.023	0.836$\pm$0.013	0.222$\pm$0.039	0.179$\pm$0.030	0.302$\pm$0.064
Base+B+C	COT	0.103$\pm$0.025	0.193$\pm$0.022	0.845$\pm$0.011	0.289$\pm$0.036	0.297$\pm$0.042	0.283$\pm$0.037
Base+B+C	Response	0.102$\pm$0.030	0.231$\pm$0.028	0.855$\pm$0.013	0.350$\pm$0.055	0.313$\pm$0.061	0.404$\pm$0.053
Base+A+C	COT	0.056$\pm$0.016	0.226$\pm$0.025	0.861$\pm$0.011	0.236$\pm$0.035	0.373$\pm$0.057	0.173$\pm$0.028
Base+A+C	Response	0.169$\pm$0.040	0.335$\pm$0.044	0.894$\pm$0.013	0.375$\pm$0.056	0.393$\pm$0.066	0.362$\pm$0.050
Base+A+B+C	COT	0.131$\pm$0.029	0.261$\pm$0.034	0.886$\pm$0.015	0.292$\pm$0.043	0.317$\pm$0.046	0.275$\pm$0.045
Base+A+B+C	Response	0.170$\pm$0.041	0.330$\pm$0.044	0.896$\pm$0.013	0.367$\pm$0.054	0.377$\pm$0.062	0.361$\pm$0.051

Table 4. Table 4: Ablation results on the Cataract Surgery Benchmark (Entity-level and NLI Consistency)

Model	Task	Entity F1	Entity Prec.	Entity Rec.	NLI Cons.
Base	COT	0.226$\pm$0.056	0.243$\pm$0.064	0.245$\pm$0.076	0.859$\pm$0.328
Base+B+C	COT	0.293$\pm$0.079	0.299$\pm$0.085	0.324$\pm$0.088	0.829$\pm$0.365
Base+A+C	COT	0.279$\pm$0.057	0.396$\pm$0.097	0.232$\pm$0.060	0.976$\pm$0.141
Base+A+B+C	COT	0.258$\pm$0.083	0.294$\pm$0.096	0.270$\pm$0.082	0.158$\pm$0.324

Full text

1 Introduction

Cataract is the leading cause of blindness globally, and with the accelerating aging of the population, the demand for cataract surgery continues to rise sharply (Davis, 2016). Modern cataract surgery critically depends on meticulous preoperative planning, which requires ophthalmologists to integrate heterogeneous examination reports, including optical coherence tomography (OCT), Pentacam imaging, fundus examinations, and other medical reports, into a unified clinical assessment. This process culminates in high-stakes decisions, such as IOL selection and surgical risk assessment, and imposes considerable cognitive and time burdens on clinicians.

Recent advancements in artificial intelligence have shown promising results in ophthalmic diagnosis, particularly in tasks such as disease detection, severity grading, and image interpretation (Lu et al., 2018; Tong et al., 2020). However, these data-driven AI models require large amounts of labeled data, which is costly. Trained models may also fail to generalize across data acquired from different imaging devices. LLMs, as a subset of generative AI, have demonstrated significant potential in image interpretation, reasoning, and clinical decision-making (Panagoulias et al., 2024; Perera Molligoda Arachchige, 2023). While models such as ChatDoctor (Li et al., 2023a), PMC-LLaMA (Wu et al., 2024), and Med-PaLM (Tu et al., 2024) have demonstrated capabilities in medical reasoning and multi-turn dialogue generation, most remain limited to general-purpose QA or static imaging tasks.

Effectively applying LLMs to ophthalmic surgery demands the ability to integrate highly structured and visual data, adapt to ophthalmology-specific workflows, and generate interpretable, clinically grounded recommendations. Developing models tailored for cataract preoperative planning introduces domain-specific challenges that need to be systematically addressed. First, ophthalmic examination reports integrate multimodal data that demand domain-specific expertise for accurate interpretation. General-purpose LLMs often lack the specialized knowledge required to extract clinically meaningful insights from heterogeneous inputs. Second, large-scale datasets for surgical decision-making in cataract care are still limited. This gap is critical due to the need for transparent, explainable AI to assist in preoperative planning, where nuanced clinical reasoning and risk stratification are crucial. In addition, existing models are rarely evaluated in settings that reflect the actual clinical decision-making process, leading to a disconnect between technical performance and practical utility.

In this paper, we introduce a comprehensive and systematic framework for AI-driven cataract surgical planning that delivers three core innovations: a knowledge-driven multi-agent reasoning architecture, the first large-scale structured benchmark for ophthalmic surgical reasoning, and a domain-specialized large language model tailored for clinical decision support. Our approach not only bridges the gap between general-purpose LLMs and real-world clinical workflows but also establishes a reproducible pipeline for transparent, structured, and high-fidelity surgical recommendations. The main contributions of this work are as follows:

• •

Knowledge-driven multi-agent reasoning architecture. We propose a novel MAS that, for the first time, decomposes ophthalmic surgical planning into collaborative specialist agents, each simulating the diagnostic workflow of an expert ophthalmologist. This modular, prompt-driven framework enables interpretable, clinically aligned extraction and consolidation of heterogeneous examination data, facilitating structured and traceable reasoning processes that go beyond prior single-agent or black-box LLM approaches.

• •

CataractSurg-80K: the first benchmark for structured ophthalmic surgical reasoning. We construct CataractSurg-80K, the first large-scale, expert-annotated dataset specifically designed for benchmarking structured clinical reasoning in cataract surgery. Each sample systematically models the full decision-making process—consisting of diagnostic-style questions, multi-step chain-of-thought (CoT) reasoning, and structured IOL recommendations—thereby supporting fine-grained supervision and evaluation for both reasoning quality and clinical decision support.

• •

Qwen-CSP: a domain-specialized LLM for cataract surgical planning. We introduce Qwen-CSP, a large language model optimized for ophthalmic surgical reasoning via a multi-stage fine-tuning paradigm that successively injects general, medical, and domain-specific knowledge. Qwen-CSP achieves state-of-the-art performance across information extraction, reasoning, and recommendation tasks on CataractSurg-80K, substantially outperforming general-purpose LLMs. Our open-source training pipeline provides a reproducible and extensible foundation for future medical AI research.

2 Related Work

2.1 Large Language Models in Reasoning

LLMs have rapidly progressed in few-shot learning, dialogue generation, and reasoning. Models such as GPT-3.5/4 (Floridi and Chiriatti, 2020; OpenAI Achiam et al., 2023), PaLM (Chowdhery et al., 2023), and LLaMA (Touvron et al., 2023) demonstrate strong CoT capabilities (Wei et al., 2022; Press et al., 2022), laying the foundation for domain-specific adaptations. Biomedical variants like BioGPT (Luo et al., 2022) and Med-PaLM (Tu et al., 2024) extend these models to extract and synthesize clinical knowledge. However, general-purpose LLMs still struggle with tasks requiring structured reasoning over multimodal inputs—such as those seen in surgical planning. To address this, methods like ReAct (Yao et al., 2023) and AgentBench  (Liu et al., 2023) simulate specialist workflows via agent-based architectures, though limitations remain in domain-specific multimodal reasoning and structured decision outputs in clinical contexts.

2.2 Large Language Models in Medicine

Medical LLMs such as ChatDoctor (Li et al., 2023a), Med-Alpaca (Han et al., 2023), and PMC-LLaMA (Wu et al., 2024) have shown strong performance in biomedical question answering and dialogue generation through instruction-tuning. However, these models cannot generally integrate structured EHR data and diagnostic images, essential for clinical decision-making. Recent multimodal models like Flamingo (Alayrac et al., 2022), CLIP (Radford et al., 2021), and LLaVA-Med (Li et al., 2023b) have extended LLMs to handle medical visual inputs, with promising results in radiology and pathology. Yet, surveys (Lin et al., 2023; Joshi et al., 2023; Ben Abacha et al., 2019) note that current systems still struggle with domain-specific reasoning, structured output generation, and specialty areas like ophthalmology that demand precise clinical logic.

2.3 Benchmarks for Evaluating Large Language Models in Medicine

Recent efforts in benchmarking medical LLMs and vision-language systems have laid important groundwork for evaluating clinical AI systems. Current benchmarks for medical LLMs, such as MedQA (Jin et al., 2020), CMB (Wang et al., 2023), and CliMedBench (Ouyang et al., 2024), primarily evaluate factual recall using multiple-choice QA formats. Medical Visual Question Answering (VQA) iassess the ability to interpret medical images by responding to natural language queries about X-rays, MRIs, and CT scans. (Zhang et al., 2023; Lau et al., 2018; Ben Abacha et al., 2019) . These approaches have advanced the understanding of single-modality reasoning and language-image alignment. However, current datasets often focus on describing text and image content rather than generating structured outputs needed for clinical decision support, such as treatment recommendations or risk stratification. These limitations call for the development of more comprehensive benchmarks capable of evaluating collaborative, decision-oriented capabilities in medical AI systems.

3 Methodology

3.1 Knowledge-Driven Multi-Agent System for Heterogeneous Ophthalmic Reports Interpretation

To address the limitations of conventional LLMs in clinical data extraction, such as hallucination, lack of transparency, and poor adaptation to heterogeneous medical inputs, we propose a knowledge-driven Multi-agents system (MAS). In our paradigm, each agent is meticulously designed to emulate a specific ophthalmic subspecialty’s diagnostic reasoning and data extraction workflow (e.g., OCT, corneal topography, fundus imaging, or master biometric summaries). Unlike generic LLM pipelines, which rely on monolithic, end-to-end extraction and are prone to over-interpretation or privacy risks, our MAS leverages hierarchical, domain-specialized prompting rooted in clinical best practices. Each agent operates with tailored, expert-crafted instructions to extract only those parameters that are clinically verifiable and relevant, strictly avoiding the generation of unsupported or fabricated content.

Each specialist agent executes an independent, modality-aware extraction protocol.The OCT agent uses the same retinal pathology criteria as real ophthalmologists, identifying only observable abnormalities while avoiding any speculative diagnoses. The master reporting agent reconstructs structured biometric data using explicit privacy controls. All agents are constrained by a shared output schema, ensuring that their outputs are both interoperable and traceable, and that each extracted fact is directly associated with its source and the responsible agent. This design not only ensures privacy protection and authenticity, but also enables auditability and clinical trust by providing a transparent chain of reasoning from raw input to structured summary.

Raw ophthalmic reports from the hospital resource management system are first routed to specialized agents, each responsible for interpreting a specific report type using domain-informed criteria. All agents conform to a unified output schema, enabling seamless aggregation and synthesis of their findings into comprehensive patient descriptions. An overview of the MAS architecture and the agent coordination flow is illustrated in Figure 2.

3.2 Reasoning-enhanced Dataset Construction

Figure 3 illustrates the construction of the CataractSurg-80K dataset, which bridges raw patient records and actionable clinical recommendations through a multi-stage reasoning framework. Starting from structured patient descriptions, the process unfolds through successive layers of clinical questioning, expert reasoning, and decision synthesis. Each component is tightly coupled with real clinical workflows.

Question Generation. To simulate the initial step of clinical decision-making and provide structured input for downstream reasoning and response generation, this stage transforms multimodal patient summaries into concise diagnostic-style questions. These questions are designed to highlight clinically relevant findings, such as axial length, lens opacity grade, corneal astigmatism, and biometric anomalies, without introducing premature diagnosis or treatment bias. The transformation process is guided by prompts co-designed with ophthalmologists, ensuring alignment with real-world inquiry patterns used in clinical consultations. As a result, the generated questions serve as an interpretable and domain-relevant anchor for both reasoning training and surgical plan generation.

Chain-of-Thought Annotation. To enhance the model’s interpretability and emulate expert reasoning pathways, this stage introduces structured CoT annotations for each diagnostic question. Each annotation is generated through an eight-step reasoning framework that covers biometric parameter analysis, corneal topography interpretation, identification of abnormal findings, IOL selection logic, surgical risk assessment, and patient-specific adaptations. The design of this framework is informed by ophthalmologists’ clinical workflows and emphasizes the traceable derivation of each decision from observed data. By aligning annotations with ophthalmologists’ decision-making logic, the resulting <Question, Complex_CoT> pairs serve as an explicit supervisory signal for training interpretable reasoning processes in surgical planning.

Medical Recommendation Synthesis. To produce actionable, patient-communicable medical recommendations grounded in clinical reasoning, this stage generates structured responses that synthesize the diagnostic question, reasoning path, and physician-aligned recommendation. Each recommendation includes a summary of key findings, personalized IOL selection with type and power, surgical technique suggestions, risk alerts, and postoperative management plans. The generation of recommendation is guided by prompts that require the output to align with actual physician decisions while remaining understandable to patients. These <Question, Response> pairs support training on full-scenario recommendation generation, ensuring the model can deliver coherent and clinically appropriate plans in real-world deployment contexts.

The overall construction process of CataractSurg-80K explicitly separates reasoning supervision and medical recommendation generation. By combining <Question, Complex_CoT> and <Question, Response> pairs in a structured manner, our dataset supports fine-grained control over model interpretability and recommendation quality.

3.3 Multi-Stage Domain-Aware Fine-Tuning for Ophthalmic Surgical Reasoning

We applied the general language model Qwen3-4B to cataract surgery planning in three steps. First, we fine-tuned it on a cataract-related knowledge dataset to enable it to better understand ophthalmic terms and concepts. Then, we trained it on <Question, Complex_CoT> pairs from CataractSurg-80K to help the model learn how ophthalmologists reason through real clinical cases. Finally, we further fine-tuned the model on <Question, Response> pairs to enable it to provide practical surgical recommendations based on real data. This step-by-step process enables the model to accumulate medical knowledge, reasoning ability, and the ability to make clinical recommendations.

4 Experiments

4.1 Benchmark Dataset Description

CataractSurg-80K is a large-scale dataset curated from ophthalmology electronic medical records at Peking University Third Hospital. For our evaluation, we chose a benchmark subset of 777 cases instead of using the entire dataset. This subset includes all common types of cataracts, along with a small number of normal (non-cataract) cases to serve as negative controls. All data were de-identified and received ethical approval.

Figure 4 summarizes the main features of the benchmark dataset. The age distribution indicates that the patients seeking medical treatment are primarily elderly, with an average age of 69.4 years. This aligns with the typical cataract population, which also has a high percentage of women. Most cases in the dataset are classified as abnormal, while a small number are marked as normal or unspecified to simulate the reality of missing data in clinical settings.

We also analyzed the availability of six key biometric parameters: AL, ACD, K1, K2, LT, and CCT. Most patients had complete measurement data, while some records were missing specific values due to incomplete documentation or inaccurate data extraction, and certain cases lacked essential information. The majority of patients underwent phacoemulsification, while a small number opted for capsular tension ring (CTR) implantation. Finally, we provided text length statistics for three categories in the dataset: question, chain of thought, and response, to offer a clearer understanding of the complexity and depth of annotation within the dataset.

4.2 Benchmark Experiment

We conducted quantitative evaluations on general LLMs and customized fine-tuned models for this work on two core tasks: chain of thought generation (Question -> CoT) and final surgical advice (Question -> Response). The evaluation indicators cover text generation quality (BLEU, ROUGE-L, BERTScore), key term extraction (k-F1, k-Precision, k-Recall), entity-level accuracy (Entity F1/Precision/Recall), and chain of reasoning consistency (NLI Consistency), which comprehensively reflect the performance of the model in multiple dimensions such as medical facts, clinical terms, reasoning continuity, and logical consistency.

For text generation quality, we selected three mainstream automatic evaluation indicators: BLEU, ROUGE-L, and BERTScore. BLEU mainly examines the precise phrase matching at the n-gram level, emphasizing the accurate restoration of medical terms and professional expressions; ROUGE-L focuses on the longest common subsequence, which is more suitable for evaluating the overall structure of the reasoning chain, logical integrity and key point coverage; BERTScore is based on deep semantic vectors, which can evaluate the similarity between the generated text and the reference answer in the semantic space, and has stronger adaptability to diversified medical expressions. The experimental results show that after domain fine-tuning, the model significantly outperforms the general large model across all three metrics. The most notable improvement is in BLEU, highlighting that the injection of medical knowledge enhances the accurate restoration of professional terminology. Although the BERTScore baseline is already high, the improvements are modest, indicating that mainstream large models excel in general semantic understanding.

By reviewing Table 1, we find that our fine-tuned model Qwen-CSP outperforms mainstream general models across all three indicators. BLEU is sensitive to exact n-gram matching, which means our Qwen-CSP shows significant improvement when domain knowledge is included, especially in restoring specialized terms. BERTScore, which assesses deep semantic similarity, has a higher tolerance for diverse expressions, but improvements are limited for models already strong in general semantics. In general, our method excels in the restoration of professional terminology and structured medical expressions while maintaining a strong advantage in semantic understanding.

To further evaluate the model’s ability to extract key medical information, including core terms and entities in the medical field, covering surgical methods, anatomical structures, disease names, etc., Scispacy entity recognition and automatic extraction of medical terminology vocabulary were combined. We designed three indicators: k-F1, k-Precision, and k-Recall. The key information k-Precision represents the proportion of correctly restored terms in the generated text, k-Recall measures the coverage of key information in the reference answer, and k-F1 is the harmonic average of the two. Table 2 has shown our Qwen-CSP achieved the best results in all three indicators, especially in k-Recall and k-F1, which showed that domain adaptation not only improved the accuracy of medical entity recognition but also enhanced the coverage of key information, while effectively suppressing the generation of irrelevant or illusory entities.

For the CoT generation task, we further introduce stepwise entity-level evaluation and NLI-based consistency metrics. The entity-level F1, Precision, and Recall are computed by automatically segmenting each reasoning step, extracting medical entities, and comparing them with the reference at each step, thereby quantifying the factual alignment of the reasoning chain. NLI consistency leverages a pre-trained natural language inference model to assess the logical relationship between each pair of consecutive reasoning steps, reporting the proportion of “entailment” as an indicator of overall logical coherence.

The experimental results are shown in Table 2. The fine-tuned model performs outstandingly in the step-by-step medical entity recognition task, with leading F1 value, precision and recall. However, the NLI consistency score dropped from 0.859 to 0.158 after domain fine-tuning, indicating that although the model’s ability to restore medical details has improved, the logical coherence of the reasoning chain has weakened. The reason is that fine-tuning prompts the model to generate richer and more diverse clinical content, which increases the amount of information while also increasing logical fluctuations. To address this problem, in the future, step-by-step NLI discrimination or medical knowledge graph constraints can be introduced in training to take into account both the richness of medical expressions and the consistency of the reasoning chain.

4.3 Ablation study

To rigorously evaluate the effectiveness of the main components of our training pipeline, we performed a series of ablation experiments on the CataractSurg-80K dataset and evaluated on the benchmark. The transition from a base model to a clinical model capability is very complex, as shown in the figure 1. We used a multi-stage training approach: First, we performed supervised fine-tuning (SFT) on a large-scale cataract dataset to enhance the model’s domain knowledge (A). Subsequently, we performed two-stage task-specific training using our carefully selected dataset, first generating a CoT reasoning process (B) and then generating the final surgical recommendations (C). Ablation experiments aim to quantify the respective contributions of domain knowledge adaptation and CoT supervision in this pipeline.

We first ablated the initial supervised fine-tuning on the cataract dataset and trained the model directly on the task dataset without prior domain adaptation. As shown in Table 3 and Table 4, the removal of prior domain adaptation led to a consistent performance drop in all evaluation metrics for CoT and response tasks. In particular, clinical reliability and accurate extraction of key medical points (k-F1, k-Precision, k-Recall) were adversely affected, highlighting the critical role of underlying domain knowledge in enabling the model to reliably capture and reproduce medical concepts and entities.

We secondly removed explicit CoT annotations during training so that the model is only supervised to generate final recommendations without intermediate reasoning steps. As shown in Table 3 and Table 4, the absence of CoT supervision led to a further drop in interpretability and overall performance, as evidenced by the drop in BLEU, ROUGE-L, BERTScore, and entity-level metrics. This highlights the importance of incorporating CoT annotations in training to guide the model toward more explainable and clinically grounded decision-making processes.

Interestingly, the partial ablation model outperformed the full model on some secondary indicators (such as entity-level F1 and NLI Consistency). We believe this is because after domain adaptation and reasoning chain supervision, the model output has become more detailed and diverse, which improves clinical interpretability but may lead to certain variations in entity expression and reasoning steps. The ablation model tends to be more templated and simplifies output, so it performs better in consistency and entity overlap, but the amount of information and reference value in actual clinical decision-making are limited.

4.4 Discussion

The results of this study fully verify the effectiveness of structured multi-agent processes in AI-assisted preoperative decision-making for cataract surgery. By combining multi-agent clinical information understanding, enhanced reasoning supervision, and domain knowledge fine-tuning, our framework significantly outperforms general and existing large medical models in multiple clinical tasks such as information extraction, IOL recommendation, and surgical feasibility assessment, and performs well in interpretability and reasoning transparency. This not only improves the credibility of the model in actual clinical applications but also provides a new paradigm for AI assistance in complex medical decision-making tasks.

It is worth noting that the multi-agent structure and multi-stage training strategy can effectively cope with the complexity of heterogeneous multi-source clinical data and realize the automation of the entire process from original reports to structured reasoning and final recommendations. In the experiment, ablation analysis further confirmed the key role of basic domain knowledge guidance and CoT supervision in improving the clinical performance and reasoning quality of the model. In addition, the support of large-scale, high-quality datasets enables the model to have stronger generalization ability and stability. These achievements provide a solid foundation for the development of intelligent decision-making systems in ophthalmology and the wider medical field.

5 Limitations

We acknowledge that there are still some limitations in the study. First, our architecture is capable of handling multiple reports, but due to funding constraints, this study focuses on four key ophthalmic reports that are the most influential and selected by field experts. This simplification may sometimes miss key information for physician decision-making.

Second, the current model size is relatively small and may be limited in its ability to capture the full complexity of rare or atypical cases. The increase in the number of rare diseases in the dataset and the number of model parameters will address these issues.

The current evaluation lacks validation in clinical practice. To address this, we have developed a user interface for a pilot deployment that applies the model to clinical scenarios, aiming to reduce the costs associated with doctor-patient communication. However, further research involving direct interactions with clinicians and patients is still ongoing.

6 Conclusion

In this study, we proposed a comprehensive AI-driven preoperative decision support framework for cataract surgery. First, we designed a novel multi-agent reasoning process and multi-stage fine-tuning strategy to achieve structured understanding and reasoning of complex clinical information, effectively improving the intelligence level of preoperative decision-making. Second, we built a large-scale, expert-annotated CataractSurg-80K dataset, and finely annotated 777 benchmarks based on real cases, systematically supporting the training and objective evaluation of the model. We also open-sourced a set of multi-stage large model training methods, which achieved an organic connection from step-by-step clinical reasoning to end-to-end surgical planning, and improved the interpretability and practical application value of the model. Our datasets and benchmarks provide a foundation for industry-standard evaluation and model reproducibility research. Our open-source training framework also provides a powerful tool for the community to conduct large-scale clinical reasoning and decision support research.

Appendix A Data

A.1 Open-sourced Multi-source Cataract Medical QA/CoT Datasets

License. All resources are released under the CC BY-NC-SA 4.0 license.

Training. We provide a collection of Chinese and English datasets for supervised fine-tuning and evaluation in the field of cataract clinical reasoning and report understanding.

• •

Base Knowledge

https://huggingface.co/datasets/my2000cup/CataractSurg-80K

source PDF file comes from:

https://apacrs.org/ppp_cataract_EN/files/assets/common/downloads/APACRS-PPP-Cataract%20Surgery..pdf

https://apacrs.org/ppp_cataract_cn/files/assets/common/downloads/APACRS-PPP-Cataract%20Surgery-cn.pdf

• •

CataractSurg-80K:

https://huggingface.co/datasets/my2000cup/CataractSurg-80K

cataract_base_zh, cataract_base_en: Instruction-based QA (Chinese/English)

inference01_zh, inference01_en: Chain-of-Thought reasoning

inference02_zh, inference02_en: Clinical recommendations

All subsets follow the Alpaca SFT format.

Model.

• •

Qwen-CSP:

https://huggingface.co/my2000cup/Qwen-CSP

Fine-tuned Qwen3-4B model on the CataractSurg-80K datasets.

Code.

• •

CataractCareMAS:

https://anonymous.4open.science/r/CataractCareMAS-A363

Data processing, model training, and evaluation code.

A.2 Notes on open source datasets and code

Due to the requirements of patient privacy protection and ethical compliance, we cannot open-source the original patient electronic medical record data, like the original data before preprocessing by the multi-agent system. Therefore, the multi-agent cleaning and structuring process for the original data cannot be directly reproduced by a third party. The actual open source dataset only contains the downstream training data that has been strictly desensitized and structured. This part of the data has removed all personal identity information and obtained hospital ethics approval.

Although the complete pipeline cannot be directly opened to the public, we still open source the code of multi-agent data cleaning and structuring. The code has good versatility and scalability and can be migrated to other medical or multimodal scenarios. Researchers can combine their own legal and compliant data resources and quickly implement data structuring and cleaning processes suitable for their own tasks based on our code framework. For different application scenarios, users need to make appropriate adjustments and optimizations to the role division, rules, and prompt word engineering of multi-agents based on actual needs.

We believe that with the opening of more compliant data, the multi-agent data cleaning framework will provide valuable reference and tool support for medical AI data engineering, structured knowledge extraction and other fields.

Appendix B Prompts

B.1 Prompts for Heterogeneous Ophthalmic Reports Interpretation

B.2 Prompts for Constructing Reasoning-enhanced Datasets

Appendix C Case Study

We chose a representative case in cataract surgery, namely, surgical risk reminders for diabetic patients. In this case, the OCT report detected diabetes, so the relevant reminder appeared in our Question column. We simulated the lack of OCT and observed significant differences in clinical recommendations for patients with diabetic cataracts.

When the OCT report was available, the multi-agent system was able to identify diabetic macular edema, a key complication that directly affects surgical prognosis and postoperative management. In contrast, when the OCT report was missing, the system failed to identify this key risk, resulting in generic recommendations that may overlook important diabetic complications. The result summary is shown in Figure 8

This case highlights the advantage of our multi-agent approach: the more comprehensive the information obtained and processed, the more reliable the clinical reasoning. Integrating a full set of reports enables the system to provide clinicians with more accurate, personalized, and safer decision support.

We also selected an elderly female patient with multiple complications as a complex case for testing. The patient also had risk factors such as abnormal corneal morphology, significant astigmatism, dry eyes, and glaucoma, which placed high demands on the preoperative evaluation and surgical selection of cataract surgery. The experimental results showed that the multi-agent system can effectively integrate multi-source examination data, accurately identify the main risk points of patients, and give surgical recommendations that are highly consistent with the opinions of real clinical experts. This result further verifies that our method has good reliability and clinical practical value in actual clinical scenarios with complex and multi-factor interventions. Result summary shown in Figure 9.

Appendix D Further Discussion

D.1 Discussion of social impact, ethics, and limitations

Our system is expected to improve the accessibility and fairness of healthcare by empowering clinicians in resource-poor areas. Multi-agent systems can improve doctors’ efficiency in diagnosing common diseases, especially in primary care institutions, and help alleviate the problem of uneven distribution of high-quality medical resources. Most diseases encountered in clinical scenarios are routine cases, so this has practical significance for improving the overall level of medical care. However, in the face of complex or rare diseases, artificial intelligence systems still have a long way to go and need to continue to accumulate data and optimize reasoning capabilities.

From the perspective of multimodal and multi-source data processing, multi-agent systems have stronger professionalism and scalability than single, general large models. This is also the original intention of our development and introduction of multi-agent collaboration to process different types of medical reports. By focusing on different subsystems on different medical subtasks, the processing efficiency and professional reliability of the output can be effectively improved.

In terms of ethics and privacy, the bottleneck of the promotion of medical large models is often not the technology itself, but the high attention to data privacy security and ethical compliance. Multi-agent systems have the natural advantages of data classification and decoupling, which help to better achieve de-identification and privacy protection during data flow and processing. This study strictly desensitized and standardized the original data under the hospital’s ethical approval and compliance license, providing a model for the safe application of subsequent artificial intelligence systems.

It is worth emphasizing that our tool only serves as a clinical decision support system to assist doctors in diagnosis and treatment judgment, rather than replacing doctors for autonomous diagnosis or treatment. We recommend that manual review and final review should always be maintained during use to prevent new risks caused by technology abuse or over-reliance on automated suggestions. At the same time, it is recommended that the mechanism construction in ethical governance, data security, and model interpretability should be continuously strengthened in the subsequent actual application and promotion process.

D.2 Preparation and impact discussion of basic knowledge SFT dataset

Preparation and impact discussion of basic knowledge SFT dataset. In the first stage of training our large model, the preparation of the basic knowledge dataset also adopted a multi-agent system, but its structure was much simpler than that of the cataract surgery multi-agent system, focusing mainly on the automatic organization of medical knowledge and question-answer generation. The specific process is as follows:

First, the data comes from the internationally recognized authoritative guide-APACRS "Patient Guide for Cataract Surgery" (Chinese and English versions, link in the appendix). We removed the references and content before the table of contents in the original file, and only retained the pure text information. For the convenience of subsequent processing, the text content is divided into several blocks of about 5,000 words each.

For each small block, GPT4o-mini is used as the core of the agent, and professional questions related to the text content are automatically generated through the prompt word engineering, and further answers based on the text block are generated. In order to improve the accuracy and authority of the answers, the system also calls the ReAct Agent to automatically retrieve relevant Wikipedia content to supplement and unify the answers. Through the above process, a high-quality SFT paired dataset was finally formed.

Unlike actual clinical data, the text content of the basic knowledge dataset is more standardized and consistent, and the question-answer pairs can cover the mainstream preoperative, intraoperative, and postoperative knowledge points of cataract surgery. This injects a solid medical foundation and professional expression ability into the model. The introduction of the multi-agent process ensures the relevance, consistency, and traceability of the generated questions and answers, and effectively avoids subjective bias or omissions in the manual sorting process.

The experimental results show that with the SFT pre-training of this part of basic knowledge, the model shows stronger medical concept understanding and medical terminology restoration capabilities in subsequent chain reasoning (CoT) and clinical recommendation tasks, and improves the overall generation quality indicators (such as BLEU, BERTScore, etc.). This shows that in the implementation of the large model medical field, the systematic and standardized injection of basic knowledge is a key link to improve the performance of downstream tasks.