LAMMI-Pathology: A Tool-Centric Bottom-Up LVLM-Agent Framework for Molecularly Informed Medical Intelligence in Pathology

TL;DR

LAMMI-Pathology is a scalable, tool-centric LVLM-agent framework designed for molecularly informed pathology diagnosis, leveraging hierarchical planning and trajectory-aware fine-tuning to improve reasoning and tool utilization.

Contribution

The paper introduces a novel bottom-up, tool-centric agent architecture with a trajectory construction mechanism and fine-tuning strategy for pathology image analysis.

Findings

Enhanced reasoning robustness in pathology tasks.

Effective hierarchical coordination of domain-specific tools.

Improved inference accuracy through trajectory-aware training.

Abstract

The emergence of tool-calling-based agent systems introduces a more evidence-driven paradigm for pathology image analysis in contrast to the coarse-grained text-image diagnostic approaches. With the recent large-scale experimental adoption of spatial transcriptomics technologies, molecularly validated pathological diagnosis is becoming increasingly open and accessible. In this work, we propose LAMMI-Pathology (LVLM-Agent System for Molecularly Informed Medical Intelligence in Pathology), a scalable agent framework for domain-specific agent tool-calling. LAMMI-Pathology adopts a tool-centric, bottom-up architecture in which customized domain-adaptive tools serve as the foundation. These tools are clustered by domain style to form component agents, which are then coordinated through a top-level planner hierarchically, avoiding excessively long context lengths that could induce task drift. Based on that, we introduce a novel trajectory construction mechanism based on Atomic Execution Nodes (AENs), which serve as reliable and composable units for building semi-simulated reasoning trajectories that capture credible agent-tool interactions. Building on this foundation, we develop a trajectory-aware fine-tuning strategy that aligns the planner's decision-making process with these multi-step reasoning trajectories, thereby enhancing inference robustness in pathology understanding and its adaptive use of the customized toolset.