# Visual recognition limitations in multimodal large language models: A comparative analysis of histological image interpretation

## TL;DR

This study compares how well AI systems interpret microscopic tissue images and finds they perform significantly worse than with text-based questions, highlighting the need for better visual processing in medical AI.

## Contribution

The study establishes technical benchmarks for multimodal LLMs in histological image interpretation and reveals significant cross-modal performance gaps.

## Key findings

- Gemini outperformed other models in histological image interpretation with a mean score of 3.35/4.00.
- AI systems showed a significant performance gap between text-based (90% accuracy) and image-based tasks (51-80% accuracy).
- Epithelial tissue interpretation had the greatest inter-model variation in performance.

## Abstract

Multimodal large language models (LLMs) with image recognition capabilities have emerged as potential tools for medical image analysis, yet their performance in specialized domains like histology remains largely unexplored. The objective of this study was to systematically evaluate the performance of leading multimodal LLMs in histological image interpretation and assess their visual recognition capabilities. Four multimodal LLMs (GPT-4o, Claude Sonnet 4, Gemini 2.5 Flash, and Copilot) were evaluated using 144 histological images representing four tissue types (epithelial, connective, muscle, and nervous) at three magnification levels. Each image was assessed using three standardized questions: tissue identification, morphological features, and functional analysis. Three expert faculty members independently graded responses using a 4-point scale (1 = Poor to 4 = Excellent). Friedman tests, ICC, and post-hoc power analyses were performed with statistical significance set at p < .05. A clear performance hierarchy emerged with Gemini demonstrating superior performance (mean score: 3.35/4.00), significantly outperforming all other models. Copilot and GPT-4o tied for second place (both 2.76/4.00), while Claude showed the lowest performance (2.55/4.00). Performance varied across tissue types, with epithelial tissue showing the greatest inter-model variation. Inter-rater reliability was good across all models (ICC > 0.85), confirming assessment consistency. Post-hoc power analysis validated statistical significance for primary comparisons but indicated insufficient power to distinguish between the three lower-performing models. Current multimodal LLMs exhibit significant limitations in visual recognition relative to text processing performance. The substantial cross-modal performance gaps reveal some constraints in visual processing architectures, though the underlying mechanisms require further investigation. These findings establish technical benchmarks for multimodal LLM development and highlight the need for specialized visual processing innovations in their imaging processes.

In our increasingly digital world, artificial intelligence (AI) systems are being developed to help medical professionals analyze medical images. Our research examines how well the latest AI systems interpret microscopic images of human tissues compared to their ability to answer written questions about the same topic. We tested four leading AI systems by asking them to identify tissues, describe features, and explain functions based on microscopic images. Three expert faculty members evaluated their responses. We discovered that while these AI systems excel at answering text-based medical questions (over 90% accuracy), they struggle significantly with interpreting images (51-80% accuracy). This performance gap varied between different AI systems and tissue types. Our findings reveal that current AI technology demonstrates substantial performance gaps in processing visual medical information despite having the necessary medical knowledge, suggesting opportunities for architectural improvements. This research provides important benchmarks for measuring progress in medical AI development and highlights challenges that must be addressed before these systems can be reliably used in medical education and practice.