Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis

TL;DR

This paper evaluates the performance of large vision language models and segmentation models on microscopy images, revealing their strengths and limitations compared to domain experts in tasks like classification, segmentation, counting, and VQA.

Contribution

It is the first comprehensive assessment of VLMs and SAM on microscopy images, highlighting their capabilities and current limitations in scientific image analysis.

Findings

ChatGPT and Gemini understand microscopy visual features well.

SAM effectively isolates artifacts but struggles with complex image impurities.

Models are less accurate than domain experts due to image impurities.

Abstract

Vision language models (VLMs) have recently emerged and gained the spotlight for their ability to comprehend the dual modality of image and textual data. VLMs such as LLaVA, ChatGPT-4, and Gemini have recently shown impressive performance on tasks such as natural image captioning, visual question answering (VQA), and spatial reasoning. Additionally, a universal segmentation model by Meta AI, Segment Anything Model (SAM) shows unprecedented performance at isolating objects from unforeseen images. Since medical experts, biologists, and materials scientists routinely examine microscopy or medical images in conjunction with textual information in the form of captions, literature, or reports, and draw conclusions of great importance and merit, it is indubitably essential to test the performance of VLMs and foundation models such as SAM, on these images. In this study, we charge ChatGPT, LLaVA, Gemini, and SAM with classification, segmentation, counting, and VQA tasks on a variety of microscopy images. We observe that ChatGPT and Gemini are impressively able to comprehend the visual features in microscopy images, while SAM is quite capable at isolating artefacts in a general sense. However, the performance is not close to that of a domain expert - the models are readily encumbered by the introduction of impurities, defects, artefact overlaps and diversity present in the images.