Grounding Multimodal Large Language Models with Quantitative Skin Attributes: A Retrieval Study

TL;DR

This study investigates enhancing the interpretability of multimodal large language models in skin disease diagnosis by grounding their embeddings in quantitative skin attributes, demonstrated through a retrieval case study.

Contribution

It introduces a method to ground MLLM embeddings in quantitative skin attributes, improving interpretability in skin lesion diagnosis tasks.

Findings

MLLM embeddings can be grounded in quantitative skin attributes.

Fine-tuning enables attribute prediction from images.

Grounded embeddings improve interpretability in retrieval tasks.

Abstract

Artificial Intelligence models have demonstrated significant success in diagnosing skin diseases, including cancer, showing the potential to assist clinicians in their analysis. However, the interpretability of model predictions must be significantly improved before they can be used in practice. To this end, we explore the combination of two promising approaches: Multimodal Large Language Models (MLLMs) and quantitative attribute usage. MLLMs offer a potential avenue for increased interpretability, providing reasoning for diagnosis in natural language through an interactive format. Separately, a number of quantitative attributes that are related to lesion appearance (e.g., lesion area) have recently been found predictive of malignancy with high accuracy. Predictions grounded as a function of such concepts have the potential for improved interpretability. We provide evidence that MLLM embedding spaces can be grounded in such attributes, through fine-tuning to predict their values from images. Concretely, we evaluate this grounding in the embedding space through an attribute-specific content-based image retrieval case study using the SLICE-3D dataset.

Tables

Table 1. Table 1: Subset of quantitative attributes from the SLICE-3D dataset  Kurtansky et al. ( 2024 ) that are considered in this study. These attributes were selected following consultation with medical domain experts and a recent study detailing the efficacy of various attributes for diagnostic prediction  Marchetti et al. ( 2023 ) .

Attribute	Definition
areaMM2	Lesion area.
minorAxisMM	Smallest lesion diameter.
norm_color	Color variation.
radial_color_std_max	Color asymmetry within lesion.
deltaB	Average B (LAB) contrast (inside vs outside lesion).
deltaL	Average L (LAB) contrast (inside vs outside lesion).
deltaLB	L (LAB) contrast between lesion and immediate surrounding skin.
stdLExt	Standard deviation of L (LAB) outside lesion.
clin_size_long_diam_mm	Maximum diameter of lesion.
perimeterMM	Perimeter of lesion.
norm_border	Lesion border irregularity.
area_perim_ratio	Lesion border jaggedness; ratio between lesion perimeter and area.
A	A (LAB) inside lesion.
Aext	A (LAB) outside lesion.
B	B (LAB) inside lesion.
Bext	B (LAB) outside lesion.

Equations

$$D_{tr}=\{(I,Q(a_w^I),y_{a_w^I}(I):I\in {\mathcal{I}}_{tr},w\in \{1,\dots ,W\}\}.$$

$$h^{(im)}(I)=\frac{1}{N_I}\sum _{i=1}^{N_I}{f}_i(g(I))\in {\mathbb{R}}^d,$$

$$h^{(im,a)}(I)=f_{-1}([g(I),\psi (Q(a)])\in {\mathbb{R}}^d,$$

Full text

*$$*footnotetext: Correspondence to: torop.m@northeastern.edu

1 Introduction

Skin diseases and, in particular, cancer, significantly affect millions of lives annually, serving as one of the leading causes of doctor visits. Access to dermatology care is limited, especially for underserved populations, due to factors like socioeconomic status and provider distribution, leading to poor outcomes associated with late-stage skin cancer diagnoses Glazer and Rigel (2017); Vaidya et al. (2018). The shortage of dermatologists, projected increase in visits, and existing barriers to care constitute a significant public health concern. Recent advances in Artificial Intelligence (AI), particularly in deep learning, show promise for addressing this problem. AI models have demonstrated remarkable performance, often achieving diagnostic accuracy for skin cancers, such as melanoma, comparable to that of experienced dermatologists Krakowski et al. (2024). Such advances have the potential to enhance diagnostic workflows, improve early detection rates, and democratize access to specialist-level assessment.

However, the adoption of these tools into routine clinical practice is severely limited by a fundamental challenge: their opacity. The "black box" nature of AI models, where the reasoning process is obscured from human users, creates a significant barrier to clinical trust and adoption. For clinicians to utilize AI recommendations, especially in high-stakes fields like oncology, it is essential that they understand the rationale behind their decisions. Clinicians adhere to an evidence-based reasoning model, where each diagnosis must be supported by identifiable evidence. The inability of AI systems to provide interpretable recommendations, despite their high aggregate accuracy, contradicts this principle. This lack of transparency also poses a risk to patient safety by concealing a critical vulnerability in deep learning models: their propensity to exploit spurious correlations in the training data, e.g. surgical skin markings, rulers, and hair, with melanoma Combalia et al. (2022).

In this work, we explore the combination of two promising avenues for improving model interpretability: quantitative attribute usage and Multimodal Large Language Models (MLLMs) Li et al. (2023). A recent study has shown that a variety of numerical attributes, such as lesion area and border jaggedness, are highly predictive of malignancy Marchetti et al. (2023). Models grounded in such human understandable attributes have the potential for improved interpretability via usage in downstream tasks. Separately, MLLMs present a potentially effective method for facilitating clinicians’ usage of AI, due to their interactive nature and use of natural language. We take an initial step toward grounding MLLMs in quantitative attributes that can be predicted from images. This involves fine-tuning Qwen 2 VL Wang et al. (2024) to predict these attributes using images from the SLICE-3D Kurtansky et al. (2024) dataset used in the ISIC 2024 challenge. We first illustrate the model’s prediction performance and then demonstrate its application to the downstream task of image retrieval. The resulting image embeddings can effectively retrieve images that are visually similar to the queried ones, and can be additionally specialized to retrieve images that also match the query in specific quantitative attributes. The efficacy of our approach provides evidence that MLLM embedding space can be grounded in these important quantitative attributes.

2 Related Works

Saliency maps (e.g., Grad-CAM Selvaraju et al. (2017)) are among the most widely used techniques for AI explainability in dermatology. These methods generate heatmaps meant to highlight image regions deemed important by the model for its prediction. While intuitively appealing, a growing body of evidence reveals that these maps are often unreliable and frequently fail to align with true, diagnostically relevant features. They can be influenced by confounding artifacts and do not provide a "faithful" explanation of the model’s internal logic in a language that is meaningful to a clinician Zhang et al. (2023). Explanations that are accessible to humans should preferably be semantic in nature, clearly expressing the presence, absence, or value of specific, diagnostically linked concepts.

While such semantic features are not represented in pixel-level heatmaps, they can be found in natural language descriptions of images. A variety of works make use of Vision Language Models (VLMs) Radford et al. (2021); Li et al. (2023) to capture these features. By training on extensive datasets that include paired images and textual descriptions, such as dermatological images alongside their corresponding clinical reports or textbook annotations, these models capture the relationship between visual morphology and the clinical terminology used for description. Approaches such as MONET Kim et al. (2024), DermLIP Yan et al. (2025a) and PanDerm Yan et al. (2025b) tailor CLIP-like VLMs Radford et al. (2021) –which jointly align the embedding spaces of text and image encoders– on such data. Among many uses, the resulting embeddings can be utilized to predict the presence or absence of semantic concepts in a zero-shot fashion. A related strain of work involves training MLLMs Li et al. (2023) –generative models with a vision encoder and text decoder– on similar data (e.g., visual question answering derived variants) Zeng et al. (2025); Codella et al. (2024); Zhou et al. (2024). MLLMs have the potential to further improve clinicians’ usage of AI as they answer questions in natural language and may be amenable to multi-turn interactions.

A recent study Marchetti et al. (2023) reveals that a number of quantitative (i.e., numerical) characteristics of lesions, such as size, border irregularity, and the contrast between the lesion and surrounding skin, serve as effective predictors of malignancy. However, the extent to which current MLLMs may be grounded in such attributes is not immediately obvious. It has been found that, despite their broad capabilities, many state-of-the-art MLLMs can struggle with "number sense", i.e., dealing with concepts such as angle, length, area, and volume in images Weng et al. (2025).

Complementary to the works above, we provide evidence –in the form of an image-to-image retrieval study– that MLLM embedding space can indeed be grounded in many of these quantitative attributes. Further, our approach facilitates the retrieval of images that are not only visually similar to a query image, but also in a quantitative attribute of interest. This is done through use of the decoder to merge image and text embeddings (i.e., composed search Vo et al. (2019)), standing in contrast to retrieval in CLIP-like VLMs Kim et al. (2024); Yan et al. (2025a, b), as they contain separate embedding spaces for image and text, and the MLLMs above, which, if used for retrieval, do not tailor embeddings to attributes at inference time  Codella et al. (2024).

3 Methods

In this section, we detail an approach to grounding an MLLM in quantitative attributes of skin images by fine-tuning the model to predict multiple attributes from images. We also showcase the applicability of this grounding to attribute-specific content-based image retrieval, using the learned embedding space to retrieve images that are not only visually similar to a query but also match it in an attribute of interest.

3.1 Preliminary

Notation: Tokens are granular units of language, e.g., "hello" and "1". We denote the set of tokens as $V$, with size $|V|$, and the set of finite-length sequences of tokens as $X$. Examples of $x\in X$ are ("How", "are", "you", "?") and ("1", ".", "2", "3"). We denote the space of images as $\mathcal{I}$. We define an attribute to be a measurable numeric property of an image (e.g., lesion area) and denote the set of attributes under consideration as $A$. We associate each attribute $a\in A$ with a question $Q(a)\in X$ and a function which returns a token sequence corresponding to the value of $a$ for a given image $y_{a}:\mathcal{I}\rightarrow X$. For instance, for the attribute $a=$"area" one may have $Q(\text{area})=$ ("What", "is", "the", "area", "of", "the", "lesion", "in", "mm2", "?"), and given an image $I\in\mathcal{I}$ with an area of $1.23$ mm2 we have $y_{area}(I)=("1",".","2","3")$.

Architecture: MLLMs generally consist of a decoder $F$ and ViT-based Dosovitskiy et al. (2020) vision encoder $g$. An input image $I\in\mathcal{I}$ is transformed into $N_{I}$ patches (with patch size a hyperparameter), which are processed by $g$ into $d$-dimensional embeddings: $g(I)\in\mathbb{R}^{N_{I}\times d}$. Similarly, token sequences are vectorized using a learned function $\psi:V\rightarrow\mathbb{R}^{d}$ which, in an abuse of notation, can be applied to $x=v_{1}\ldots v_{k}$ via $\psi(x)=[\psi(v_{1})^{T},\ldots,\psi(v_{k})^{T}]\in\mathbb{R}^{k\times d}$. Pairing $g(I)$ and $\psi(x)$ together, $F$ outputs next token logit scores: $F([g(I),\psi(x)])\in\mathbb{R}^{|V|}$. The decoder can be written as a prediction head $h:\mathbb{R}^{d}\rightarrow\mathbb{R}^{|V|}$ applied to the final output row of a feature extractor $f:\mathbb{R}^{m\times d}\rightarrow\mathbb{R}^{m\times d}$, where $m$ can be arbitrary (above, $m=N_{I}+k$). The latter consists of a composition of layers, which contain multi-head self-attention and multi-layer perceptron operations. The output rows of $f$ correspond to the extracted features of the tokens, each drawing information only from earlier tokens. The encoder $g$, token vectorization function $\psi$ and decoder $f$ are shown in Figure 1.

3.2 Training on Attributes

We fine-tune the model to predict attribute values from images, in the process learning embedding spaces which reflect these attribute and general lesion appearance. We denote $\mathcal{I}_{tr}\subseteq\mathcal{I}$ as the set of training images. The training set $D_{tr}$ consists of tuples: images paired with attribute specific questions and values. For each $I\in\mathcal{I}_{tr}$, we sample $W\leq|A|$ attributes from $A$ : $a^{I}_{1},\ldots,a^{I}_{W}$ uniformly at random and generate tuples of the form $(I,Q(a^{I}_{w}),y_{a^{I}_{w}}(I))$, $w=1,\ldots,W$. Thus, $D_{tr}$ contains $|D_{tr}|=|\mathcal{I}_{tr}|\times W$ tuples, and may be written as

[TABLE]

We use Low Rank Adaptation (LoRA) Hu et al. (2022) to jointly fine-tune $g$ and $F$, maximizing the log-likelihood of attribute $y_{a_{w}^{I}}(I)$ given image $I$ and question $Q(a)$.

3.3 Quantitative Image Retrieval

We use the embedding space of our fine-tuned MLLM to retrieve images which are similar to a query image of interest, $I_{q}\in\mathcal{I}$. We experiment with two types of embedding spaces, one derived from images alone and one from images paired with question $Q(a)$, given an attribute $a$ of interest. Intuitively, both capture general visual similarity, with the latter additionally capturing information specific to $a$. As in prior work Yu et al. (2025), we use the outputs of $f$ to construct our embeddings.

Image-Only Embeddings: For the image only approach we create a database of embeddings by averaging the outputs of $f$ for each image token. We define the function which sends an image $I$ to such embeddings as

[TABLE]

where $f_{i}(g(I))\in\mathbb{R}^{d}$ represents the $i^{th}$ row of $f(g(I))$. This embedding function is shown in the top-left corner of Figure 1.

We define $B^{(im)}$, an operator which sends collections of images to a matrix of image features (our database). Applying $B^{(im)}$ to any image collection $\mathcal{I}^{\prime}$ yields $B^{(im)}(\mathcal{I}^{\prime})\in\mathbb{R}^{|\mathcal{I}^{\prime}|\times d}$, a matrix with rows of the form: $B^{(im)}_{j}(\mathcal{I}^{\prime})=h^{(im)}(I_{j}),\ j\in\{1,\ldots,|\mathcal{I}^{\prime}|\}$. One may encode the query image $I_{q}$ via $h^{(im)}(I_{q})$ and retrieve the top $k$ most similar images in $\mathcal{I}^{\prime}$ using the cosine similarity between $h^{(im)}(I_{q})$ and the rows of $B^{(im)}(\mathcal{I}^{\prime})$. Such an approach yields images which are broadly visually similar to $I_{q}$ across many attributes. In this work we set $\mathcal{I}^{\prime}=\mathcal{I}_{tr}$.

Image-Text Embeddings: While the approach above yields images which are visually similar to $I_{q}$, it does not consider similarities in specific attribute values $a$. We employ the following embedding function to retrieve images which are similar in both of these respects:

[TABLE]

the final row (token feature) of $f([g(I),\psi(Q(a)])$. This embedding function is shown in the top-right corner of Figure 1. We use the final token following prior work Rimsky et al. (2024), as it draws information from all other tokens, i.e. the entire image and question. As above, we create a queryable database $B^{(im,a)}(\mathcal{I}^{\prime})\in\mathbb{R}^{|\mathcal{I}^{\prime}|\times d}$ with rows $B^{(im,a)}_{j}(\mathcal{I}^{\prime})=h^{(im,a)}(I_{j}),\ j\in\{1,\ldots,|\mathcal{I}^{\prime}|\}$, setting $\mathcal{I}^{\prime}=\mathcal{I}_{tr}$.

Hierarchical Retrieval: We experiment with a hierarchical search procedure to overcome the space limitations of the image-text approach, which requires maintaining $|A|$ databases. Here, we make use of both the image-only and image-text embeddings. First, we retrieve $b$ images from pre-computed database $B^{(im)}(\mathcal{I}_{tr})$: $\ \mathcal{I}^{(temp)}\in\mathbb{R}^{b\times d}$, where $k<<b<<|\mathcal{I}_{tr}|$. This narrows down the search space to $b$ images which may be broadly similar to $I_{q}$ in each of the attributes. Next, we create a temporary database for only these images on the fly: $B^{(im,a)}(\mathcal{I}^{(temp)})$. Finally, we query the top $k$ images from this database. Large $b$ values require more forward passes but allow searching more images, representing a tradeoff.

4 Experiments

4.1 Setup

We use the SLICE-3D dataset Kurtansky et al. (2024), which consists of $401,059$ total body photography images of cropped lesions, for this case-study. SLICE-3D is a natural choice for exploring quantitative attribute grounding, as each image contains a number of metadata attributes. In this work, we consider $16$ quantitative attributes, such as lesion area, border jaggedness and luminance (as seen in Table 1). We divide the images of SLICE-3D into $\mathcal{I}_{tr}$ and $\mathcal{I}_{te}$, stratifying by patients to make sure there is no overlap. The sets $\mathcal{I}_{tr}$ and $\mathcal{I}_{te}$, respectively contain $|\mathcal{I}_{tr}|=282,564$ and $|\mathcal{I}_{te}|=118,495$ images, corresponding to $802$ and $240$ patients. We sample $W=5$ attributes for each image in $\mathcal{I}_{tr}$, yielding $D_{tr}$ with $|D_{tr}|=1,412,820=282,564\times 5$ tuples.

We use the Qwen 2 VL 7B Wang et al. (2024) MLLM which consists of a 7.6B parameter decoder (Qwen 2 Yang et al. (2024)) and a 365M parameter image encoder Dosovitskiy et al. (2020). We train for $1$ epoch using $4$ NVIDIA A6000 GPUs, a LoRA rank of $8$, a batch size of $16$ ($4$ per machine), and a cosine learning rate schedule (starting at $1e$-$4$).

4.2 Results

Quantitative Prediction: We asses the extent to which the model has learned to predict the attributes. For each $I\in\mathcal{I}_{te}$, we predict all $16$ attributes and compare to the ground truth. Figure 2 shows the corresponding $R^{2}$ values, which range between $0.71$ and $0.96$. These results indicate that the model has effectively learned to predict the attributes from images.

Qualitative Retrieval: We qualitatively compare the top-$5$ images retrieved by image-only and image-text search (using $Q(area)$) in Figure 3. All images retrieved are visually similar to the corresponding queries. While images from both methods tend to have similar areas to the queries, there are some notable exceptions for the image-only method: e.g., the area of the first query is significantly lower than that of the fourth retrieved image.

Quantitative Retrieval: Figure 4 provides a quantitative measure of retrieval quality. Each subplot corresponds with one of the $16$ attributes; each boxplot with a retrieval approach. For each method we retrieve the top-$5$ images from $\mathcal{I}_{tr}$ for each $I_{q}\in\mathcal{I}_{te}$. For each $I_{q}$, we compute the squared attribute difference between it and (1) the retrieved images, and (2) all $I\in\mathcal{I}_{tr}$. We record the percentile rank of each retrieved image’s attribute difference within the full set of differences. Boxplots show the distribution of these percentile ranks, with lower values indicating that retrieved images are highly similar to their query in the attribute. The first three boxplots in a subplot use our fine-tuned Qwen model with (a) image $+$ text (b) hierarchical (using $200$ initial images) and (c) image only embeddings for retrieval. The fourth boxplot corresponds to (d) image only embeddings on a non fine-tuned Qwen model and the final plot to (e) embeddings from the image encoder in MONET Kim et al. (2024).

The order of method efficacy is consistent across attributes. The vanilla Qwen model retrieves images with very different attribute values from the query whereas MONET –despite also not being trained on these attributes– performs better, as it was trained on general skin data. The image-only embeddings from our fine-tuned Qwen model are significantly more effective then the prior approaches, as they are used in our fine-tuning procedure to predict each attribute. The image-text embeddings are most effective in retrieving images with similar attributes. This is intuitive, as this embedding space is tailored to the specific attribute at retrieval time, as opposed to the image-only approach in which embeddings are tailored to all attributes at train time. Last, the hierarchical modification to the text-image approach does not lower the quality of the results, despite only using $200$ images for image-text search.

5 Conclusion

The initial results in this work suggest that MLLMs can be effectively grounded in quantitative attributes. Our model achieves high $R^{2}$ scores when predicting the 16 attributes for which it was fine-tuned. Additionally, we show that embeddings derived from our model’s internal representations can be used for retrieval of dermatology images. While the average output of the image token representations in the decoder can be effectively used to retrieve images which are generally visually similar to a query image, the representation of the final question token can be used to tailor the retrieval to also match the query in a specific attribute of interest.

The efficacy of our attribute-specific image-text retrieval provides evidence for the feasibility of multi-attribute retrieval, in which images match the query in multiple attributes simultaneously. Such an approach may be implemented as a straightforward extension of this work, by taking final token embeddings from image-text pairs in which the text includes questions about multiple attributes. Future work stands to incorporate both this multi-attribute retrieval and conversational training data, with the aim of creating an interpretable and interactive system that supports clinicians.