# A triple pronged approach for ulcerative colitis severity classification using multimodal, meta, and transformer based learning

## TL;DR

This paper presents a deep learning approach combining multimodal, meta, and transformer-based methods to classify the severity of ulcerative colitis with high accuracy.

## Contribution

A novel triple-pronged deep learning framework for UC severity classification using multimodal inference, few-shot meta-learning, and ViT ensembling.

## Key findings

- A Swin-Base model achieves 90% accuracy in UC severity classification.
- An ensemble of ViT backbones improves performance to 93%.
- Multimodal pre-trained models with ML algorithms reach 83% accuracy.

## Abstract

Ulcerative colitis (UC) is a chronic inflammatory disorder necessitating precise severity stratification to facilitate optimal therapeutic interventions. This study harnesses a triple-pronged deep learning methodology—including multimodal inference pipelines that eliminate domain-specific training, few-shot meta-learning, and Vision Transformer (ViT)-based ensembling—to classify UC severity within the HyperKvasir dataset. We systematically evaluate multiple vision transformer architectures, discovering that a Swin-Base model achieves an accuracy of 90%, while a soft-voting ensemble of diverse ViT backbones boosts performance to 93%. In parallel, we leverage multimodal pre-trained frameworks (e.g., CLIP, BLIP, FLAVA) integrated with conventional machine learning algorithms, yielding an accuracy of 83%. To address limited annotated data, we deploy few-shot meta-learning approaches (e.g., Matching Networks), attaining 83% accuracy in a 5-shot context. Furthermore, interpretability is enhanced via SHapley Additive exPlanations (SHAP), which interpret both local and global model behaviors, thereby fostering clinical trust in the model’s inferences. These findings underscore the potential of contemporary representation learning and ensemble strategies for robust UC severity classification, highlighting the pivotal role of model transparency in facilitating medical image analysis.