# Improving Brain Tumor Detection by Cortical Surface and Vessels Segmentation Through RGB-to-HSI Transfer Learning

## TL;DR

This paper introduces a two-stage AI method to improve brain tumor detection during surgery by first identifying the brain surface and blood vessels, then classifying healthy and non-healthy tissue.

## Contribution

A novel two-stage segmentation strategy using RGB-to-HSI transfer learning to enhance tumor detection by separating cortical and vascular segmentation.

## Key findings

- The method achieves a 15.48% increase in F1 score for tumor segmentation.
- Brain cortex segmentation reaches a mean Dice similarity coefficient of 92.08%.
- Blood vessel detection accuracy is 95.42% in the HSI dataset.

## Abstract

Precise delimitation of brain tumors during surgical intervention remains challenging. Hyperspectral imaging, which captures information beyond the visible spectrum, can be a valuable tool for identifying biological tissues when combined with deep learning algorithms. However, artificial-intelligence-based methods often struggle to distinguish malignant areas from highly vascularized structures, leading to potential misclassification. To address this limitation, we propose a two-stage segmentation strategy where a model first identifies the exposed brain surface and blood vessels. Then, a secondary model classifies pixels from the remaining tissue. To train these models, a set of pseudo-labels is generated with minimal manual intervention using both RGB and hyperspectral images acquired during surgical procedures. By segmenting the brain cortex and its vessels, the proposed approach simplifies the multitissue classification into a binary classification of healthy versus non-healthy tissue. This strategy improves tumor segmentation accuracy, exploring the potential use of hyperspectral imaging for real-time intraoperative brain tumor guidance.

Background: Accurate in vivo brain tumor detection using hyperspectral imaging (HSI), a non-invasive technique that captures spectral information beyond the visible range, is challenging due to the complexity of biological tissues and the difficulty in distinguishing malignant from healthy areas. Conventional neural-network-based methods often misclassify tumor tissue as blood vessels, largely due to high vascularization and the scarcity of annotated data. Method: To address this issue, this work proposes an underexplored approach that decomposes the problem into two tasks: (1) segmentation of the brain cortical surface and its blood vessels, and (2) segmentation of biological tissues within the segmented craniotomy site. The cortical segmentation task is addressed independently of the segmentation model used in the second stage. To achieve this, a set of pseudo-labels is generated from RGB and HSI captures acquired during in vivo brain surgeries. These pseudo-labels support a multimodal training strategy that leverages both imaging domains, yielding a model capable of segmenting the craniotomy site and the blood vessels contained in it. The model is further refined on HSI using weakly supervised fine-tuning with sparse ground truth annotations. Results: The final segmentation map combines cortical and tissue segmentation outputs, considering only cortex pixels not overlapped by vessels as potential tumor regions. This simplifies the HSI tissue segmentation task, reframing it as a binary segmentation of healthy vs. other tissues, while still enabling a comprehensive multiclass output. Conclusions: The proposed method achieves up to a 15.48% increase in F1 score for the tumor class, while segmenting the brain cortex with a mean Dice similarity coefficient (DSC) of 92.08% and accurately detecting 95.42% of labeled blood vessel samples in the HSI dataset.