Evaluation and optimization of deep learning models for enhanced detection of brain cancer using transmission optical microscopy of thin brain tissue samples

TL;DR

This study evaluates deep learning models, specifically ResNet50 and DenseNet121, for detecting brain cancer from optical microscopy images, demonstrating DenseNet121's superior accuracy and robustness on a curated dataset.

Contribution

The paper introduces a transfer learning protocol and compares CNN architectures, highlighting DenseNet121's effectiveness for brain tissue classification in limited medical datasets.

Findings

DenseNet121 achieved 88.35% test accuracy

DenseNet121 showed higher precision and recall than ResNet50

The approach demonstrates robust generalization with minimal bias

Abstract

Optical transmission spectroscopy is one method to understand brain tissue structural properties from brain tissue biopsy samples, yet manual interpretation is resource intensive and prone to inter observer variability. Deep convolutional neural networks (CNNs) offer automated feature learning directly from raw brightfield images. Here, we evaluate ResNet50 and DenseNet121 on a curated dataset of 2,931 bright-field transmission optical microscopy images of thin brain tissue, split into 1,996 for training, 437 for validation, and 498 for testing. Our two stage transfer learning protocol involves initial training of a classifier head on frozen pretrained feature extractors, followed by fine tuning of deeper convolutional blocks with extensive data augmentation (rotations, flips, intensity jitter) and early stopping. DenseNet121 achieves 88.35 percent test accuracy, 0.9614 precision, 0.8667 recall, and 0.9116 F1 score the best performance compared to ResNet50 (82.12 percent, 0.9035, 0.8142, 0.8563). Detailed analysis of confusion matrices, training and validation curves, and classwise prediction distributions illustrates robust convergence and minimal bias. These findings demonstrate the superior generalization of dense connectivity on limited medical datasets and outline future directions for multi-class tumor grading and clinical translation.