AI pipeline for accurate retinal layer segmentation using OCT 3D images

TL;DR

This paper presents an AI pipeline combining multiple models for accurate retinal layer segmentation in OCT 3D images, addressing issues like eye movement and shadow suppression with improved registration and segmentation accuracy.

Contribution

It introduces a hybrid AI-based registration method, compares models for shadow detection, and develops an analytical brightness adjustment equation within a comprehensive segmentation pipeline.

Findings

Registration correlation > 0.7 using hybrid AI and optical flow.

Shadow detection with ResNet50 achieves 0.9 loss value.

Layer thickness estimation has only 6% error.

Abstract

Image data set from a multi-spectral animal imaging system is used to address two issues: (a) registering the oscillation in optical coherence tomography (OCT) images due to mouse eye movement and (b) suppressing the shadow region under the thick vessels/structures. Several classical and AI-based algorithms in combination are tested for each task to see their compatibility with data from the combined animal imaging system. Hybridization of AI with optical flow followed by Homography transformation is shown to be working (correlation value>0.7) for registration. Resnet50 backbone is shown to be working better than the famous U-net model for shadow region detection with a loss value of 0.9. A simple-to-implement analytical equation is shown to be working for brightness manipulation with a 1% increment in mean pixel values and a 77% decrease in the number of zeros. The proposed equation allows formulating a constraint optimization problem using a controlling factor {\alpha} for minimization of number of zeros, standard deviation of pixel value and maximizing the mean pixel value. For Layer segmentation, the standard U-net model is used. The AI-Pipeline consists of CNN, Optical flow, RCNN, pixel manipulation model, and U-net models in sequence. The thickness estimation process has a 6% error as compared to manual annotated standard data.