Early Detection of Cystoid Macular Edema in Retinitis Pigmentosa Using Longitudinal Deep Learning Analysis of OCT Scans

TL;DR

This study uses deep learning on OCT scans to detect early signs of CME in retinitis pigmentosa patients, improving early intervention possibilities.

Contribution

The first use of longitudinal OCT data for AI-driven CME prediction in retinitis pigmentosa patients.

Findings

ResNet-34 achieved 98.68% accuracy in detecting CME from longitudinal OCT data.

The model showed high specificity (99.45%) and F1-score (98.36%).

Longitudinal data improved detection of subtle disease progression.

Abstract

Background/Objectives: Retinitis pigmentosa (RP) is a progressive hereditary retinal disorder that frequently leads to vision loss, with cystoid macular edema (CME) occurring in approximately 10–50% of affected patients. Early detection of CME is crucial for timely intervention, yet most existing studies lack longitudinal data capable of capturing subtle disease progression. Methods: We propose a deep learning–based framework utilizing longitudinal optical coherence tomography (OCT) imaging for early detection of CME in patients with RP. A total of 2280 longitudinal OCT images were preprocessed using denoising and data augmentation techniques. Multiple pre-trained deep learning architectures were evaluated using a patient-wise data split to ensure robust performance assessment. Results: Among the evaluated models, ResNet-34 achieved the best performance, with an accuracy of 98.68%, specificity of 99.45%, and an F1-score of 98.36%. Conclusions: These results demonstrate the potential of longitudinal OCT–based artificial intelligence as a reliable, non-invasive screening tool for early CME detection in RP. To the best of our knowledge, this study is the first to leverage longitudinal OCT data for AI-driven CME prediction in this patient population.

Figures

Tables

Table 1. Architectural distinctions and rationale for selecting the six dl models evaluated in this study.

Model	Architectural Characteristics	Rationale for Inclusion
ResNet-18/ResNet-34	Residual blocks, identity skip connections, stable deep training	Most widely used OCT classifiers, robust to overfitting, strong performance in CME/DME tasks
VGG16	Deep sequential 3 × 3 conv blocks, high parameter count	Classical OCT baseline; widely used for transfer learning on OCT
AlexNet	Early CNN with large initial filters, shallow architecture	Lightweight baseline, useful historical benchmark in OCT literature
Xception	Depthwise-separable convolutions, efficient feature extraction	Strong performance with limited OCT data, captures localized retinal patterns efficiently
ViT	Patch embeddings + self-attention, global context modeling	Modern architecture; relevant to OCT where global spatial relations matter, emerging in ophthalmology

Full text

1. Introduction

Retinitis pigmentosa (RP) is a progressive, inherited retinal dystrophy that primarily affects rod photoreceptors, leading to gradual vision loss [1,2]. While the peripheral vision is typically affected first, the occurrence of cystoid macular edema (CME) can cause additional acute or subacute deterioration in central vision [3]. As a result of fluid buildup in the macula, CME develops, leading to the formation of cyst-like spaces. The exact pathogenesis remains unclear, but proposed mechanisms include macular Müller cell impairment, autoimmune responses, blood-retinal barrier breakdown, vitreous traction, and dysfunction of the retinal pigment epithelial cells (RPE) pump [2,4]. Symptoms typically involve progressive loss of visual field, night vision impairment, and decreased visual acuity due to cone photoreceptor loss, ultimately resulting in central vision deterioration [5,6].

Early detection of CME in RP patients is essential. CME impacts central vision in 25% of cases, worsening vision loss and increasing the need for costly eye care [7,8]. Delayed treatment often requires more invasive procedures with higher risks. Timely screening can prevent CME from becoming chronic, reduce the need for complex treatments, and improve patient outcomes [9,10]. Effective early detection preserves vision, minimizes drug side effects, and enhances treatment prognosis. Failure to diagnose CME promptly may lead to disease progression and resistance to treatment, which requires a more cautious and stepwise approach [2,11]. Early detection of CME in RP patients is crucial and can be effectively achieved with OCT imaging [8]. However, research on CME in RP patients is limited by insufficient RP-specific data and the lack of longitudinal imaging.

OCT offers high-resolution, non-invasive imaging that detects subtle retinal abnormalities like subclinical CME [12]. Spectral-domain OCT (SD-OCT) excels in measuring central foveal thickness and identifying CME without fluorescein dye. Its non-invasive nature, with its ability to monitor retinal changes and treatment response, enhances its utility [11,13]. Additionally, OCT’s safety, speed, and repeatability make it ideal for routine screening and managing CME in RP patients.

Despite these advantages, the complexity of retinal patterns in OCT scans can lead to missed diagnoses during manual review [2,6]. Deep learning (DL), a subset of artificial intelligence (AI), has shown great promise in identifying hidden patterns in OCT images, which can aid in the early detection of CME and improve screening for RP patients [14,15,16,17,18]. The consistent, high-resolution 3D data provided by macular OCTs makes them particularly well-suited for DL applications [19,20]. Furthermore, recent studies have shown the potential of combining DL with telemedicine tools to enhance screening and diagnostic capabilities in primary care settings [21,22,23,24].

Over the past five years, DL for OCT analysis has expanded rapidly, with many studies reporting strong performance in classifying retinal diseases. CNNs have been used to identify DME, CME, AMD, and other macular pathologies from B-scan and volumetric OCT data [25,26]. Recent work includes explainable models such as DeepOCT [27], multitask approaches for DME classification [28], and hybrid CNN-RNN or Transformer models that capture spatial and contextual retinal information [15,29]. Most existing studies focus on cross-sectional diagnosis rather than longitudinal prediction. No prior work has examined future CME development in RP using longitudinal OCT data. This gap motivates the predictive framework proposed in the present study.

Convolutional neural networks (CNNs) are particularly promising, which have shown accuracy rates between 80% and 99% in detecting and segmenting CME in diseases such as diabetic macular edema (DME) and age-related macular degeneration [30]. However, there is a significant research gap in CME in RP patients, despite the high clinical relevance. CME in RP presents unique challenges due to factors such as Müller cell dysfunction and retinal pigment epithelium abnormalities, which differ from other retinal diseases. While early DL models like ResNet and AlexNet have shown promise, their effectiveness is limited by the lack of representative RP data and the progressive [19,31,32], heterogeneous nature of photoreceptor loss in these patients. Most existing studies rely on cross-sectional datasets and overlook the importance of longitudinal imaging, which is vital for identifying subtle, progressive changes that precede the onset of CME. This lack of long-term data hampers early detection and delays timely intervention.

To address this gap, our study leverages a unique longitudinal OCT dataset that includes multiple follow-up visits, enabling the development of predictive DL models tailored to RP. By capturing early patterns and risk factors associated with CME progression, our goal is to support clinicians in making earlier, more informed treatment decisions—ultimately improving patient outcomes.

2. Materials and Methods

2.1. Data Set

In this retrospective cohort study, 570 longitudinal OCT images were collected from 114 eyes diagnosed with RP, with or without CME complications. The study population had a mean age of 42.2 ± 14.43 years and consisted of 51% males. To facilitate the early detection of CME, imaging data from patients’ earlier clinical visits were analyzed and labeled based on the development of CME in follow-up visits. This approach involved reviewing the earlier visit images and categorizing them based on whether CME was subsequently diagnosed. CME was defined as cases in which CME was found in the OCT scans, whereas No CME referred to cases with no CME at the time of imaging. However, No CME does not invariable classify as a normal or healthy retina but includes RP patients with other retinal abnormalities who did not satisfy the clinical criterion for CME. Figure 1 shows the labeling process. Group A includes images from patients without CME at both initial and follow-up visits. Group B includes images from patients without CME initially but with CME in follow-up. This labeling enabled analysis of early OCT images (A1, B1) based on later CME development (A2, B2) to train a DL model for early CME detection in RP patients. Following the guidelines, which required five scans per eye, a total of 270 slides were collected from RP patients with CME complications, alongside 300 slides from patients without CME complications.

The study was approved by the Research Ethics Committees of Vice-Chancellor in Research Affairs—Shahid Beheshti University of Medical Sciences, Tehran, Iran (IR.SBMU.RETECH.REC.1403.608), and informed consent was obtained from all participants. All methods were performed following relevant guidelines and regulations, including the principles outlined in the Declaration of Helsinki. Data were sourced from the Iranian National Registry for Inherited Retinal Diseases (IRDReg^®^) between 2013–2024 [33].

Each OCT scan was independently reviewed and annotated by three retina specialists, each with more than five years of experience. Discrepancies in annotations were resolved through structured consensus meetings. In these meetings the reviewers jointly examined and discussed the cases to arrive at a final agreement on labels. Inter-rater agreement was assessed during a pilot phase to establish consistency across annotators. Additionally, a random subset of annotated scans (10%) was re-evaluated by a senior retina specialist to confirm labeling quality and minimize bias. Identifying information was removed and each patient was assigned a unique number.

Patients were informed about procedures involving pupil dilation, which may temporarily affect vision and were instructed to remain still during imaging for accuracy. Patients with difficulty were given additional support. Operators calibrated imaging devices, positioned patients correctly, and reviewed image quality, repeating the process if necessary. The collected data was divided into two groups: RP patients without CME during follow-up and those who developed CME after the initial consultation. Spectral-domain optical coherence tomography (SD-OCT, Heidelberg Engineering, Heidelberg, Germany) was utilized, employing a 6 × 6 mm 3D macular scan protocol with an image resolution of 400 × 500 pixels.

2.2. Data Preparation

We performed image resizing, data augmentation, and noise removal to prepare the dataset for improved model performance. The optimal size for resizing OCT images depends on the DL model, image characteristics, and computational resources. OCT images, being high-resolution, often require resizing to reduce complexity and memory usage [34,35]. Studies suggest common resizing dimensions of 244 × 244 or 256 × 256 pixels, suitable for CNN architectures, including pre-trained models like those on ImageNet [36,37,38]. We extracted the OCT images from the Heidelberg device to focus on the retinal cross-section and then resized them to 224 × 224 pixels for uniform analysis.

We use Fast Non-Local Means Denoising (fastNlMeansDenoising) from OpenCV (version 4.10.3) to reduce noise, employing a 7 × 7 window and a 21 × 21 search window. Additionally, we adjust contrast and brightness with alpha = 1.0 and beta = 0 to ensure uniform light intensity (Figure 2, step 2). All OCT images were organized into patient-specific folders before preprocessing to prevent data leakage. Data augmentation (rotation, flipping, and scaling) was applied only within each patient’s folder so that original and augmented images remained grouped by patient. After augmentation, a strict patient-wise split was performed: each patient’s entire folder was assigned exclusively to either the training (70%), validation (15%), or test (15%) subsets. This resulted in 40 patients in the training set, 9 patients in the validation set, and 8 patients in the test set. No images from the same patient appeared in more than one subset. This procedure expanded the dataset, balanced the class distribution, and ensured the complete elimination of both image-level and subject-level leakage. The number of OCT scans per patient was not uniform due to variations in scan quality and availability. Consequently, the test subset contained 8 patients who collectively contributed 68 original OCT scans. After augmentation, this resulted in 342 test images, as reflected in the confusion matrix.

2.3. Deep Learning Models

To ensure a representative comparison across classical, residual, efficient, and transformer-based architectures, six widely validated models, including ResNet-18, ResNet-34, VGG16, AlexNet, Xception, and Vision Transformer (ViT), were selected. Their use is supported by prior OCT-based DL research and recent reviews, including our own systematic review on AI for CME detection [19], which identified these architectures as the most frequently applied and best-performing families in retinal OCT analysis. ResNet models offer stable training through residual connections, VGG16 and AlexNet serve as well-established baselines, Xception provides efficient depthwise-separable convolutions suitable for retinal structure modeling, and ViT introduces global self-attention, which has recently shown promise in ophthalmic imaging. A concise overview of the architectural distinctions and motivations for their inclusion is provided in Table 1. These six architectures were implemented and evaluated for their effectiveness in early CME detection in patients with RP [19,39,40]. These models are widely used in transfer learning since they have been pre-trained on large datasets like ImageNet [41]. These models have shown higher performance in ophthalmic image processing. Although ResNet-18, VGG16, and AlexNet are older, they still perform well with OCT images [19,42,43,44]. The dataset was split into training, testing, and validation sets in a 70:15:15 ratio using the sci-kit-learn library, employing a subject-wise splitting method to ensure that images from the same patient were only present in one of the subsets. A fixed random seed was used for all stochastic processes including dataset splitting, weight initialization, and data augmentation to ensure reproducibility. This kept initial conditions identical across architectures, reduced variability, and enabled reliable model comparisons. The models received SD-OCT images of RP patients, both with and without CME complications, as input and generated a numerical output, providing a real value between 0 and 1 for each prediction. All deep learning models were trained using a unified and standardized training setup to ensure a consistent baseline for architectural comparison [19,41]. Input OCT images were resized to 224 × 224 pixels, a dimension widely adopted for CNN architectures transferring from ImageNet [36,38]. The loss function used was nn.CrossEntropyLoss() from the PyTorch (version 2.0.1) library. Training was conducted for a maximum of 30 epochs, employing an early stopping mechanism with a patience of five epochs based on validation loss to prevent overfitting.

Regarding optimization, we evaluated both Adam and SGD, strictly following protocols established in recent OCT classification literature [19,39,40]. Adam was utilized with default parameters (beta_1 = 0.9, beta_2 = 0.999) and a learning rate of 0.001. SGD was configured with a learning rate of 0.01, momentum of 0.9, and weight decay of 1 × 10^−4^. These configurations are consistent with standard practices in medical image analysis [20,34]. A batch size of 32 was chosen to balance GPU memory constraints with convergence stability. While we acknowledge that extensive hyperparameter tuning per model could yield marginal gains, we adopted a fixed-hyperparameter strategy. This approach isolates the impact of the architectural design as the primary variable, minimizing the bias introduced by unequal optimization efforts across different models [20,32].

(Figure 2, Step 3, 4).

2.4. Performance Evaluation and Statistical Analysis

Model performance was evaluated on a separate validation set during training and on a test set post-training, as shown in Figure 2 steps 3 and 4. Receiver operating characteristic (ROC) curves were generated to visualize the tradeoff between sensitivity and specificity. Accuracy, specificity, precision, recall, F1-score, and a confusion matrix including True Positive (TP), False Positive (FP), False Negative (FN), and True Negative (TN) were used to assess model efficiency and classification performance.

3. Results

3.1. This Patient and Imaging Characteristics

The study consists of 57 patients, including 27 with CME and 30 without CME, contributing data from a total of 114 eyes. Each eye was represented by 5 SD-OCT slides, with 270 from eyes with CME and 300 from eyes without CME. Through data augmentation, the total number of images increased to 2280, maintaining a balanced distribution.

3.2. Training Loss and Accuracy

The ROC curves derived from the testing process of models are presented in Figure 3. The average area under the ROC curve (AUC) for all predictors was 0.95. Among the 6 algorithms analyzed, ResNet-34, ResNet-18, and VGG16 achieved the highest AUC indices of 0.98, 0.97, and 0.97, respectively.

ResNet-34 and ResNet-18 had the lowest loss values, 0.014 and 0.019, respectively, significantly lower than other models, which had a minimum loss of 0.021. Figure 4 illustrates this comparative loss analysis.

Similarly, ResNet-34, ResNet-18, and VGG16 achieved the highest accuracy values (0.986, 0.976, and 0.970). As shown in Figure 5, ViT, Xception, and AlexNet architectures had lower accuracy compared to these models.

3.3. Detection Performance

In addition, to check the predictive and detection performance of the models, the indicators of specificity, precision, F1-score, recall, and accuracy based on the best performance of the optimizer were examined, the full details of which are given in Table 2.

Overall, ResNet-34, ResNet-18, and VGG16 provided the best overall accuracy, reaching 0.986, 0.976, and 0.97, respectively. VGG16 achieved high precision (0.9931) and specificity (0.9934), minimizing false positives. Its F1-score (0.9695) and recall (0.947) indicate strong true positive detection but with some missed cases. Accuracy (0.9702) confirms overall reliability. Using the ADAM optimizer, VGG16 is optimal for reducing false positives while maintaining high sensitivity in CME detection. Xception and ViT provided balanced performance, with Xception achieving 0.9603 specificity and 0.9404 recall, and ViT reaching 0.946 recall. AlexNet had moderate accuracy (0.9272) but the lowest recall (0.8742). Overall, ResNet-34 is optimal for minimizing false positives, ResNet-18 is best for detecting true positives, and VGG16, Xception, and ViT offer strong general-purpose performance for CME screening.

The choice of optimizer impacts model performance, with SGD and ADAM being used across different architectures. ResNet-34 and ResNet-18, which achieved the highest overall accuracy (0.9868 and 0.9765, respectively), performed equally well with both SGD and ADAM, indicating their robustness to optimizer selection. VGG16, Xception, and AlexNet, all trained with ADAM, showed strong precision and specificity, with VGG16 achieving the highest precision (0.9931) and specificity (0.9934). ViT, optimized with SGD, had lower precision (0.9259) and accuracy (0.9356) compared to ADAM-optimized models, suggesting that ADAM may provide better overall performance for CME detection. Models trained with ADAM generally showed higher specificity and precision, reducing false positives. The results suggest that while ResNet architectures are optimizer-independent, ADAM tends to enhance performance in models where reducing false positives is critical.

The confusion matrices for different models show the classification of cases with and without CME. The best results were achieved by a ResNet-34 (accuracy of 98.68% and F1-score of 98.36%), while the model with a ViT showed relatively low performance at 93.56% accuracy. ResNet-based models and VGG16 maintained high recall and specificity which means equal reduction in both false positive and false negative. However, certain models like AlexNet showed low recall (87.42%) due to difficulties in detecting some CME cases. In general, among different models, evaluation metrics from the confusion matrix result are also well matched, advanced ones like ResNet-34 and VGG16 would be more favorable for a more precise classification of CME (Figure 6).

4. Discussion

RP progressively affects rods and cones, with CME contributing to visual impairment in 10–50% of cases [3,45]. While DL models have shown promise in detecting CME in DME and age-related macular degeneration, there is a clear gap in studies focused on CME in RP. Although recent DL studies have reported high performance in OCT-based detection of DME, AMD, and other macular disorders, these works remain predominantly cross-sectional and do not explore longitudinal prediction. This important gap in the literature highlights the relevance and novelty of our RP-specific longitudinal framework. Existing work often relies on cross-sectional data, which fails to capture the dynamic nature of CME progression. Furthermore, many models are trained on single-slice OCT inputs, limiting their ability to detect spatially variable fluid patterns. In contrast, our study leveraged longitudinal OCT data and multi-section inputs, allowing for improved detection of early CME changes across layers such as INL and ONL. Among DL models developed in this study, ResNet showed the highest performance in early detection of CME. These findings address a critical gap in the literature and demonstrate the potential of longitudinal DL approaches in improving CME detection and management in RP. While our results show strong performance for early CME detection in RP, direct numerical comparison with prior OCT studies is not appropriate because those studies involve different diseases, imaging conditions, and datasets. Our reported accuracy reflects performance within the specific context of RP-associated CME, not a head-to-head comparison with other retinal disease studies.

Although our model achieved high accuracy (AUROC 0.98), these results should not be interpreted as superiority over studies on other retinal diseases or populations. Differences in disease mechanisms, imaging devices, dataset size, and class balance prevent direct numerical comparison. Performance values from non-RP datasets are used only as contextual benchmarks.

Strengths of this study include the use of a longitudinal RP dataset, strict patient-wise splitting, and evaluation of multiple established architectures under a unified training pipeline. Limitations include the small RP sample size, lack of external validation, and the rarity of RP-associated CME, which limits generalizability. These points motivate future work involving multicenter datasets, cross-device external validation, longer longitudinal follow-up, and integration of explainable AI tools.

Models based on DL have been explored for diagnosing CME in various retinal diseases, but few studies have focused on rare diseases like RP [19]. DL models require substantial amounts of well-labeled data for effective training and validation to detect subtle patterns. While public datasets such as the 1000 OCT images used by Ahmed et al. for diabetic CME detection [46], provide value, private datasets often offer improved model performance due to their specificity, clinical relevance, and contemporaneity [47,48]. Our study used a unique dataset of 2280 OCT images from RP patients, collected under expert supervision. To the best of our knowledge, it is the first to identify early CME in patients with RP. Additionally, we used five scans from each eye to train models for CME prediction, which differentiates our approach from studies focused on the detection of CME in other retinal diseases.

Compared to previous studies on CME detection in non-RP populations, our model outperformed several benchmarks. For instance, Kaothanthong et al. reported an accuracy of 94.8%, while Bai et al. achieved AUROC of 0.99 in detecting CME across broader retinal conditions [16,39]. Our ResNet-34 model achieved an AUROC of 0.98, sensitivity of 97.12%, and specificity of 99.45%, indicating strong potential for early CME detection in RP patients. The comparative overview of our study with previously published works in this area of AI-assisted retinal disorder detection using retinal OCT images is presented in Table 3. In our study, the ResNet-34 model showed the best performance among all implemented algorithms with values of sensitivity, specificity, and AUROC of 97.12%, 99.45%, and 0.98%.

Table 3 provides a qualitative overview, as the included studies differ in sample size, patient populations, imaging protocols, and disease categories, and therefore incorporate OCT-based works from other retinal diseases for contextual comparison. The 2280 images in the current study represent the fully preprocessed and augmented set derived from 570 original OCT scans.

To develop an effective CME screening strategy for community settings, it is important to think about several key factors. This includes the impact of misdiagnoses, the burden of false positives on specialized care resources, and cost-effectiveness [15]. Among the evaluated models for screening early detection of CME in RP patients, ResNet-34 showed the highest accuracy of 0.98 and specificity of 0.99, making it the most reliable for identifying true negatives that offer strong general-purpose performance for CME screening.

ADAM enhanced precision and specificity in most architectures, which suggests its effectiveness in reducing false positives, particularly in models like VGG16, Xception, and AlexNet. In contrast, ResNet models maintained consistently high accuracy regardless of optimizer choice, indicating their robustness and adaptability. The lower precision and accuracy of ViT with SGD further reinforce ADAM’s advantage in optimizing performance for CME detection. These findings suggest that while ResNet architectures offer stability across optimizers, ADAM is preferable for models where high specificity and precision are critical.

Our study identified CME early in patients with RP. Unlike many DL studies that focus on CME in non-RP populations, particularly diabetic patients, this study offers preliminary evidence that could help monitor patients and their family members at risk for RP complications. Moreover, the longitudinal nature of the dataset used in this study adds a novel dimension to AI-driven CME screening. By incorporating temporal data, our study offers a more comprehensive view of disease progression and highlights the potential of longitudinal follow-up data to improve both sensitivity and specificity in CME detection. Future research should explore how such datasets can be expanded and applied to other rare retinal conditions. One limitation of this study is the absence of external validation datasets. Future research can address this by incorporating data from diverse imaging sources and populations. Although RP with CME complications is rare, we tried to maintain gender balance to ensure fairness and reduce bias. Furthermore, to our knowledge, no public OCT dataset includes RP images with or without CME, and none provide longitudinal follow-up. Available datasets focus on DME, CNV, drusen, or AMD, which differ fundamentally from RP-associated CME in anatomy and pathophysiology. Consequently, external benchmarking with public datasets is not feasible or scientifically relevant for early CME detection in RP. We believe that combining our approach with technologies like augmented reality and telemedicine could significantly improve CME screening for RP patients, both in clinical settings and in remote or underserved areas.

DL models often function as “black boxes,” which makes it difficult to understand how they arrive at their conclusions, especially when detecting CME early in RP patients [49,50]. To improve model interpretability, future studies should use explainable AI (xAI) methods to enhance model transparency [51,52]. By integrating xAI techniques, we may better understand the reasoning behind the DL model’s predictions for CME diagnosis in RP patients. It would be helpful to explore how anatomical data and imaging metrics, such as retinal layer thickness, can aid in predicting CME. Emerging tools like Large Language Models (LLMs) can help integrate imaging data with clinical context to improve model transparency and workflow integration [53]. Additionally, immersive decision support systems using spatial computing and Augmented Reality (AR) may offer a promising path to evaluate and interact with AI-AR fusion tools for patient care in various sets of clinical workflows, such as when multiple caregivers collaborate in diagnosis and treatment processes [54,55,56,57,58]. Finally, studying how these DL models can fit into clinical workflows and affect patient care and outcomes is another vital area for future research [59,60].

5. Conclusions

This study investigated how DL models can help in identifying CME at an early stage in patients with RP. Among the various models evaluated, ResNet-34 emerged as the most accurate and specific. These findings highlight the potential for using DL-based OCT analysis in clinical settings to support earlier diagnosis and improve the management of RP-related complications. Future research should aim to evaluate these results by testing more diverse datasets and imaging methods. Using DL-based OCT screening holds potential promise for improving CME detection, particularly in resource-limited environments where early intervention can make a big difference.