Relative Afferent Pupillary Defect Screening through Transfer Learning

TL;DR

This paper presents RAPDNet, an automated, transfer learning-based framework for reliable detection of relative afferent pupillary defect (RAPD), outperforming handcrafted methods with high sensitivity and specificity in clinical testing.

Contribution

The study introduces RAPDNet, a novel transfer learning approach transforming a deep learning model into a pupil detector for RAPD screening, validated through clinical trials.

Findings

RAPDNet achieved 90.6% sensitivity and specificity.

It outperformed all handcrafted algorithms across nine metrics.

The method demonstrated an AUC of 0.929 in ROC analysis.

Abstract

Abnormalities in pupillary light reflex can indicate optic nerve disorders that may lead to permanent visual loss if not diagnosed in an early stage. In this study, we focus on relative afferent pupillary defect (RAPD), which is based on the difference between the reactions of the eyes when they are exposed to light stimuli. Incumbent RAPD assessment methods are based on subjective practices that can lead to unreliable measurements. To eliminate subjectivity and obtain reliable measurements, we introduced an automated framework to detect RAPD. For validation, we conducted a clinical study with lab-on-a-headset, which can perform automated light reflex test. In addition to benchmarking handcrafted algorithms, we proposed a transfer learning-based approach that transformed a deep learning-based generic object recognition algorithm into a pupil detector. Based on the conducted experiments, proposed algorithm RAPDNet can achieve a sensitivity and a specificity of 90.6% over 64 test cases in a balanced set, which corresponds to an AUC of 0.929 in ROC analysis. According to our benchmark with three handcrafted algorithms and nine performance metrics, RAPDNet outperforms all other algorithms in every performance category.

Tables

Table 1. Table 1: Existing pupil datasets captured with closely-mounted setups and the RAPD-HD dataset.*

	Swirski [12]	Kasneci  [13]	Sippel [14]	ExCuSe [15]	ElSe [16]	Microscope [17]	LPW [18]	RAPD-HD
Number of videos	-	40	20	-	-	-	66	64
Resolution	640x480	384x288	384x288	384x288	384x288	512x388	640x480	1,920x1,080
Number of subjects	2	40	20	17	7	2	22	24
Acquisition setup	head-mounted camera system	head-mounted eye tracker	head-mounted eye tracker	head-mounted eye tracker	head-mounted eye tracker	microscope	head-mounted eye tracker	headset
Subject activity	-	driving	shopping	driving (9) shopping (8)	driving (5) indoor (2)	-	staring at a moving target	staring within a headset
Metadata	pupil ellipse, center position	center position	center position	center position	center position	center position	pupil ellipse, center position	swinging flashlight and neutral density tests, medical history

Table 2. Table 2: Dataset statistics

	No RAPD	RAPD +ve
Test cases (% of total)	32 (50.0%)	32 (50.0%)
Male-Female (% of total)	57.0%-43.0%	40.0%-60.0%
Age (mean $\pm$ $\sigma$, in years)	49.5 $\pm$ 15.98	58.3 $\pm$ 9.14
Prior History (% of group)	35.7%	100.00%

Table 3. Table 3: Pupil localization methods

	Starburst [22]	ExCuSe [15]	ElSe [16]	RAPDNet
Stage 1	Initialize pupil center as image center	Filtering contours from canny edge maps based on curvature and connected components	Selection of edges that most likely belong to elliptical edges	Classify overlapping patches as pupil and no pupil
Stage 2	Project rays outwardly from pupil center in all directions and mark pixels at high gradients as feature points	Curve selection using Starburst algorithm to determine feature points of elliptical curves	Ellipse fitting and selection on best edges	Sort the patches based on classification confidence and determine top-5 patches
Stage 3	Reiteration after updating pupil center based on feature points until convergence	Ellipse fitting on selected curve	Ellipse evaluation using manually determined ellipse properties that most resemble the pupil	Obtain the medium location of top-5 patch centers and crop a 60x60 region
Stage 4	Calculate the mean of all feature points	Obtain ellipse center as the pupil center	Obtain ellipse center as the pupil center	Obtain patch center as the pupil center

Table 4. Table 4: Validation and test accuracy of the developed pupil recognition algorithm based on transfer learning.

50x50	75x75	100x100	125x125	150x150
Validation Set
0.974	0.965	0.963	0.952	0.931
Test Set
0.957	0.954	0.954	0.948	0.933

Table 5. Table 5: Detection performance metrics.

Term	Description
Positive ($P$)	Number of RAPD positive cases
Negative ($N$)	Number of cases without RAPD
True positive ($TP$)	Number of correct RAPD detections
True negative ($TN$)	Number of correct no RAPD detections
False positive ($FP$)	Number of false RAPD detection
False negative($FN$)	Number of undetected RAPD cases
Area under Curve ($AUC$)	AUC for ROC plot
Term	Formulation
Precision	$TP/\left(TP+FP\right)$
Sensitivity/Recall/TPR	$TP/\left(TP+FN\right)$
FPR	$FP/\left(TN+FP\right)$
Specificity/TNR	$TN/\left(TN+FP\right)$
FNR	$FN/\left(TP+FN\right)$
Accuracy	$\left(TP+TN\right)/\left(TP+TN+FN+FP\right)$
Balanced Accuracy	$\left(sensitivity+specificity\right)/2$
$f_1$ score	$\frac{2\cdot precision\cdot recall}{precision+recall}$

Table 6. Table 6: RAPD detection benchmark.

Localization Algorithm	Patch Size	Smoothing	Precision	Sensitivity (TPR)	FPR	Specificity (TNR)	FNR	Accuracy	${𝒇}_{\mathrm{𝟏}}$	$𝑨𝑼{𝑪}_{𝑷𝑹}$	$𝑨𝑼{𝑪}_{𝑹𝑶𝑪}$
Starburst	image size/2	-	78.6	68.8	18.8	81.3	31.3	75.0	0.733	0.844	0.789
		mov avg	84.4	84.4	15.6	84.4	15.6	84.4	0.844	0.885	0.844
	60x60	-	87.5	65.6	9.4	90.6	34.4	78.1	0.750	0.860	0.795
		mov avg	88.9	75.0	9.4	90.6	25.0	82.8	0.814	0.894	0.848
ExCuSe	image size/2	-	75.8	78.1	25.0	75.0	21.9	76.6	0.769	0.741	0.749
		mov avg	78.8	81.3	21.9	78.1	18.8	79.7	0.800	0.762	0.780
	60x60	-	75.9	68.8	21.9	78.1	31.3	73.4	0.721	0.640	0.691
		mov avg	74.3	81.3	28.1	71.9	18.8	76.6	0.776	0.644	0.704
ElSe	image size/2	-	70.3	81.3	34.4	65.6	18.8	73.4	0.754	0.654	0.709
		mov avg	76.5	81.3	25.0	75.0	18.8	78.1	0.788	0.676	0.744
	60x60	-	75.0	75.0	25.0	75.0	25.0	75.0	0.750	0.655	0.730
		mov avg	78.1	78.1	21.9	78.1	21.9	78.1	0.781	0.670	0.743
RAPDNet	image size/2	-	90.3	87.5	9.4	90.6	12.5	89.1	0.889	0.940	0.914
		mov avg	90.6	90.6	9.4	90.6	9.4	90.6	0.906	0.956	0.929
	60x60	-	93.5	90.6	6.3	93.8	9.4	92.2	0.921	0.930	0.918
		mov avg	85.7	93.8	15.6	84.4	6.3	89.1	0.896	0.907	0.902

Equations

$$1.0-\frac{min(|{\Delta }_R|,|{\Delta }_L|)}{max(|{\Delta }_R|,|{\Delta }_L|)},$$

Full text

Figure 1: RAPD screening of a patient with positive RAPD on the right eye. Subjective examination is based on the visual assessment of the manual swinging flashlight test whereas objective examination is based on the automated swinging flashlight test combined with algorithmic analysis.

{IEEEkeywords}

Pupillary light reflex video dataset, relative afferent pupillary defect (RAPD), pupil detection, deep learning, transfer learning

1 Introduction

\IEEEPARstart

Approximately 1.3 million Americans are considered blind, which is expected to increase to 2.2 million by 2030 according to the National Eye Institute statistics [1]. Another 2.9 million Americans have low vision, which is estimated to be 5 million by 2030. The rapidly expanding geriatric population, the increased burden of eye diseases, and time spent in front of digital displays significantly increase the patient pool size [1]. Based on the American Academy of Ophthalmology survey, there were more than 35 million eye disease cases in 2010 [2] whereas the number of active ophthalmologists was less than 20 thousand in United States. Existing ocular technologies are not suitable for large-scale examinations worldwide because of their immobility, cost, lack of automation, and limited capabilities. In-person tests require experienced personnel and variation between subjective opinions can lead to inconsistencies and inaccuracies. Therefore, there is an emerging need for low-cost, portable, programmable, and multi-purpose vision monitoring and testing systems that conduct consistent and accurate eye exams.

In this study111This study is supported by the Georgia Research Alliance. The authors of this paper, Georgia Institute of Technology and Emory University are entitled to royalties in case of commercialization of the developed technology [3, 4, 5]., we focus on pupillary light reflex assessment, which can be an indicator of numerous conditions including but not limited to optic nerve disorders, trauma, autism, alcohol and recreational drugs, exposure to toxins, and response to infections [6]. Specifically, we investigate the RAPD condition, which corresponds to relative afferent pupillary defect. RAPD is based on the differences between the reactions of the eyes when they are exposed to light [7]. In healthy subjects, light stimulation on one eye should lead to equal constriction of both pupils and constricted pupils should enlarge equally when there is no stimulation. In case of positive RAPD, patient’s pupils constrict less or do not constrict when light stimuli swings from the unaffected eye to the affected eye as illustrated in Fig. Relative Afferent Pupillary Defect Screening through Transfer Learning.

Traditionally, RAPD is tested with a swinging flashlight test in which practitioner asks the patient to fixate on a distant target in a dark environment, swings the light source back and forth between eyes, and observes the size of the pupils and reactions [7]. An alternative to swinging flashlight test is the utilization of filters such as neutral density (NDF) [8]. A set of filters is placed between the light and the eyes in the NDF test, and the density of the filter that compensates for pupillary defects indicates the RAPD level. Even though existing methodologies are easy to practice, limited control over the environment and the subjectivity of the procedures can affect the reliability of the RAPD test significantly. A study conducted with experienced nurses in a postanesthesia care unit showed that pupillary examination findings significantly vary depending on the light source [9]. Moreover, practitioners have difficulty in detecting subtle abnormalities in case of small pupils, dark irises, and poorly reactive pupils [10, 11].

To eliminate shortcomings in terms of subjective bias and environmental conditions, digital pupilometers have been used in the literature [19, 20, 21]. Even though these clinical studies shed a light on the utilization of automated methods, they do not disclose utilized algorithms or experimental data. Therefore, to design RAPD detection algorithms, it was necessary to develop a data acquisition platform and conduct a clinical study. Previously, we developed an RAPD detection algorithm based on Starburst [22] algorithm and validated its performance over 32 cases [23]. In this study, we extend the handcrafted algorithm benchmark with ExCuse [15] and ElSe [16] algorithms and propse a new algorithm that outperforms all the benchmarked algorithms. Overall, the contributions of this study are five folds.

• ▪

We provide a detailed comparison of existing pupil datasets in terms of setup, subjects, size, and metadata.

• ▪

We benchmark the performance of three RAPD detection algorithms obtained from our framework using incumbent pupil localization and measurement techniques.

• ▪

We introduce a transfer learning-based approach to convert generic visual recognition algorithms into pupil detection and localization algorithms.

• ▪

We introduce an RAPD detection algorithm (RAPDNet) based on visual representations obtained from deep learning architecture AlexNet.

• ▪

Proposed algorithm RAPDNet achieves a sensitivity and a specificity of $90.6\%$ with an AUC of 0.929, which outperforms all benchmarked algorithms over 64 test cases.

Outline: We briefly analyze existing pupil datasets along with related algorithmic approaches in Section 2. We describe the test dataset in Section 3 and explain the RAPD detection framework in Section 4. We discuss the experiments in Section 5 and conclude our work in Section 6.

2 Related Work

The main mechanisms in an automated RAPD detection algorithm are can be divided into three main blocks as (i) pupil localization, (ii) pupil size measurement, and (iii) pupil size comparison of left and right eyes in the test sequence. The majority of existing pupil localization algorithms require initialization of pupil location estimation based on fixed location [22], thresholding [15], and pattern matching [12, 16]. After the initialization, preprocessing, edge detection, and shape fitting methods are commonly used to locate the pupils. Specifically, prepossessing leads to more descriptive representations, edge detection results in estimated pupil locations, and shape fitting eliminates false positive. Preprocessing techniques include adaptive thresholding [22], smoothing [22, 17], normalization [22, 16], morphological operations [12, 15, 16]. After preprocessing stages, edge detection is usually applied over the entire image [22], region of interest (ROI) [12], both ROI and entire image [15, 16], and multiple-scales [17]. Pupil location estimations are obtained from edge maps and false positives are minimized with shape fitting methods including RANSAC-based ellipse fitting[22, 12], least square-based ellipse fitting [15, 16], and circle fitting [17]. On contrary to existing handcrafted approaches [22, 12, 15, 16, 17], data-driven approaches were used for pupil localization [24, 25] 222We implemented methods in [24, 25] to benchmark but we could not reproduce their results. We contacted the authors for their implementation but they mentioned that their codes will not be publicly available until it was published so we only refer to them in the related work section.. Even though learning visual representations from domain-specific images is a promising alternative to handcrafting representations, training a network from scratch requires more data, time, and hyper-parameter optimization compared to using a pretrained model. Therefore, we proposed a transfer learning-based approach in which we train a single fully connected layer, which only took a few epochs and minutes to converge.

Once pupil locations are obtained, shape-fitting methods can provide pupil sizes as explained in [26, 27, 28]. Finally, size variation of right and left pupils can be compared to obtain an RAPD score. To validate and benchmark the performance of RAPD detection algorithms, we need a pupil dataset. Therefore, we analyzed and reported the main characteristics of existing pupil datasets in Table 1. The resolution of existing datasets varies between 384x288 and 640x480 (SD). Number of subjects participated in the pupil studies is in between 2 and 40. Most of the datasets [13, 14, 15, 16, 18] were obtained with a head-mounted eye tracker whereas a camera system was used in [12] and a microscope ocular in [17]. Subjects were involved in different activities during the acquisition of these datasets. In Kasneci [13] and Sippel [14], researchers investigated the effect of visual field loss in daily activities and the subjects were either driving or shopping, which also contributed to the majority of images in ExCuSe [15] and ElSe [16] datasets. Activities of the subjects were not specified in Swirski [12] and Microscope [17] datasets whereas subjects were staring at a moving target in the LPW [18] dataset. All of the analyzed datasets [12, 13, 14, 15, 16, 17, 18] provide pupil center position as metadata and pupil ellipses are provided in Swirski [12] and LPW [18].

The shortcomings of existing pupil datasets are three folds in terms of detecting RAPD. First, the majority of existing pupil datasets are based on static images instead of videos, which makes it difficult or impossible to assess pupillary reflex. Even though Kasneci [13] and Sippel [14] datasets were originally obtained as videos, static images were provided and utilized in pupil detection experiments [15, 16]. Labeled pupils in the wild (LPW) is the only analyzed dataset [18] that provides video sequences. Second, limited control over the acquisition conditions in existing datasets make it unfeasible to assess light reflex. Subjects were exposed to varying stimuli in the acquisition process and environmental conditions are not reported in the datasets. Therefore, it is not possible to isolate and assess the pupillary reflex with respect to the light sources. Third, subjects were not examined for RAPD and there is no corresponding metadata. Therefore, it is not possible to validate RAPD algorithms in these datasets.

3 RAPD-HD Dataset

To compensate the shortcomings of existing datasets in terms of limited video data, uncontrolled environments, and lack of pupillary defect annotations, we generated an RAPD dataset with HD video sequences denoted as RAPD-HD dataset whose main characteristics are summarized in Table 1. In total, there are 64 test cases with 2 sequences per test case as left and right. Each test case includes multiple light reflex instances. Videos were originally acquired in high definition (HD) resolution and we provide RAPD labels for each video pair as RAPD positive or no RAPD.

Acquisition Setup: We developed a platform denoted as lab-on-a-headset [3, 4] that can perform an automated swinging flashlight test as shown in Fig. Relative Afferent Pupillary Defect Screening through Transfer Learning. Developed headset, shown in 2, provides a controlled environment in which each eye has an individual enclosure isolated from external environment. We stimulate eyes independently with a predefined light sequence and record their reactions.

Clinical Study: We obtained the approvals from the IRB committees of Emory University and Georgia Institute of Technology and started a clinical study at the Grady Memorial Hospital (IRB00099796). RAPD conditions of subjects were diagnosed by the practitioners involved in the clinical study with a swinging flashlight test and a neutral density filter test. The demographics of clinical subjects are summarized in Table 2. We can observe that average age of RAPD positive subjects are higher than no RAPD subjects. Moreover, all RAPD positive subjects have prior ocular history whereas majority of no RAPD subjects do not have ocular history.

4 RAPD Detection Framework

First, we detect the pupils in captured video sequences. Second, we measure relative pupil size variation during the test. Third, we measure the dissimilarity between left and right pupillary reactions to obtain an index between 0 and 1, which is denoted as the RAPD index. Healthy subjects should lead to an RAPD index close to 0 whereas subjects with RAPD positive should lead to higher values. In the following sections, we describe the algorithms that are utilized to perform pupil localization, pupil size measurement, and RAPD assessment.

4.1 Pupil Localization with Handcrafted Algorithms

We localize pupils with three handcrafted pupil localization algorithms in the literature including Starburst [22], ExCuSe [15], and ElSe [16]. Main characteristics of these algorithms are summarized in Table 3 along with an illustration of each stage in Fig. 3.

Starburst [22]: Pupil center is initialized as the image center in Stage 1 and rays are projected outwardly from the center to detect edge features via gradients in Stage 2. Then, pupil center is updated with mean feature location as shown with white circle from which rays are projected. These pupil center update and ray projection procedures are iterated until all the gradients are swept in Stage 3. Finally, pupil center is estimated based on mean location of the eventual feature points.

ExCuSe [15]: Curved segments are obtained from Canny edge maps which are filtered to exclude thin lines and small rectangular surfaces in Stage 1. Then, longest curve that encloses the darkest area is selected to project a ray between the curve and the image center in Stage 2. After the ray projection, an ellipse is fit around the estimated pupil in Stage 3 whose center corresponds to the pupil location.

ElSe [16]: Canny edge maps are extracted based on the edges split around orthogonal connections with more than two neighbors. These detected edges are thinned and straightened to enhance curved segment selection. Then, ellipse fitting is performed over detected edges and a subset of ellipses are selected based on the intensity within elliptic regions and their width-height ratio. Finally, the ellipse with the lowest inner gray values and a width-height ratio close to one is selected to obtain the pupil center estimate.

4.2 Pupil Localization based on Transfer Learning

Visual representations learned by state-of-the-art object recognition models include low level characteristics based on edges and shapes. We cannot directly use these models for localizing or detecting pupils in eye images because they are originally trained for generic object recognition tasks in large-scale datasets such as ImageNet [29]. However, the visual representations learned by these models can still be useful for recognizing and localizing eye structures. In this study, we focus on convolutional neural networks (CNNs), which include convolutional layers that learn visual representations and fully connected layers that map these visual representations to target classes. Specifically, we use the AlexNet [30] architecture that is based on five convolutional layers and three fully connected layers. Size of the convolutional layers vary between 11x11, 5x5, and 3x3.

In this study, we use a pretrained AlexNet [31], which was trained with ImageNet to classify generic objects into 1,000 classes. We transformed this object recognition model into a pupil detector with transfer learning as shown in Fig. 4. Specifically, we keep the convolutional layers that output visual representations and exclude all the pretrained fully connected layers at the end of AlexNet. Then, we added a single fully connected layer to map visual representations in the final layer to two classes as pupil and no pupil. In the training, we froze all the convolutional layers and trained the fully connected layer with image patches labeled as pupil and no pupil. We used a total of 3,620 images from datasets XII, X, XIII, XIV, XVI, XVII, and XVIII in the ElSe study [16] for training, validation, and testing. We visually inspected all 24 datasets in the ElSe study and selected the ones without lens and glare artifact. Data was split into 60% training set, 20% validation set, and 20% test set. In the ElSe dataset, a patch of 50x50 is wide enough to cover pupil region so we split the images into 50x50 overlapping patches with a stride of half the patch size to increase the number of patches. We labeled the patches that include pupil center as pupil images and the remaining ones as no pupil images. To balance the training and test set, we used a subset of no pupil images whose number is equal to pupil images. Moreover, we experimented with different patch sizes including 75, 100, 125, and 150.

Transfer learning of AlexNet resulted in a pupil detection model denoted as RAPDNet. To localize pupils in an image, we scan the image with overlapping patches and classify each patch as pupil or no pupil with RAPDNet. After obtaining a class for each patch, we sort the pupil patches based on their classification confidence and compute the median location of the top-5 pupil patches with the highest confidence. In recognition tasks, top-5 accuracy is a commonly used metric, which considers recognition as correct if the target class is among the top-5 estimates. Similarly, we look at the top-5 estimates but instead of focusing on one estimate, we obtain the median location of all 5 estimates. We proposed a patch-based localization instead of bounding box-based detection because dataset annotations only included pupil centers. Main characteristics of the proposed algorithm are summarized in Table 3 along with an illustration of each stage in Fig. 3.

4.3 Pupil Size Measurement

We perform pupil size measurement over image patches automatically cropped around the estimated pupil center. In the first crop configuration image size/2, estimated pupil region is cropped to half of the downsampled input image size. In the second crop configuration 60x60, a 60 by 60 patch is cropped around estimated pupil center. We detect and measure pupils with Circular Hough Transform, which transforms points in the 2-dimensional image plane into circular cones in the Hough space. Image points in the same circle will have cones in the Hough space that intersect at points, and these intersection points will indicate pupil size measurements.

In this study, we use the Hough gradient method to perform circular Hough transform (CHT) [32]. The number of detected circles per image is limited to one because there is at most one pupil in the field of view. Search range for radius is set to 5-30 pixels, which cover the average pupil range between 2mm and 8mm [33]. In CHT, there are two threshold parameters denoted as Canny threshold and accumulator threshold. Canny is the edge detection threshold and accumulator threshold corresponds to the detection confidence. Threshold values can be manually fine tuned according to target applications. However, in this study, we automate the threshold selection process by sweeping a range of values. For Canny threshold, we set the maximum sweep value to 255 (maximum gradient value of an 8-bit image) and the minimum sweep value to mean of the image. For accumulator threshold, the maximum sweep value is set to 100, which corresponds to the maximum accumulator value. Automated threshold selection starts from the maximum sweep values and iterates to lower values until a circle is detected.

4.4 RAPD Assessment

We measure pupil size variations in swinging flashlight video sequences by combining pupil localization algorithms with pupil size measurement methods, which leads to baseline algorithms. Pupillary light reflex measurements are processed with a median filter to eliminate outliers. Moreover, we also experiment with an additional moving average step in Section 5 to analyze the contribution of smoothing out signals. To measure dissimilarity between pupillary reflexes, we propose a dissimilarity index denoted as $RAPD~{}index$, which is formulated as

[TABLE]

where $\Delta_{R}$ is the pupil size change in the right eye, $\Delta_{L}$ is the pupil size change in the left eye, $|~{}|$ is the absolute value operator, $min$ is the minimum operator, and $max$ in the maximum operator. $RAPD~{}index$ is 0.0 when compared pupil size changes are identical whereas it converges to 1.0 as pupil size changes differ from each other. In addition to $RAPD~{}index$, we also use 1.0 minus correlation metrics in the Section 5 to compare the performance of alternative metrics. We need a numerical threshold to determine the RAPD status of the subjects based on the $RAPD~{}index$. To determine the classification threshold for all the benchmarked algorithms, we measure the sensitivity and specificity for all possible threshold values and select the values that correspond to highest sensitivity and specificity for each algorithm.

5 Experiments and Discussion

At first, we analyze the classification performance of the proposed pupil detection algorithm in Section 5.1. Then, we report and analyze the performance of RAPD detection algorithms including benchmarked hancrafted algorithms and proposed data-driven algorithm RAPDNet in Section 5.2.

5.1 Pupil Recognition

We report the classification performance of the developed pupil recognition algorithm in terms of accuracy and cross entropy loss in Fig. 5. In these plots, x-axis corresponds to the epoch index and y-axis corresponds to the performance metrics. In the training experiments, we tried different patch sizes including 50, 75, 100, 125, and 150. We combined the results of the models trained with different patch sizes. For example, at a specific epoch, minimum and maximum values in the plot are obtained from the minimum and the maximum performance of these five patch configurations. We can observe that classification performance variations attenuate after first eight epochs and converge to final validation performance.

With 60 million parameters and 650,000 neurons, training AlexNet would take many epochs until convergence. However, in the proposed algorithm, we use the convolutional features as they are, remove the pretrained fully connected layers, add a single fully connected layer with two output classes at the end, and only train this single layer. Given that we only train a single layer and the task is binary classification, the algorithm converges in a few epochs as shown in Fig. 5. Not only the algorithm converges in a few epochs, but it also leads to a 90+% accuracy after the first epoch. In order for an algorithm to reach high accuracies after one epoch and to converge after few epochs, the pretrained features should be a very good fit for the target application. Visual representations obtained from AlexNet include curvy and linear patches that can be transposed and weighted to represent a circular region and the intensity information can be used to differentiate the darker pupil region from the surrounding. Because of the direct usability of the pretrained feature representations, we were able to generalize well through transfer learning.

After the training stage, we test the models over the final test set and obtain their performance as reported in Table 4 in which we highlight the highest classification performance with a yellow background and a bold font. Overall, we can achieve an accuracy of $0.974$ in the validation set and an accuracy of $0.957$ in the test set. Based on the results reported in Table 4, 50x50 patch size leads to the highest classification accuracy in the validation set as well as in the test set. In the lab-on-a-headset setup, 60 pixel corresponds to a physical distance slightly more than 9mm, which is wide enough to cover dilated pupils. A patch size significantly wider than pupil size can lead to more false positives. Therefore, we used a patch of 60x60 in the RAPD-HD dataset experiments, which is re-scaled to 50x50 before feeding into the trained models. We set the stride as half of the patch size to be consistent with the training setup.

5.2 RAPD Detection

We describe the detection performance metrics in Table 5 and report the results in Table 6. We obtain the area under curve (AUC) values for receiver-operating-characteristic (ROC) plots by sweeping the detection threshold and measuring the swept area. For each localization algorithm, we report the results for two patch sizes and smoothing configurations. Detection threshold was set separately for each algorithm to maximize sensitivity and specificity. We highlight the algorithm results with bold font and yellow background that correspond to the highest detection performance for each performance metric in Table 6. Out of 9 performance metrics, RAPDNet leads to the highest performance in all the performance categories. In the remaining of this study, we utilize the patch size and smoothing configuration that leads to the highest AUC in ROC analysis for each algorithm. Specifically, we use moving average for all the algorithms, and we utilize half image size as the patch size for all the algorithms other than Starburst.

We investigate the importance of RAPD index by performing a ROC curve analysis with alternative dissimilarity metrics in Fig. 6. We measure the dissimilarity with 1 minus correlation metrics including Pearson, Spearman, and Kendall. In the ROC plot, each curve represents a pupil localization algorithm combined with a RAPD assessment metric. Y axis corresponds to true positive rate and x axis corresponds to false positive rate. We obtained the curves for each algorithm configuration by sweeping the detection threshold and highlighted the highest sensitivity and specificity with a colored circle. We can observe that RAPD index outperforms all the correlation metrics analyzed in this study. To further analyze the RAPD detection performance, we show the scatter plots of algorithmic configurations with highest detection performance for each localization algorithm in Fig. 7. Y axis separates the ground truth results as no RAPD and RAPD positive. X axis corresponds to scores obtained from RAPD detection algorithms. Overall, RAPDNet outperforms all other algorithms by maximum true decisions (29TP and 29TN) and minimum false decisions (3FN and 3FP).

Handcrafted algorithms Starburst, ExCuSe, and ElSe have numerous parameters that require tuning to obtain state-of-the-art performances, which makes it difficult to use these models in new datasets. However, in case of data-driven approaches, visual representations are learned from generic images rather than being handcrafted for specific tasks. Therefore, it can be relatively easier to use these generic representations for new tasks and datasets. In this study, we automatically learned the mapping between generic visual representations and target applications via transfer learning, which enabled achieving higher performance in the RAPD-HD dataset compared to benchmarked handcrafted algorithms.

6 Conclusion

We introduced an algorithmic framework based on pupil localization and pupil size measurement to assess relative afferent pupillary defect. Experimental results show that state-of-the-art pupil localization algorithms can lead to an AUC of 0.848 over 64 test cases. In addition to benchmarking handcrafted state-of-the-art algorithms, we introduced a transfer learning-based RAPD detection algorithm that can achieve an AUC of 0.929, which outperformed all benchmarked algorithms. In this study, temporal information was used in the postprocessing stage but not directly in the pupil detection stage. However, temporal relationship between frames can be utilized to obtain a more robust pupil detection algorithm. Therefore, instead of utilizing spatial and temporal information independently, we need to investigate the utilization of spatiotemporal information available in the video sequences for pupillary light reflex assessment.