Comparative Assessment of Risk and Turn-Around Time between Sequence-Based Typing and Next-Generation Sequencing for HLA Typing

TL;DR

This study compares the risk and time efficiency of two HLA typing methods, finding that next-generation sequencing involves higher risk and longer processing time than sequence-based typing.

Contribution

The study provides a comparative risk and time analysis of HLA typing methods using FMEA and RAM, highlighting NGS's increased risk and TAT.

Findings

NGS has a significantly higher risk priority number (RPN) score than SBT.

NGS requires more steps and longer total turn-around time and hands-on time than SBT.

Transitioning to NGS for HLA typing increases risk and delays processing.

Abstract

This study compared laboratory risk and turn-around time (TAT) between sequence-based typing (SBT) and next-generation sequencing (NGS) for human leukocyte antigen (HLA) typing. For risk assessment, we utilized the risk priority number (RPN) score based on failure mode and effect analysis (FMEA) and a risk acceptability matrix (RAM) according to the Clinical Laboratory Standards Institute (CLSI) guidelines (EP23-A). Total TAT was documented for the analytical phase, and hands-on time was defined as manual processes conducted by medical technicians. NGS showed a significantly higher total RPN score than SBT (1169 vs. 465). NGS indicated a higher mean RPN score, indicating elevated severity and detectability scores in comparison to SBT (RPN 23 vs. 12, p = 0.001; severity 5 vs. 3, p = 0.005; detectability 5 vs. 4, p < 0.001, respectively). NGS required a greater number of steps than SBT (44 vs. 25 steps), all of which were acceptable for the RAM. NGS showed a longer total TAT, total hands-on time, and hands-on time per step than SBT (26:47:20 vs. 12:32:06, 03:59:35 vs. 00:47:39, 00:05:13 vs. 00:01:54 hh:mm:ss, respectively). Transitioning from SBT to NGS for HLA typing involves increased risk and an extended TAT. This study underscored the importance of evaluating these factors to optimize laboratory efficiency in HLA typing.

Full text

1. Introduction

Human leukocyte antigen (HLA) typing plays a pivotal role in determining compatibility between donor and recipients in transplantation and providing insights into the genetic basis of autoimmune diseases [1,2,3,4]. HLA typing can be classified into three resolution levels: low resolution (serological-level typing), medium resolution (allelic group typing), and high resolution (allelic-level typing) [1]. Serological-level typing indicated lower sensitivity and specificity, along with reported inconsistencies, prompting the introduction of DNA-based testing [5]. DNA-based testing is considered a reliable alternative, identifying two alleles of major HLA loci—HLA-A, -B, -C, -DRB1, -DQB1, and -DPB1 [6].

Sequence-based typing (SBT) is considered the gold standard at the allele level for determining the full nucleotide sequence of the alleles present [7]. However, SBT can result in genotype ambiguity, allele ambiguity, and phase ambiguity. A growing list of ambiguities required additional testing, leading to a time-consuming process, additional costs, and delayed results [7]. Next-generation sequencing (NGS) has recently been actively introduced for HLA typing, utilizing the massive parallel sequencing of clonal DNA molecules [8,9]. This approach significantly increased genomic coverage and effectively improved ambiguity resolution [10].

Clinical laboratories are consistently under pressure to provide qualified and reliable results within a limited time, without compromising laboratory efficiency [11]. Risk assessment contributes to enhancing patient safety and satisfaction, ensuring the delivery of high-quality laboratory results [12]. Laboratory efficiency can be achieved by decreased risk and turn-around time (TAT) [13]. A shorter TAT facilitates early diagnosis and treatment, reduces hospital stays, and improves patient satisfaction [14,15].

According to the International Organization for Standardization (ISO) 22367:2020 and Clinical Laboratory Standards Institute (CLSI) guidelines (EP18-A2), it was recommended to conduct a failure mode and effect analysis (FMEA) as part of the risk assessment when introducing new test methods or equipment into a clinical laboratory [16,17]. FMEA systematically identifies and evaluates potential failures and their causes across the entire process by a top-down approach [17]. The risk associated with each step was determined by calculating a risk priority number (RPN) score, derived from the multiplication of severity, occurrence, and detectability scores [18]. In addition, ISO 14971:2007 and CLSI guidelines EP23-A have introduced a risk acceptability matrix (RAM) for qualitative risk evaluation [19,20]. This matrix determined the risk based on the combination of two factors: the severity and occurrence of risk.

Various fields, such as clinical chemistry, diagnostic hematology, and transfusion medicine, have actively assessed the risks associated with testing methods to improve their laboratory efficiency [21,22]. However, to our knowledge, there have been no studies evaluating FMEA and RAM for HLA typing methods. Thus, this study aimed to evaluate the relative risk of SBT, considered the gold standard for high-resolution HLA typing compared to the newly introduced NGS, using FMEA and RAM. A comparative analysis of the TAT was also conducted to gain insights into the multifaceted aspects of SBT and NGS.

2. Materials and Methods

2.1. Sample Collection

This comparative study was conducted between July 2022 and May 2023 at Korea University Medical Center (KUMC) in Seoul, South Korea. The Institutional Review Board of KUMC exempted approval for the study protocol (K2023-0107) and waived the requirement for informed consent, given the utilization of the remaining samples after testing.

2.2. HLA Typing Using SBT and NGS

We analyzed samples for HLA-A, -B, -DRB1, and -DQB1 typing, with genomic DNA extracted from peripheral blood collected during the study. The initial steps, from sample collection to DNA extraction, were excluded from the comparative assessment as the process was identical for both SBT and NGS. The evaluation of risk and TAT for SBT and NGS was conducted from DNA amplification to the reporting of results. SBT was performed using AVITA plus^TM^ HLA SBT kits (Biowithus Inc., Seoul, Republic of Korea) and BigDye Terminator mixture (Biowithus Inc.) according to the manufacturer’s instructions. Polymerase chain reaction (PCR) amplification, PCR product clean-up, and sequencing reaction were performed using GeneAmp^®^ PCR System 9700 Thermal Cycler (PE-Biosystems, Foster City, CA, USA). The sequenced results were analyzed using BIOWITHUS SBT Analyzer software ver. 2.7.5 (Biowithus Inc.). NGS was performed using NGSgo^®^-AmpX c2 HLA kit (GenDx, Utrecht, The Netherlands) and NGSgo^®^ Library Full kit (GenDx, Utrecht, Holland) according to the manufacturer’s instructions. PCR amplification, fragmentation with adaptor ligation, and indexing PCR were performed using GeneAmp^®^ PCR System 9700 Thermal Cycler (PE-Biosystems, Foster City, CA, USA). Library amplification and sequencing were performed on a Miseq platform (Illumina Inc., San Diego, CA, USA). Sample genotypes were assigned using NGSengine version 2.30.0.28772 from GenDx based on the IPD-IMGT/HLA Database version 3.51.0.

2.3. Assessment of Risk and TAT

Risk assessment followed a systematic six-step process, as follows: (1) determination of the analytical test, (2) assembly of a team, (3) risk identification, (4) risk analysis, (5) risk assessment, and (6) action for risk improvement. The risk assessment team comprised a director, a medical doctor, and three qualified medical technicians. The director, experienced with FMEA, trained the other team members to identify potential defects, preventive actions, and outcomes. Team members independently conducted their risk assessment by reviewing each step of SBT or NGS and identifying risks based on their experience and information in the literature. Any discrepancies were analyzed and discussed until a consensus was reached for objectivity. For the risk assessment and evaluation, two tools, FMEA and RAM, were used. Using the FMEA rating scale, severity, occurrence, and detectability scores were assigned for each step [16,17,23]. Each score was determined as the mean of the scores assigned by the team members. The overall RPN score was calculated by multiplying the severity, occurrence, and detectability scores. Using the RAM rating scale, severity and occurrence were assigned for each step. Severity was categorized into five levels: negligible (inconvenience or temporary discomfort); minor (temporary injury or impairment not requiring medical intervention); serious (injury or impairment requiring medical intervention); critical (permanent impairment or life-threatening injury); and catastrophic (patient death). Occurrence was categorized into five levels: frequent (once per week); probable (once per month); occasional (once per year); remote (once every few years); and improbable (once during system measurement). Each step was judged as acceptable or unacceptable according to the classification described in Table 1. After the risk assessment and evaluation for each step, team members suggested preventive actions for steps with high RPN scores or unacceptable RAM results.

The TATs for each step in both SBT and NGS were recorded using a digital camera and stopwatch. The recording process was repeated three times to ensure precision in the comparison. The mean value of the three measurements was determined as the TAT for each step. The total TAT was calculated as the sum of TATs for all steps, further categorized into hands-on time and machine running time. Hands-on time represented manual steps performed by medical technicians during the test, while machine running time represented the non-manual steps.

2.4. Statistical Analysis

A Mann–Whitney U test was utilized for the nonparametric unpaired comparison between SBT and NGS. Statistical analysis was conducted using IBM SPSS v.26 (IBM Corp., Armonk, NY, USA), MedCalc v.19.1.7 (MedCalc Software, Ostend, Belgium), and Microsoft Excel 2013 on Windows 10. p-values < 0.05 were considered indicative of statistical significance.

3. Results

In SBT, the most common action for a potential defect was a repeat of the step, with outcomes resulting in delays or incorrect results (Table 2). According to the FMEA rating scale, the severity score ranged from 2 to 8, occurrence score ranged from 1 to 2, and detectability score ranged from 1 to 6. The resulting RPN scores for each step ranged from 3 to 72. Based on the RAM rating scale, severity showed 1 minor and 24 negligible steps, and occurrence showed 1 occasional, 5 remote, and 19 improbable steps. All steps revealed varying levels of risk, but according to the RAM rating scale, the risks were clinically acceptable.

Similarly, in NGS, a repeat of the step and delay or incorrect results were the most common actions and outcomes (Table 3). According to the FMEA rating scale, the severity score ranged from 3 to 8, occurrence score ranged from 1 to 2, and detectability score ranged from 3 to 6. The resulting RPN scores for each step ranged from 12 to 102. Based on the RAM rating scale, severity showed 1 minor and 43 negligible steps, and occurrence showed 1 occasional, 11 remote, and 32 improbable steps. All steps revealed varying levels of risk, but according to the RAM rating scale, the risks were clinically acceptable.

In SBT, the shortest step for hands-on time and overall TAT were the PCR product clean-up step and the interpretation and report step (00:02:30, 00:11:28 hh:mm:ss, respectively) (Table 4). The longest hands-on time, machine running time, and overall TAT were purification for the dye removal step (00:19:32, 05:56:57, 06:16:29 hh:mm:ss, respectively).

In NGS, the shortest hands-on time and overall TAT were the library quantification step (both 00:09:04 hh:mm:ss) (Table 5). The longest hands-on time was the interpretation and reporting step, (01:01:40 hh:mm:ss) and the longest machine running time and overall TAT were the next-generation sequencing step (17:00:00 and 17:13:43 hh:mm:ss, respectively).

NGS had a higher number of total steps than SBT (44 vs. 25 steps) (Table 6). The risks at all steps in both SBT and NGS were clinically acceptable. NGS showed a higher total RPN score than SBT (1169 vs. 465 RPN). NGS showed a higher mean RPN score with higher mean severity and detectability scores than SBT (RPN 23 vs. 12, p = 0.001; S 5 vs. 3, p = 0.005; D 5 vs. 4, p < 0.001, respectively). NGS indicated longer total TAT, total hands-on time, and hands-on time/step than SBT (26:47:20 vs. 12:32:06, 03:59:35 vs. 00:47:39, 00:05:13 vs. 00:01:54 hh:mm:ss, respectively).

4. Discussion

We conducted a comparative analysis of risk and TAT between SBT and NGS for HLA typing. Our findings revealed that the total RPN score of NGS was higher than SBT (1169 vs. 465 RPN), primarily due to the increased number of processing steps (44 vs. 25 steps). The median RPN score in NGS was also higher than SBT (23 vs. 12 RPN). NGS showed higher median severity and detectability scores than SBT (severity 5 vs. 3; detectability 5 vs. 4). Despite the elevated risk of NGS compared to SBT, it remained acceptable according to the RAM rating scale. The total TAT, total hands-on time, and hands-on time/step of NGS were longer than SBT (26:47:20 vs. 12:32:06, 03:59:35 vs. 00:47:39, 00:05:13 vs. 00:01:54 hh:mm:ss, respectively).

Recently, there has been a growing interest in assessing laboratory efficiency in clinical laboratories to improve productivity and patient safety [11]. The assessments involved the evaluation of risk assessment, TAT, and costs. FMEA, initially used in the industrial field, has recently found application in laboratory risk assessment. In a previous study by Nam et al. [24], a risk assessment comparison using FMEA was conducted between the tube test and automated column agglutination technology for anti-A/B isoagglutinin titers. The automated column agglutination technology demonstrated lower RPN scores than the tube test (337 vs. 1843 RPN), with the manual serial dilution step of the tube test accompanying the highest RPN score of 1080. This finding highlighted that RPN scores significantly increased when a processing step involved manual procedures, required skilled techniques, or included repetitive and numerous steps. NGS involved numerous complex steps, such as DNA fragmentation, adapter ligation, library pooling, size selection, and library quantification. The critical steps in NGS required more manual procedures and skilled techniques compared to SBT, leading to increased mean severity, mean detectability, mean RPN, and total RPN score. RAM, another indicator of laboratory risk management suggested by CLSI EP23-A, was also evaluated in this study [19]. Although the risk of NGS was higher than SBT, NGS was still deemed acceptable for HLA typing. Since there is currently no research applying the RAM in laboratory efficiency, further research is needed.

Our novel findings involved a comparison between FMEA and RAM. FMEA demonstrated significant advantages over RAM in comparing two or more items by quantifying the risk through scores [17]. However, a disadvantage of FMEA was its inability to establish a clear standard for evaluating the true acceptability of each processing step [23]. There was no standardized or validated cutoff for RPN scores. Nevertheless, Han et al. [23] suggested a careful review of the steps exceeding a score of 300 RPN. In this study, none of the processing steps in SBT and NGS exceeded 300 RPN. Regarding RAM, the rating scale appeared to be a more suitable method than FMEA for performing conformity assessments at each step [19]. However, RAM was not suitable for performing comparisons between different methods. In this study, a risk comparison between SBT and NGS could not be performed using RAM due to the same proportion of acceptable steps. Each laboratory should consider the appropriate criteria for conducting a risk assessment or evaluation based on its purpose.

TAT, another critical indicator of laboratory efficiency, has been actively used in previous research. Kim et al. [25] measured the mean, 99th percentile, and CV% TAT to assess the laboratory efficiency of total laboratory automation. The mean TAT, representing the timeliness of result reporting, decreased by 6.1%; the 99th percentile TAT, representing the outlier rate, decreased by 13.3%; and the CV% TAT, representing predictability, decreased by 70.0%. Despite the effectiveness of TAT in assessing laboratory efficiency, there is no clear consensus on which period should be incorporated when establishing TAT for a specific test [26,27]. In this study, we did not analyze the pre-analytical phase and DNA extraction step due to performing the same process in SBT and NGS. TATs were assessed after DNA extraction until the final reporting. Similar to risk assessment, TAT increased when the test involved manual procedures, required skilled techniques, or included repetitive and numerous steps. Due to the complex and numerous steps involved in NGS, NGS indicated longer total TAT, total hands-on time, and particularly hands-on time/step.

According to several reports, an increase in hands-on times has been correlated with increasing medical costs. Shin et al. [28] reported labor costs into the total expense, considering the average technician salary and the hands-on time required for testing a single sample. In the case of unexpected antibody screening, the automated analyzer significantly reduced labor costs compared to manual methods (KRW 539 vs. 2961 Korean), subsequently reducing the total expense (KRW 5920 vs. 7157 Korean). In this study, the increased hands-on time in NGS could potentially elevate labor costs, consequently contributing to an overall increase in total expense. However, we did not comprehensively compare costs in this study; further research is needed to compare direct and indirect costs between NGS and SBT.

Recent studies have shown controversial results regarding the impact of laboratory automation in terms of laboratory efficiency. Chung et al. [21] revealed that the automated crossmatching method has a lower RPN score (229 vs. 1435 RPN), shorter TAT (19.1 vs. 23.3 min), shorter hands-on time (1.1 vs. 5.3 min), and lower costs/test (USD 1.44 vs. 2.70) than the manual crossmatching method. Nam et al. [24] found that automated column agglutination technology showed lower RPN scores (33,700 vs. 184,300 RPN) than manual tube tests for anti-A/B isoagglutinin titers. However, TAT and cost were similar in automated and manual methods (TAT, 15:23:00 vs. 14:26:40; cost, 1377.4 vs. 1312.4, respectively). Nam et al. [22] revealed that automated white blood cell counting in leukopenic samples showed an even longer TAT than manual counting. In this study, NGS showed an increased risk and TAT due to the increase in the number and complexity of steps. Despite the controversial results on laboratory efficiency for automation, we expect that the automation of several NGS processes can improve laboratory efficiency. In particular, the DNA clean-up step revealed the highest RPN score, and the DNA quantification step revealed a remarkably longer hands-on time. The automation of these steps can contribute to improving the laboratory efficiency of NGS.

This study has some limitations. It involved a comparison of risk and TAT for a single sample without the evaluation of multiple samples. Given that NGS utilized a flow cell for analyzing multiple samples simultaneously, there is potential for a significant reduction in risk and TAT. In addition, TAT might differ between qualified and unskilled technicians. Therefore, it will be necessary to minimize the possibility of variation by conducting multiple measurements of the performance of a qualified and skilled technician.

5. Conclusions

This is the first study to compare risk and TAT between NGS and SBT for HLA typing. NGS required more processing steps and a more significant manual workload than SBT, resulting in higher RPN scores and longer TATs. These results suggest that it is necessary to consider the high risk and long TAT in NGS when contemplating the transition from SBT to NGS for HLA typing. Furthermore, companies involved in NGS development need to optimize the testing workflow, reduce the manual workload, and increase the automation process.