Diagnostic performance of a point-of-care molecular system for the detection of SARS-CoV-2 in Peru

TL;DR

This study evaluated a rapid test for SARS-CoV-2 in Peru and found it highly accurate compared to a standard PCR test.

Contribution

The study provides empirical validation of the Xpert®Xpress SARS-CoV-2 test's performance in a Peruvian population.

Findings

The test showed 98.70% overall concordance with the reference PCR test.

It had 100% sensitivity and 96.15% relative specificity for SARS-CoV-2 detection.

The test also demonstrated 100% analytical specificity against other respiratory viruses.

Abstract

The present study assessed the diagnostic performance of the Xpert®Xpress SARS-CoV-2 test in comparison with the Charité protocol real-time RT PCR for the detection of SARS-CoV-2 in Peruvian patients. This was a diagnostic test study that included 100 nasal and pharyngeal swab samples. We obtained an overall concordance of 98.70% (95%CI: 92.98-99.97), with a kappa coefficient of 0.97 (95%CI: 0.86-1.00) and sensitivity and relative specificity rates of 100% and 96.15%, respectively. Additionally, the percentage of the area under the ROC curve was 98.08% in both cases, and an analytical specificity rate of 100% was obtained for the different respiratory viruses evaluated. In conclusion, the Xpert®Xpress SARS-CoV-2 test, by using nasal and pharyngeal swab samples, was highly sensitive and specific, and the kappa coefficient showed an excellent correlation when compared to the reference test.

En el presente estudio se estimó el rendimiento diagnóstico de la prueba Xpert®Xpress SARS-CoV-2 en comparación con la RT PCR en tiempo real-protocolo Charité, para la detección de SARS-CoV-2 en pacientes peruanos. Se trató de un diseño de prueba diagnóstica que incluyó 100 muestras de hisopado nasal y faríngeo. Se obtuvo una concordancia global de 98,70% (IC95%: 92,98-99,97), con un coeficiente kappa de 0,97 (IC95%: 0,86-1.00); se estimó una sensibilidad y especificad relativa de 100% y 96,15%, respectivamente. Adicionalmente, el porcentaje del área bajo la curva ROC fue 98,08% en ambos casos y se obtuvo una especificidad analítica del 100% para los diferentes virus respiratorios evaluados. En conclusión, la prueba Xpert®Xpress SARS-CoV-2 a partir de muestras de hisopado nasal y faríngeo fue altamente sensible y específica, así mismo el coeficiente kappa mostró una excelente correlación, al compararla con la prueba de referencia.

Full text

INTRODUCTION

In 2019, the first cases of coronavirus disease (COVID-19) were reported in Wuhan province, China. Subsequently, SARS-CoV-2 virus was identified as the causative agent 1, and at the end of February 2020 it was declared a pandemic by the World Health Organization (WHO) 2. To control this pandemic, one of the key points was the development and implementation of technologies for the detection of the virus causing the disease, which led to the development of molecular tests that aided in the rapid and timely detection of SARS-CoV-2 3. Different platforms were designed with the aim of finding an alternative that was fast, easy to implement, and efficient. However, most of them require complex infrastructure and highly trained personnel, which in many cases limits the decentralization of molecular testing in Peru.

The GeneXpert platform is a fully automated closed system based on real-time PCR, which integrates in a single system and automates all the procedures of sample preparation, nucleic acid extraction and amplification, as well as target sequence detection 4^,^5. This system requires the use of disposable cartridges that include the necessary controls to validate the sample run, as well as the detection of target genes. In Peru, by the end of 2020, 38 laboratories had been implemented with this platform by the National Institute of Health (INS) for use in the detection of tuberculosis and/or HIV viral load 6^,^7. For this reason, one of the intervention strategies was the implementation of the Xpert Xpress SARS-CoV-2 cartridge 4, which was developed for the molecular detection of the virus causing COVID-19.

The Xpert Xpress SARS-CoV-2 cartridge specifically detects two very important target genes, namely “N2” (protein nucleocapsid) and “E” (protein envelope), increasing the sensitivity of the test for the detection of SARS-CoV-2 8^,^9. The performance obtained for this test in terms of concordance was 100% in agreement with the assay performed using a quantified reference material of virus particles. Likewise, the analytical limit of detection (LoD) was established, which, according to the manufacturer, was 0.02 PFU/ml 4.

In view of the above, this study aimed to evaluate the diagnostic performance of the real-time automated nucleic acid amplification test (Xpert®Xpress SARS-CoV-2) in comparison with real-time RT PCR - Charité protocol 3, for the qualitative molecular detection of SARS-CoV-2 (COVID-19), from nasal and pharyngeal swab samples contained in universal viral transport medium from Peruvian patients.

KEY MESSAGESMotivation for the study: To describe and evaluate a closed molecular platform, easy to use and of importance in Peru for the management of diseases of public health priority, now implemented for the detection of SARS-CoV-2.Main findings: Highly sensitive and specific molecular test, with excellent correlation compared to the reference test for detecting SARS-CoV-2.Implications: Can be used in point-of-care laboratories for rapid molecular detection of different infectious agents, including SARS-CoV-2. Little expertise and minimal infrastructure are required to implement it.

THE STUDY

Design and samples

This was an observational, cross-sectional study designed as a diagnostic test evaluation, which was carried out using nasal and pharyngeal swab samples contained in universal transport media collected in 2020 from Peruvian patients and stored in the INS sample bank. These previously tested positive or negative by real-time RT-PCR for SARS-CoV-2.

The Epidat 4.2 program was used to calculate the sample size. Considering a confidence level of 95%, that the RT-PCR classified 50% of samples as positive (50 positive and 50 negative) 10, and assuming that the automated amplification test classified a similar number as positive; then a sample size of 100 samples allows finding a Cohen’s kappa coefficient of 0.90 with a precision of +/- 0.085; a lower precision would have meant requiring less sample. The expected kappa coefficient corresponded to a very good correlation, close to the 0.98 reported by Zheng et al. 11. Finally, the selected samples were previously identified as positive (n=52) and negative (n=50) for SARS-CoV-2 by real-time RT-PCR. Among these 50 negative samples, 25 were positive for other viruses in order to evaluate cross-reactivity.

RT-PCR platforms and instruments

Real-time RT-PCR

RNA extraction was performed according to the manufacturer’s specifications with the QIAamp Viral RNA Mini kit either manually or using the QIACUBE® automated kit. For SARS-CoV-2 detection, the extract was subjected to real-time RT PCR-Charité protocol 3 and real-time RT-PCR using primers and probes specific for SARS-CoV-2 RdRP and human GAPDH genes 12. Multiplex real-time RT-PCR in TaqMan system (qRT-PCR_Multiplex) was used for the detection of influenza A and influenza B viruses ^(^13. Finally, for the detection of respiratory syncytial virus, human rhinovirus and metapneumovirus, we used a multiplex RT-PCR for other respiratory viruses designed at the INS of Peru.

Automated Real-Time Nucleic Acid Amplification Assay

Xpert® Xpress SARS-CoV-2 was performed according to the manufacturer’s instructions 4. Briefly, 300 µL of sample was added to the cartridge, which was closed and then placed in a GeneXpert IV kit to perform the test. The cartridges include two internal controls (SPC -sample processing control- and PCC -probe check control-) that ensure the correct functioning of the test 5. SARS-CoV-2 detection is based on the identification of two genes: “N2” (protein nucleocapsid) and “E” (protein envelope), which are used for the interpretation of results. The results are automatically generated by the GeneXpertTM DX Software version 6.2. Results are reported qualitatively, and the analyzed samples are considered positive when both N2 and E genes are detected or when only N2 is detected. When only the E gene was detected, the manufacturer’s recommended interpretation was a presumptive positive result; however, it was considered positive during the pandemic. This is because, at the beginning of the pandemic, there were molecular detection tests designed for betacoronaviruses, and SARS-CoV-2 was the most prevalent virus of this genus during the pandemic, so at that time the implementation of confirmation by detection of a single genetic marker was recommended, taking into account that the curves, as well as other quality assurance parameters, were optimal. PCR of the E gene showed better sensitivity, so it was recommended to prioritize the E gene as the selected marker 14. However, later tests based on protein S began to be performed, which were more specific, and that is why, this type of markers were additionally included for detection for the later tests.

Statistical analysis

Stata v15.1 (Stata Corporation, College Station, Texas, USA) was used for data analysis. Qualitative variables were reported as absolute and relative frequencies. Numerical variables were described with median and interquartile range (IQR), because the small sample size does not guarantee a normal distribution; and with range of minimum and maximum values. Concordance, relative diagnostic performance, Cohen’s kappa coefficient, with 95% confidence intervals were analyzed. Likewise, relative sensitivity (percentage of positive concordance) and relative specificity (percentage of negative concordance) were obtained using real-time RT-PCR as an imperfect reference standard, since the ideal reference standard is virus isolation in cell culture from different patient samples; however, due to the complexity of the procedure this is not usually used, so real-time RT-PCR was used as a surrogate. The area under the ROC (Receiver operating characteristic) curve was also calculated as a percentage with its respective confidence interval.

On the other hand, analytical specificity was obtained using only RT-PCR negative tests for SARS-CoV-2 but positive for other respiratory viruses. All evaluations were performed at a significance level of 0.05. The different obtained parameters were compared by evaluating their cross-confidence intervals. If their 95% confidence intervals did not cross, they were considered statistically different.

Ethical aspects

This study was approved by the INS ethics committee, with document No. RD: 176-2020-OGITT/INS.

RESULTS

We included 102 samples, 77 from COVID-19 suspect patients, collected between April and September 2020; and 25 from pre-pandemic samples stored at -80 °C in the INS sample bank. The former were used to calculate the relative performance of automated amplification, and the latter to evaluate the analytical specificity of the test. Of the first group, 52 were positive and 25 were negative for SARS-CoV-2 RT-PCR.

Of the samples, 55.8% (43/77) came from males. Women had a median age of 40.5 years (IQR: 34-49, minimum-maximum: 0-60), and men, 47 years (IQR: 38-56, minimum-maximum: 3-85). Out of the samples from pre-pandemic patients, 44.0% were women (11/25). There was only one discrepancy when comparing both tests in patients with clinical suspicion of COVID-19, with one sample being negative by RT-PCR and positive by automated amplification. This yielded an overall concordance of 98.70% (95%CI: 92.98-99.97), with a kappa coefficient of 0.97 (95%CI: 0.86-1.00), corresponding to an excellent correlation. The numbers were very similar when two “presumptive positives” in the automated amplification test, initially considered as positive, were excluded (Table 1 and 2).

Table 1. Comparison of results obtained by automated amplification for the detection of SARS-CoV-2, with the real-time RT-PCR test.Positivity criterionAutomated amplification resultResults of Real-time RT-PCR PositiveNegativeTotalSuspected positives are considered positivePositive51152Negative02525Total512677Suspected positives are excluded from the testPositive49150Negative02525Total492675

Table 2. Diagnostic performance of automated amplification for the detection of SARS-CoV-2 compared to real-time RT-PCR. SARS-COV-2 detection N ^a^Result (%)95%CIConsidering presumptive positive results Overall concordance76/7798.7092.98-99.97 Cohen’s kappa coefficient 0.970.86-1.00 Area under ROC curve 98.0894.31-100.00 Relative sensitivity51/51100.0093.02-100.00 Relative specificity 25/2696.1580.36-99.90Excluding presumptive positive results Overall concordance74/7598.6792.79-99.97 Cohen’s kappa coefficient 0.970.85-1.00 Area under ROC curve 98.0894.31-100.00 Relative sensitivity49/49100.0092.75-100.00 Relative specificity25/2696.1580.36-99.90a Data evaluated for the specific analysis; 95%CI: 95% confidence interval.ROC: Receiver operating characteristic

The relative performance evaluation with respect to real-time RT-PCR for SARS-CoV-2 showed a relative sensitivity of 100%, both when all samples were included and when the two “presumptive positives” in the amplification test were excluded. The relative specificity was 96.15% and the % area under the ROC curve was 98.08% in both cases (Table 1 and 2).

In addition, Table 3 presents the medians with their IQRs for the RdRP, ORF1a, E, and N genes from the real-time RT-PCR, and for the N2, and E genes from the automated amplification platform used. Overall, we found a wide range of values for all the evaluated genes.

Table 3. Median Ct values of positive tests to the different genes used by real-time RT-PCR and automated amplification for SARS-CoV-2 detection.PlatformGenNMedianIQRMin-maxReal-time RT-PCRRDRP3222.8317.71-26.2310.52-30.55ORF1821.0516.82-25.4013.25-31.81GEN_E2628.0523.97-31.9712.51-38.28GEN_N2521.6218.44-27.8014.67-34.89Automated amplificationN2_XPERT5027.4521.40-30.7012.00-39.80E_XPERT4926.1018.50-28.5011.00-44.60IQR: interquartile range; Min-max: minimum value and maximum value

Finally, the analytical specificity was 100% for all included respiratory viruses, including respiratory syncytial virus (5/5), human rhinovirus (6/6), metapneumovirus (5/5), influenza A virus (5/5), and influenza B virus (5/5) (Table 4).

Table 4. Analytical specificity of automated amplification for SARS-CoV-2 detection, relative to real-time RT-PCR, for different respiratory viruses. SARS-CoV-2 Analytical specificity Real-time RT-PCR Automated amplification PositiveNegativePositiveNegativeRespiratory Syncytial Virus (RSV)0505100.0%Human Rhinovirus (HRV)0606100.0%Metapneumovirus (MPVH)0505100.0%Influenza A virus (FLU-A)0505100.0%Virus influenza B (FLU-B)0505100,0%

DISCUSSION

The automated real-time nucleic acid amplification test (Xpert® Xpress SARS-CoV-2) is easy to handle and use. It has demonstrated high diagnostic performance and concordance with the real-time RT-PCR test for the qualitative molecular detection of the virus causing COVID-19 from nasal and pharyngeal swab samples. In addition, it has zero probability of cross-contamination between samples, since all the processes are included in a single cartridge.

Xpert® Xpress SARS-CoV-2 requires minimal infrastructure and biosafety conditions, since it is a fully automated procedure in which the laboratory technician is involved only when loading the sample 4. In addition, the short time to obtain results can become an important and vital advantage for patient management. Likewise, this test is highly sensitive, which is supported by its established detection limit of 0.01 plaque forming units (PFU)/ml 15.

Different studies have evaluated the performance of this test, finding highly concordant results 16^-^18. In addition, we did not find cross-reaction, which is in agreement with the findings of Wolters et al. 17.

On the other hand, we found a discordant result when both tests were evaluated. While the reference method was negative, the evaluated method was positive for the same sample. This could be due to a higher sensitivity of the evaluated method, which could be detecting a lower amount of RNA than the reference method, which is directly related to the detection limit of the test. This fact has already been evidenced in several publications 11^,^19. In these cases, the result was first reported as indeterminate, then a third test was performed to determine the patient’s final result. As far as possible, this third test should be different from those previously used.

When analyzing the Cycle threshold (CT), the low viral loads found in this study may be due to different factors, such as very early or very late sampling considering the course of the disease, problems during sampling, transport and preservation of the samples, or low levels of viral spread in general. On the other hand, high viral loads may have been obtained in settings where viral spread in the community was active and high. The understanding of viral load levels and the effect this may have on the patient has been very controversial, and different studies in this regard have shown contradictory results 20^,^21.

Additionally, this test, by having two detection genes, the E gene and the N2 gene, provided significant sensitivity compared to the reference method 4. The E gene allows the detection of different SARS (bat coronaviruses), including SARS-CoV-2, so the probable effects of genetic drift can be avoided. Moreover, our results show a wide variation in the viral load obtained (TC values obtained) among the samples that were selected for this study.

Therefore, the evaluated platform offers highly efficient results for the detection of SARS-CoV-2. Its strength lies in the fact that it can be used in a decentralized manner and with fewer resources (human resources, infrastructure, equipment, among others) compared to a traditional real-time RT-PCR. In addition, it works as a multiplatform for the detection of other agents of public health importance from different types of samples. Considering that more than 50 laboratories with this platform have been implemented in Peru by the end of 2023, it could be a potentially useful strategy to establish a significant overall impact in the country, prioritizing those establishments where rapid results are required, such as, for example, hospitals with intensive care in high prevalence environments for the main diseases affecting public health in the country.

The main limitation of the study is that the ideal reference standard was not used, which is the isolation of the virus in cell culture from different samples of a patient; instead, we used the closest standard, such as Rt-PCR, which has been used in multiple studies 16^-^19. Another limitation is that we did not have data regarding the patient’s disease, such as symptoms, disease duration, or severity. Even so, we consider that these factors affect the evaluated platform and the reference standard equally, so the results are still reliable. On the other hand, it should also be considered that the costs that this platform requires in terms of supplies, equipment and maintenance are still high. Therefore, its prioritization could be a good strategy to address different diseases, accompanied by other technologies that could massively address different diseases.

Finally, we conclude that the diagnostic performance of the Xpert® Xpress SARS-CoV-2 test, in comparison with the reference real-time RT-PCR method for the detection of COVID-19 virus from nasal and pharyngeal swab samples, was highly efficient in terms of sensitivity and specificity. The results obtained for sensitivity and specificity were 100% and 96%, respectively, reaching an excellent correlation according to the results obtained by the kappa coefficient. Likewise, as it is a highly efficient test and easy to implement and use, its decentralized and prioritized use at the points of patient care is recommended.