Performance of the Self‐Controlled Case Series for Drug Safety Signal Detection: A Multi‐Database Study

TL;DR

This study compares how well a drug safety method works across different health databases and finds that combining databases improves results.

Contribution

The study introduces a multi-database approach to improve the performance of the SCCS for drug safety signal detection.

Findings

SCCS sensitivity ranged from 0.57 to 0.89 across individual databases.

Combining databases increased performance metrics like AUC to 0.76.

Database differences in population and healthcare systems caused performance variation.

Abstract

Differences in performance of the Self‐Controlled Case Series (SCCS) for signal detection have been reported across different databases. However, there has been limited comparative analysis of performance and it remains unknown whether combinations of databases could enable more effective signal detection.

This study aims to compare the performance of the SCCS for signal detection across several data sources, and to determine whether combinations of databases can improve SCCS performance.

We applied the SCCS to macrolides and fluoroquinolone antibiotics, in four databases: Merative MarketScan Commercial Claims and Medicare, the Clinical Practice Research Datalink (CPRD) Aurum and the Système National des Données de Santé. We developed a reference set of 104 positive controls and 58 negative controls, using a taxonomy framework to ensure the selected drug outcome pairs are theoretically well suited to the SCCS design. The observation period lasted 2 years, with a 30‐day risk‐window after each dispensing. Diagnostic performance was measured using sensitivity, specificity and area under the receiver operating curve (AUC) with respect to the product labels, both for individual and combinations of databases.

The sensitivity of the SCCS ranged from 0.57–0.89 across individual databases, and the specificity from 0.43–0.77 when limited to drug‐outcome pairs sufficiently powered. The combination of all databases achieved the maximum sensitivity of 0.89 (0.41 specificity) for the full reference set, and a sensitivity of 1 (0.35 specificity) for drug outcome pairs with enough power. Whilst AUCs ranged from 0.66 to 0.71 across individual databases, the highest performing combination was CPRD plus MarketScan Commercial Claims (0.76 AUC).

Using a carefully designed reference set of drug‐outcome pairs well suited to the study design, the SCCS performance varied substantially by database due to differences in population, reporting, healthcare and coding systems and prescribing patterns. Multi‐database studies showed increased performance of SCCS for signal detection.

The Self‐Controlled Case Series (SCCS) is a promising method for drug safety signal detection using Real‐World Data but has not been implemented routinely because of heterogeneous performance reporting.We explored SCCS performance in individual and combinations of databases from three countries, using a tailored reference set based on macrolide and fluoroquinolone antibiotics drug labels.Sensitivity of and specificity of SCCS ranged from 0.57–0.89 and 0.43–0.77 respectively across individual databases when limited to drug‐outcome pairs sufficiently powered.Whilst AUCs ranged from 0.66 to 0.71 across individual databases, the highest performing combination was CPRD plus MarketScan Commercial Claims (0.76 AUC).Using a carefully designed reference set of drug‐outcome pairs well suited to the study design, SCCS performance varied substantially by database. Performance was increased when using multiple databases.

The Self‐Controlled Case Series (SCCS) is a promising method for drug safety signal detection using Real‐World Data but has not been implemented routinely because of heterogeneous performance reporting.

We explored SCCS performance in individual and combinations of databases from three countries, using a tailored reference set based on macrolide and fluoroquinolone antibiotics drug labels.

Sensitivity of and specificity of SCCS ranged from 0.57–0.89 and 0.43–0.77 respectively across individual databases when limited to drug‐outcome pairs sufficiently powered.

Whilst AUCs ranged from 0.66 to 0.71 across individual databases, the highest performing combination was CPRD plus MarketScan Commercial Claims (0.76 AUC).

Using a carefully designed reference set of drug‐outcome pairs well suited to the study design, SCCS performance varied substantially by database. Performance was increased when using multiple databases.

We explored the performance of a self‐controlled method, where risks of outcomes (potential adverse events) are compared between different time points in each patient, for the detection of Adverse Drug Reactions using health claims databases. We also explored the added value of a recent methodological development, active comparators, which can address some of the remaining bias. We looked at the capacity of the method at identifying known drug side effects for commonly prescribed antibiotics. The method was able to give a correct result in 57% to 89% of positive controls (known adverse drug events) depending on the databases, and 43% to 77% of negative controls (drugs known not to be associated with certain outcomes) when limited to drug‐outcome pairs with a satisfying number of occurrences. Overall, the performance of the method varied substantially by database, due to differences in population, healthcare systems and prescribing patterns in the different countries. Using combinations of databases enabled an increase in the performance of the method.

Figures