Virtual Control Group: Measuring Hidden Performance Metrics

TL;DR

This paper introduces a statistical approach using survey theory and causal inference to estimate the false positive rate in financial integrity systems, addressing the challenge of unseen bad users and outperforming existing methods in some cases.

Contribution

It presents a novel methodology for measuring hidden performance metrics like false positive rate using survey and causal inference techniques.

Findings

The proposed method effectively estimates false positive rates in financial systems.

Outcome matching can outperform traditional methods in certain scenarios.

Approaches are applicable to other domains like Cyber Security.

Abstract

Performance metrics measuring in Financial Integrity systems are crucial for maintaining an efficient and cost effective operation. An important performance metric is False Positive Rate. This metric cannot be directly monitored since we don't know for sure if a user is bad once blocked. We present a statistical method based on survey theory and causal inference methods to estimate the false positive rate of the system or a single blocking policy. We also suggest a new approach of outcome matching that in some cases including empirical data outperformed other commonly used methods. The approaches described in this paper can be applied in other Integrity domains such as Cyber Security.