Improve ROI with Causal Learning and Conformal Prediction

TL;DR

This paper introduces a robust method called rDRP that enhances ROI prediction accuracy in operational decision-making by addressing covariate shift and data scarcity using conformal prediction and Monte Carlo dropout, validated through offline and online tests.

Contribution

The paper proposes a novel rDRP method that improves ROI predictions without retraining, effectively handling covariate shift and data limitations in neural network uplift models.

Findings

Significant ROI improvements in offline tests.

Enhanced model robustness under covariate shift.

Successful online A/B test results.

Abstract

In the commercial sphere, such as operations and maintenance, advertising, and marketing recommendations, intelligent decision-making utilizing data mining and neural network technologies is crucial, especially in resource allocation to optimize ROI. This study delves into the Cost-aware Binary Treatment Assignment Problem (C-BTAP) across different industries, with a focus on the state-of-the-art Direct ROI Prediction (DRP) method. However, the DRP model confronts issues like covariate shift and insufficient training data, hindering its real-world effectiveness. Addressing these challenges is essential for ensuring dependable and robust predictions in varied operational contexts. This paper presents a robust Direct ROI Prediction (rDRP) method, designed to address challenges in real-world deployment of neural network-based uplift models, particularly under conditions of covariate shift and insufficient training data. The rDRP method, enhancing the standard DRP model, does not alter the model's structure or require retraining. It utilizes conformal prediction and Monte Carlo dropout for interval estimation, adapting to model uncertainty and data distribution shifts. A heuristic calibration method, inspired by a Kaggle competition, combines point and interval estimates. The effectiveness of these approaches is validated through offline tests and online A/B tests in various settings, demonstrating significant improvements in target rewards compared to the state-of-the-art method.