Controllable Missingness from Uncontrollable Missingness: Joint Learning Measurement Policy and Imputation

TL;DR

This paper introduces a joint learning framework for measurement policy and data imputation, enabling effective data retrieval under controllable missingness, with demonstrated superiority over baseline methods.

Contribution

It proposes a novel data generation and joint learning algorithm that simultaneously optimizes measurement policy and imputation, addressing the challenge of learning from incomplete data.

Findings

The method outperforms baseline approaches across datasets.

It is applicable to various missing data rates.

The approach effectively learns measurement policies that improve data imputation.

Abstract

Due to the cost or interference of measurement, we need to control measurement system. Assuming that each variable can be measured sequentially, there exists optimal policy choosing next measurement for the former observations. Though optimal measurement policy is actually dependent on the goal of measurement, we mainly focus on retrieving complete data, so called as imputation. Also, we adapt the imputation method to missingness varying with measurement policy. However, learning measurement policy and imputation requires complete data which is impossible to be observed, unfortunately. To tackle this problem, we propose a data generation method and joint learning algorithm. The main idea is that 1) the data generation method is inherited by imputation method, and 2) the adaptation of imputation encourages measurement policy to learn more than individual learning. We implemented some variations of proposed algorithm for two different datasets and various missing rates. From the experimental results, we demonstrate that our algorithm is generally applicable and outperforms baseline methods.