Mathematical construction of a low-bias high-resolution deprivation index for the United States

TL;DR

This paper introduces a novel, bias-resistant deprivation index for the US, constructed via diffusion maps on census data, offering a high-resolution, unbiased tool for social and policy analysis.

Contribution

The paper presents a new method for creating deprivation indices using diffusion maps, avoiding normative biases and partisan manipulation, based on complete census data.

Findings

The index aligns with income-based deprivation measures.

It reduces bias compared to existing indices.

The method is efficient and highly resilient to manipulation.

Abstract

The construction of deprivation indices is complicated by the inherent ambiguity in defining deprivation as well as the potential for partisan manipulation. Nevertheless, deprivation indices provide an essential tool for mitigating the effects of deprivation and reducing it through policy interventions. Here we demonstrate the construction of a deprivation index using diffusion maps, a manifold learning technique capable of finding the variables that optimally describe the variations in a dataset in the sense of preserving pairwise relationships among the data points. The method is applied to the 2010 US decennial census. In contrast to other methods the proposed procedure does not select particular columns from the census, but rather constructs an indicator of deprivation from the complete dataset. Due to its construction the proposed index does not introduce biases except those already present in the source data, does not require normative judgment regarding the desirability of certain life styles, and is highly resilient against attempts of partisan manipulation. We demonstrate that the new index aligns well with established income-based deprivation indices but deviates in aspects that are perceived as problematic in some of the existing indices. The proposed procedure provides an efficient way for constructing accurate, high resolution indices. These indices can thus have the potential to become powerful tools for the academic study of social structure as well as political decision making.