Prediction-powered estimators for finite population statistics in highly imbalanced textual data: Public hate crime estimation

TL;DR

This paper introduces a method combining neural network predictions with survey estimators to efficiently estimate hate crime statistics from textual police reports, reducing manual annotation effort.

Contribution

It proposes a novel approach that integrates transformer-based predictions with traditional survey sampling estimators for finite population statistics in text data.

Findings

Efficient hate crime estimates with less manual labeling.

Effective application of Hansen-Hurwitz estimator in text data.

Reduced annotation time while maintaining accuracy.

Abstract

Estimating population parameters in finite populations of text documents can be challenging when obtaining the labels for the target variable requires manual annotation. To address this problem, we combine predictions from a transformer encoder neural network with well-established survey sampling estimators using the model predictions as an auxiliary variable. The applicability is demonstrated in Swedish hate crime statistics based on Swedish police reports. Estimates of the yearly number of hate crimes and the police's under-reporting are derived using the Hansen-Hurwitz estimator, difference estimation, and stratified random sampling estimation. We conclude that if labeled training data is available, the proposed method can provide very efficient estimates with reduced time spent on manual annotation.