Considerations for developing predictive models of crime and new methods for measuring their accuracy

TL;DR

This paper emphasizes the importance of tailored evaluation of crime prediction models, introduces a flexible new accuracy measure (PPAI), and advocates for multi-criteria assessment using utility functions and ALS for better model utility assessment.

Contribution

It introduces the penalized predictive accuracy index (PPAI) and promotes utility-based and logarithmic scoring methods for comprehensive model evaluation.

Findings

PPAI offers flexible model assessment tailored to specific problems.

Expected utility functions enable multi-criteria model comparison.

ALS provides a different perspective on model accuracy for crime prediction.

Abstract

Developing spatio-temporal crime prediction models, and to a lesser extent, developing measures of accuracy and operational efficiency for them, has been an active area of research for almost two decades. Despite calls for rigorous and independent evaluations of model performance, such studies have been few and far between. In this paper, we argue that studies should focus not on finding the one predictive model or the one measure that is the most appropriate at all times, but instead on careful consideration of several factors that affect the choice of the model and the choice of the measure, to find the best measure and the best model for the problem at hand. We argue that because each problem is unique, it is important to develop measures that empower the practitioner with the ability to input the choices and preferences that are most appropriate for the problem at hand. We develop a new measure called the penalized predictive accuracy index (PPAI) which imparts such flexibility. We also propose the use of the expected utility function to combine multiple measures in a way that is appropriate for a given problem in order to assess the models against multiple criteria. We further propose the use of the average logarithmic score (ALS) measure that is appropriate for many crime models and measures accuracy differently than existing measures. These measures can be used alongside existing measures to provide a more comprehensive means of assessing the accuracy and potential utility of spatio-temporal crime prediction models.