Responsible Scoring Mechanisms Through Function Sampling

TL;DR

This paper introduces unbiased function sampling techniques for designing responsible scoring mechanisms, focusing on linear functions, and demonstrates their application in ranking scenarios to promote fairness and accountability.

Contribution

It provides unbiased samplers for the entire function space and a novel hyperplane arrangement approximation method, enabling scalable responsible scoring design.

Findings

Unbiased samplers for linear scoring functions and their vicinity.

A linear-in-samples algorithm for hyperplane arrangement approximation.

Application of sampling methods to improve responsible ranking algorithms.

Abstract

Human decision-makers often receive assistance from data-driven algorithmic systems that provide a score for evaluating objects, including individuals. The scores are generated by a function (mechanism) that takes a set of features as input and generates a score.The scoring functions are either machine-learned or human-designed and can be used for different decision purposes such as ranking or classification. Given the potential impact of these scoring mechanisms on individuals' lives and on society, it is important to make sure these scores are computed responsibly. Hence we need tools for responsible scoring mechanism design. In this paper, focusing on linear scoring functions, we highlight the importance of unbiased function sampling and perturbation in the function space for devising such tools. We provide unbiased samplers for the entire function space, as well as a $\theta$-vicinity around a given function. We then illustrate the value of these samplers for designing effective algorithms in three diverse problem scenarios in the context of ranking. Finally, as a fundamental method for designing responsible scoring mechanisms, we propose a novel approach for approximating the construction of the arrangement of hyperplanes. Despite the exponential complexity of an arrangement in the number of dimensions, using function sampling, our algorithm is linear in the number of samples and hyperplanes, and independent of the number of dimensions.