Fostering the Ecosystem of AI for Social Impact Requires Expanding and Strengthening Evaluation Standards

TL;DR

This paper argues for expanding evaluation standards in AI for social impact to include broader social contributions and more rigorous impact assessments, aiming to foster a sustainable research ecosystem.

Contribution

It highlights the need to broaden social impact evaluation criteria and emphasizes rigorous impact assessment to improve AI research sustainability in social contexts.

Findings

Current review guidelines favor deployment and innovation, limiting ecosystem diversity.

Broader impact evaluation can encourage diverse research contributions.

Rigorous impact assessments are essential for sustainable social impact AI.

Abstract

There has been increasing research interest in AI/ML for social impact, and correspondingly more publication venues have refined review criteria for practice-driven AI/ML research. However, these review guidelines tend to most concretely recognize projects that simultaneously achieve deployment and novel ML methodological innovation. We argue that this introduces incentives for researchers that undermine the sustainability of a broader research ecosystem of social impact, which benefits from projects that make contributions on single front (applied or methodological) that may better meet project partner needs. Our position is that researchers and reviewers in machine learning for social impact must simultaneously adopt: 1) a more expansive conception of social impacts beyond deployment and 2) more rigorous evaluations of the impact of deployed systems.