FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines

TL;DR

FeedbackLogs is a systematic approach to recording and utilizing stakeholder feedback in machine learning pipelines, enhancing transparency, accountability, and the ability to audit and update models based on stakeholder input.

Contribution

The paper introduces FeedbackLogs, a formalized method for documenting stakeholder feedback and its integration into ML pipelines, which was previously lacking in standard practices.

Findings

FeedbackLogs effectively captures stakeholder feedback details.

FeedbackLogs supports algorithmic auditing and transparency.

The method facilitates systematic updates to ML models based on feedback.

Abstract

Even though machine learning (ML) pipelines affect an increasing array of stakeholders, there is little work on how input from stakeholders is recorded and incorporated. We propose FeedbackLogs, addenda to existing documentation of ML pipelines, to track the input of multiple stakeholders. Each log records important details about the feedback collection process, the feedback itself, and how the feedback is used to update the ML pipeline. In this paper, we introduce and formalise a process for collecting a FeedbackLog. We also provide concrete use cases where FeedbackLogs can be employed as evidence for algorithmic auditing and as a tool to record updates based on stakeholder feedback.