ORES-Inspect: A technology probe for machine learning audits on enwiki
Zachary Levonian, Lauren Hagen, Lu Li, Jada Lilleboe, Solvejg, Wastvedt, Aaron Halfaker, Loren Terveen

TL;DR
ORES-Inspect is an open-source web tool designed to help Wikipedia editors audit and understand the performance of ML models like ORES, facilitating research into how users evaluate AI systems in collaborative environments.
Contribution
The paper introduces ORES-Inspect, a novel interface and research tool for auditing Wikipedia's ML models, addressing challenges in transparency and stakeholder engagement.
Findings
Enables editors to assess ML model performance effectively.
Facilitates research on user perceptions of AI auditing.
Supports transparency in vandalism detection models.
Abstract
Auditing the machine learning (ML) models used on Wikipedia is important for ensuring that vandalism-detection processes remain fair and effective. However, conducting audits is challenging because stakeholders have diverse priorities and assembling evidence for a model's [in]efficacy is technically complex. We designed an interface to enable editors to learn about and audit the performance of the ORES edit quality model. ORES-Inspect is an open-source web tool and a provocative technology probe for researching how editors think about auditing the many ML models used on Wikipedia. We describe the design of ORES-Inspect and our plans for further research with this system.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Advanced Text Analysis Techniques · Explainable Artificial Intelligence (XAI)
