Challenge AI Mind: A Crowd System for Proactive AI Testing
Siwei Fu, Anbang Xu, Xiaotong Liu, Huimin Zhou, Rama Akkiraju

TL;DR
This paper introduces Challenge.AI, a crowd-powered system that combines crowdsourcing and machine learning to proactively generate and evaluate testing data, improving AI testing coverage and error detection.
Contribution
The paper presents a novel crowd system integrating machine learning for proactive AI testing, enabling dynamic data generation and comprehensive error analysis.
Findings
Crowd workflow is more effective with machine learning assistance.
AI developers can discover previously unknown errors.
The system enhances proactive testing engagement.
Abstract
Artificial Intelligence (AI) has burrowed into our lives in various aspects; however, without appropriate testing, deployed AI systems are often being criticized to fail in critical and embarrassing cases. Existing testing approaches mainly depend on fixed and pre-defined datasets, providing a limited testing coverage. In this paper, we propose the concept of proactive testing to dynamically generate testing data and evaluate the performance of AI systems. We further introduce Challenge.AI, a new crowd system that features the integration of crowdsourcing and machine learning techniques in the process of error generation, error validation, error categorization, and error analysis. We present experiences and insights into a participatory design with AI developers. The evaluation shows that the crowd workflow is more effective with the help of machine learning techniques. AI developers…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Mobile Crowdsensing and Crowdsourcing · Anomaly Detection Techniques and Applications
