Threshy: Supporting Safe Usage of Intelligent Web Services
Alex Cummaudo, Scott Barnett, Rajesh Vasa, John Grundy

TL;DR
Threshy is a workflow and tool that assists developers in selecting appropriate decision thresholds for intelligent web services, considering various workflows and financial impacts, to improve safe and effective deployment.
Contribution
It introduces Threshy, a novel tool supporting threshold tuning across multiple development stages, addressing a gap in existing evaluation methods for intelligent web services.
Findings
Supports threshold tuning in pre-development, pre-release, and support workflows
Considers financial impacts of false positives in threshold selection
Exports configuration files for integration into applications
Abstract
Increased popularity of `intelligent' web services provides end-users with machine-learnt functionality at little effort to developers. However, these services require a decision threshold to be set which is dependent on problem-specific data. Developers lack a systematic approach for evaluating intelligent services and existing evaluation tools are predominantly targeted at data scientists for pre-development evaluation. This paper presents a workflow and supporting tool, Threshy, to help software developers select a decision threshold suited to their problem domain. Unlike existing tools, Threshy is designed to operate in multiple workflows including pre-development, pre-release, and support. Threshy is designed for tuning the confidence scores returned by intelligent web services and does not deal with hyper-parameter optimisation used in ML models. Additionally, it considers the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
