What Is Wrong with My Model? Identifying Systematic Problems with Semantic Data Slicing

TL;DR

SemSlicer leverages Large Language Models to enable flexible, semantic data slicing for identifying systematic issues in machine learning models, overcoming limitations of traditional feature-based methods.

Contribution

This work introduces SemSlicer, a novel framework that uses LLMs for semantic data slicing without relying on predefined features, enhancing error analysis and model debugging.

Findings

SemSlicer accurately identifies problematic data slices.

It offers low-cost, flexible semantic slicing.

It reliably detects under-performing data segments.

Abstract

Machine learning models make mistakes, yet sometimes it is difficult to identify the systematic problems behind the mistakes. Practitioners engage in various activities, including error analysis, testing, auditing, and red-teaming, to form hypotheses of what can go (or has gone) wrong with their models. To validate these hypotheses, practitioners employ data slicing to identify relevant examples. However, traditional data slicing is limited by available features and programmatic slicing functions. In this work, we propose SemSlicer, a framework that supports semantic data slicing, which identifies a semantically coherent slice, without the need for existing features. SemSlicer uses Large Language Models to annotate datasets and generate slices from any user-defined slicing criteria. We show that SemSlicer generates accurate slices with low cost, allows flexible trade-offs between different design dimensions, reliably identifies under-performing data slices, and helps practitioners identify useful data slices that reflect systematic problems.