Interestingness First Classifiers

TL;DR

This paper introduces EUREKA, a framework for building classifiers that prioritize interesting, non-obvious features using large language models, aiming to enhance interpretability and novelty in predictive models.

Contribution

The paper presents a novel method that leverages language models to select interesting features, leading to interpretable classifiers that reveal unexpected insights.

Findings

EUREKA identifies non-obvious yet predictive features across datasets.

Classifiers built with EUREKA offer meaningful insights with moderate accuracy.

The approach supports knowledge discovery and interpretability in machine learning.

Abstract

Most machine learning models are designed to maximize predictive accuracy. In this work, we explore a different goal: building classifiers that are interesting. An ``interesting classifier'' is one that uses unusual or unexpected features, even if its accuracy is lower than the best possible model. For example, predicting room congestion from CO2 levels achieves near-perfect accuracy but is unsurprising. In contrast, predicting room congestion from humidity is less accurate yet more nuanced and intriguing. We introduce EUREKA, a simple framework that selects features according to their perceived interestingness. Our method leverages large language models to rank features by their interestingness and then builds interpretable classifiers using only the selected interesting features. Across several benchmark datasets, EUREKA consistently identifies features that are non-obvious yet still predictive. For example, in the Occupancy Detection dataset, our method favors humidity over CO2 levels and light intensity, producing classifiers that achieve meaningful accuracy while offering insights. In the Twin Papers dataset, our method discovers the rule that papers with a colon in the title are more likely to be cited in the future. We argue that such models can support new ways of knowledge discovery and communication, especially in settings where moderate accuracy is sufficient but novelty and interpretability are valued.