Wild Guesses and Mild Guesses in Active Concept Learning

TL;DR

This paper compares active learning strategies in a neuro-symbolic Bayesian framework, showing that simple, human-like query strategies can outperform information-maximizing ones in certain concept learning tasks.

Contribution

It introduces a comparison between Bayesian expected information gain and a positive test strategy in active concept learning with LLM-generated hypotheses, revealing when each is effective.

Findings

EIG performs well with complex, falsification-needed concepts

PTS maintains validity and converges faster on simple rules

Confirmation bias may be an adaptive feature in human inference

Abstract

Human concept learning is typically active: learners choose which instances to query or test in order to reduce uncertainty about an underlying rule or category. Active concept learning must balance informativeness of queries against the stability of the learner that generates and scores hypotheses. We study this trade-off in a neuro-symbolic Bayesian learner whose hypotheses are executable programs proposed by a large language model (LLM) and reweighted by Bayesian updating. We compare a Rational Active Learner that selects queries to maximize approximate expected information gain (EIG) and the human-like Positive Test Strategy (PTS) that queries instances predicted to be positive under the current best hypothesis. Across concept-learning tasks in the classic Number Game, EIG is effective when falsification is necessary (e.g., compound or exception-laden rules), but underperforms on simple concepts. We trace this failure to a support mismatch between the EIG policy and the LLM proposal distribution: highly diagnostic boundary queries drive the posterior toward regions where the generator produces invalid or overly specific programs, yielding a support-mismatch trap in the particle approximation. PTS is information-suboptimal but tends to maintain proposal validity by selecting "safe" queries, leading to faster convergence on simple rules. Our results suggest that "confirmation bias" may not be a cognitive error, but rather a rational adaptation for maintaining tractable inference in the sparse, open-ended hypothesis spaces characteristic of human thought.