Limitations of Membership Queries in Testable Learning

TL;DR

This paper demonstrates that in the testable learning model, membership queries do not significantly reduce learning complexity compared to sample-only methods, establishing fundamental limitations and lower bounds for such algorithms.

Contribution

The paper provides a reduction from refutation to testable learning with queries, showing that membership queries do not drastically improve efficiency, and introduces a class of statistical MQ algorithms with inherent limitations.

Findings

Membership queries cannot substantially reduce testable learning complexity.

Lower bounds for testable learning with queries are derived via reduction from refutation.

Efficient statistical MQ learners are unlikely to be made testable due to inherent limitations.

Abstract

Membership queries (MQ) often yield speedups for learning tasks, particularly in the distribution-specific setting. We show that in the \emph{testable learning} model of Rubinfeld and Vasilyan [RV23], membership queries cannot decrease the time complexity of testable learning algorithms beyond the complexity of sample-only distribution-specific learning. In the testable learning model, the learner must output a hypothesis whenever the data distribution satisfies a desired property, and if it outputs a hypothesis, the hypothesis must be near-optimal. We give a general reduction from sample-based \emph{refutation} of boolean concept classes, as presented in [Vadhan17, KL18], to testable learning with queries (TL-Q). This yields lower bounds for TL-Q via the reduction from learning to refutation given in [KL18]. The result is that, relative to a concept class and a distribution family, no $m$-sample TL-Q algorithm can be super-polynomially more time-efficient than the best $m$-sample PAC learner. Finally, we define a class of ``statistical'' MQ algorithms that encompasses many known distribution-specific MQ learners, such as those based on influence estimation or subcube-conditional statistical queries. We show that TL-Q algorithms in this class imply efficient statistical-query refutation and learning algorithms. Thus, combined with known SQ dimension lower bounds, our results imply that these efficient membership query learners cannot be made testable.