Continual Learning in Open-vocabulary Classification with Complementary Memory Systems

TL;DR

This paper presents a novel continual learning approach for open-vocabulary image classification that combines zero-shot and exemplar-based models, introducing a fast 'tree probe' method inspired by human cognition to improve learning efficiency and flexibility.

Contribution

It introduces a combined prediction framework using CLIP zero-shot and exemplar models, along with a new 'tree probe' method for rapid learning from new data.

Findings

Achieves a balance of learning speed and accuracy across multiple incremental settings.

Demonstrates competitive performance with batch-trained linear models.

Enables flexible inference on various subsets of categories.

Abstract

We introduce a method for flexible and efficient continual learning in open-vocabulary image classification, drawing inspiration from the complementary learning systems observed in human cognition. Specifically, we propose to combine predictions from a CLIP zero-shot model and the exemplar-based model, using the zero-shot estimated probability that a sample's class is within the exemplar classes. We also propose a "tree probe" method, an adaption of lazy learning principles, which enables fast learning from new examples with competitive accuracy to batch-trained linear models. We test in data incremental, class incremental, and task incremental settings, as well as ability to perform flexible inference on varying subsets of zero-shot and learned categories. Our proposed method achieves a good balance of learning speed, target task effectiveness, and zero-shot effectiveness. Code will be available at https://github.com/jessemelpolio/TreeProbe.