GeoStack: A Framework for Quasi-Abelian Knowledge Composition in VLMs

TL;DR

GeoStack is a modular framework that enables the composition of multiple domain experts into VLMs, preserving foundational knowledge and achieving efficient inference with minimal forgetting.

Contribution

Introduces GeoStack, a geometric and structural constraint-based framework for knowledge composition in VLMs that ensures constant-time inference and mitigates catastrophic forgetting.

Findings

GeoStack achieves constant-time inference regardless of expert number.

Experimental results show effective multi-domain adaptation.

GeoStack significantly reduces catastrophic forgetting.

Abstract

We address the challenge of knowledge composition in Vision-Language Models (VLMs), where accumulating expertise across multiple domains or tasks typically leads to catastrophic forgetting. We introduce GeoStack (Geometric Stacking), a modular framework that allows independently trained domain experts to be composed into a unified model. By imposing geometric and structural constraints on the adapter manifold, GeoStack ensures the foundational knowledge of the base model is preserved. Furthermore, we mathematically demonstrate a weight-folding property that achieves constant-time inference complexity ($O(1)$), regardless of the number of integrated experts. Experimental results across multi-domain adaptation and class-incremental learning show that GeoStack provides an efficient mechanism for long-term knowledge composition while significantly mitigating catastrophic forgetting. Code is available at https://github.com/QuantitativeImagingLaboratory/GeoStack.