Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learning
Taehoon Kim, Henry Gouk, Minyoung Kim, Timothy Hospedales
TL;DR
This paper demonstrates that model fusion techniques can provide non-vacuous, data-dependent generalisation guarantees for deep networks, even with very limited data, enhancing trustworthiness in high-stakes AI applications.
Contribution
It reveals that existing model fusion methods can be adapted to certify deep networks' generalisation, especially in low-data regimes, bridging practice and theoretical guarantees.
Findings
Non-trivial generalisation guarantees for as few as 100 examples.
Guarantees are independent of base network size.
Applicable to vision and language models like VIT-B and mistral-7B.
Abstract
Certifying the IID generalisation ability of deep networks is the first of many requirements for trusting AI in high-stakes applications from medicine to security. However, when instantiating generalisation bounds for deep networks it remains challenging to obtain non-vacuous guarantees, especially when applying contemporary large models on the small scale data prevalent in such high-stakes fields. In this paper, we draw a novel connection between a family of learning methods based on model fusion and generalisation certificates, and surprisingly show that with minor adjustment several existing learning strategies already provide non-trivial generalisation guarantees. Essentially, by focusing on data-driven learning of downstream tasks by fusion rather than fine-tuning, the certified generalisation gap becomes tiny and independent of the base network size, facilitating its…
Peer Reviews
Decision·Submitted to ICLR 2026
1. The author demonstrated that certifiable generalization on ViT-B and Mistral-7B under 100-shot learning is unprecedented, compared with previous attempts (e.g., Lotfi et al. 2024) which require thousands of samples or smaller models. 2. Comprehensive experiments span both vision and language tasks (EuroSAT, GTSRB, MNIST, DTD, BBH, TweetEval), showing consistent trends across settings.
1. The introduction jumps quickly into technical framing (generalisation bounds, PAC-Bayes, model fusion) without clearly articulating the conceptual motivation: Why should we care about certifying large models in the first place? 2. The work did not conduct sensitivity experiment on prior variance $\lambda$ or the confidence parameter $\delta$.
- The paper establishes an elegant bridge between model merging and PAC-Bayesian certification, a previously unexplored direction. - It achieves non-vacuous generalization guarantees with large pretrained networks (ViT-B, Mistral-7B) using as few as 100 examples, which is unprecedented in the certification literature. - Requires only small modifications to existing merging pipelines; bound-aware optimization and Gaussian priors are straightforward. - Experiments cover both vision and language do
- While well-motivated, the paper does not provide new generalization bounds - most results hinge on reinterpreting existing theory. - Certification results are mostly numerical bounds rather than guarantees under formal verification (e.g., robustness or adversarial guarantees). - Some methodological explanations (e.g., computation of the PAC-Bayes term, choice of priors, gradient-free optimization) could be made more rigorous and reproducible. - Results exclude tasks where merging fails; it wou
1. The paper introduces an intellectually appealing idea by identifying a novel connection between model fusion methods and generalization certification. This perspective leads to a new form of generalization bound that unifies practical model-merging strategies with formal theoretical guarantees, offering a concise and elegant framework bridging empirical practice and generalization theory. 2. The paper presents comprehensive empirical validation. It evaluates the proposed generalization bounds
1. The exposition of the PAC-learning framework in Section 3 lacks conceptual precision. The PAC framework does not seek to bound the discrepancy between the empirical risk and the population risk for an arbitrary, fixed parameter $\theta$. For any $\theta$, that is independent of the data, the difference between $L(\theta)$ and $\hat{L}(\theta)$ converges at the standard Monte Carlo rate $O(1/n)$,taking square error loss function as an example. This is a purely sampling effect rather than a lea
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsModel Reduction and Neural Networks
MethodsBalanced Selection