Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces

TL;DR

This paper introduces a geometric framework for transfer learning in Hilbert spaces, proposing function encoders that enable effective interpolation and extrapolation across tasks, outperforming existing methods on multiple benchmarks.

Contribution

The paper presents a novel geometric characterization of transfer in Hilbert spaces and introduces function encoders with a new training scheme and universal approximation guarantees.

Findings

Function encoders outperform state-of-the-art methods on four benchmarks.

The approach effectively handles all three types of transfer: interpolation and two forms of extrapolation.

Theoretical guarantees support the universal approximation capability of the proposed encoders.

Abstract

A central challenge in transfer learning is designing algorithms that can quickly adapt and generalize to new tasks without retraining. Yet, the conditions of when and how algorithms can effectively transfer to new tasks is poorly characterized. We introduce a geometric characterization of transfer in Hilbert spaces and define three types of inductive transfer: interpolation within the convex hull, extrapolation to the linear span, and extrapolation outside the span. We propose a method grounded in the theory of function encoders to achieve all three types of transfer. Specifically, we introduce a novel training scheme for function encoders using least-squares optimization, prove a universal approximation theorem for function encoders, and provide a comprehensive comparison with existing approaches such as transformers and meta-learning on four diverse benchmarks. Our experiments demonstrate that the function encoder outperforms state-of-the-art methods on four benchmark tasks and on all three types of transfer.