Transfer-Learning Across Datasets with Different Input Dimensions: An Algorithm and Analysis for the Linear Regression Case

TL;DR

This paper introduces a simple, efficient transfer learning algorithm for linear regression that combines datasets with different input dimensions, backed by theoretical robustness analysis and strong empirical results on real datasets.

Contribution

It presents a novel transfer learning method for datasets with varying input dimensions, with theoretical guarantees and competitive empirical performance.

Findings

Achieves state-of-the-art results on 9 real datasets.

The algorithm is computationally efficient, similar to ordinary least squares.

It outperforms existing linear transfer learning methods and matches non-linear approaches.

Abstract

With the development of new sensors and monitoring devices, more sources of data become available to be used as inputs for machine learning models. These can on the one hand help to improve the accuracy of a model. On the other hand, combining these new inputs with historical data remains a challenge that has not yet been studied in enough detail. In this work, we propose a transfer learning algorithm that combines new and historical data with different input dimensions. This approach is easy to implement, efficient, with computational complexity equivalent to the ordinary least-squares method, and requires no hyperparameter tuning, making it straightforward to apply when the new data is limited. Different from other approaches, we provide a rigorous theoretical study of its robustness, showing that it cannot be outperformed by a baseline that utilizes only the new data. Our approach achieves state-of-the-art performance on 9 real-life datasets, outperforming the linear DSFT, another linear transfer learning algorithm, and performing comparably to non-linear DSFT.