Test-time Adaptation for Regression by Subspace Alignment
Kazuki Adachi, Shin'ya Yamaguchi, Atsutoshi Kumagai, Tomoki Hamagami
TL;DR
This paper introduces Significant-subspace Alignment (SSA), a novel method for test-time adaptation in regression tasks that aligns features in a subspace most relevant to the output, improving adaptation effectiveness.
Contribution
The paper proposes SSA, a new feature alignment approach for regression TTA that identifies significant subspaces and weights dimensions to enhance adaptation performance.
Findings
SSA outperforms existing baselines on real-world datasets.
Naive feature alignment is ineffective for regression tasks.
Subspace detection and dimension weighting improve adaptation quality.
Abstract
This paper investigates test-time adaptation (TTA) for regression, where a regression model pre-trained in a source domain is adapted to an unknown target distribution with unlabeled target data. Although regression is one of the fundamental tasks in machine learning, most of the existing TTA methods have classification-specific designs, which assume that models output class-categorical predictions, whereas regression models typically output only single scalar values. To enable TTA for regression, we adopt a feature alignment approach, which aligns the feature distributions between the source and target domains to mitigate the domain gap. However, we found that naive feature alignment employed in existing TTA methods for classification is ineffective or even worse for regression because the features are distributed in a small subspace and many of the raw feature dimensions have little…
Peer Reviews
Decision·ICLR 2025 Poster
The topic of adapting regression models to unknown target distributions is highly relevant, particularly given the increasing use of machine learning in diverse applications. The proposed SSA approach is innovative and addresses a gap in the literature.
1. In Section 3.1 of the article, the use of "two diagonal Gaussian distributions" is not adequately justified, and the advantages and limitations of employing such distributions are not discussed. 2. The proposed method in this paper exhibits limited originality, as its various components are commonly found in existing models. 3. The dataset utilized in this study has a relatively small sample size, which weakens the persuasiveness of the findings. Additionally, there is a lack of comparative a
1.SSA introduces a novel approach for TTA in regression tasks by combining subspace detection and dimension weighting, which is an innovative contribution to the field. 2.The paper conducts extensive experiments on multiple real-world datasets, validating the effectiveness of SSA. 3.Compared to the original model and other baseline methods, SSA achieves higher R2 scores across multiple datasets, demonstrating performance improvements in regression tasks.
1.SSA assumes covariate shift, where p(y|x) remains unchanged. The paper does not address distribution shifts where p(y|x) changes, such as concept drift, limiting its applicability in broader scenarios. 2.While the paper mentions the selection of the parameter λ, it lacks a detailed discussion on the impact of other hyperparameters, which could affect the model's generalizability and adaptability. 3.The paper does not discuss the computational complexity of SSA, particularly its performance wit
- The paper proposes a novel method for Test time domain adaptation for regression, and adapted common benchmarking procedures for classification to regression, especially on the UTK Face dataset. - The paper is well-written and clear. - The method and results seem convincing and easy to implement.
- In the related work, feature alignment methods for TTA are reviewed with the claim, “Although some of these methods are directly applicable to regression, we have observed that they are not effective or even degrade regression performance.” However, these methods are not included in the experiments, leaving this claim unsupported. - In Table 1, the subspace dimension for Biwi Kinect is shown as ‘34.5’, which is unclear, as dimensions are typically integers. - MLPs and CNNs seem to produce embe
Code & Models
Videos
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Machine Learning and ELM