Refine and Purify: Orthogonal Basis Optimization with Null-Space Denoising for Conditional Representation Learning

TL;DR

This paper introduces OD-CRL, a framework that enhances conditional representation learning by optimizing orthogonal bases and reducing interference through null-space denoising, leading to improved task-specific feature extraction.

Contribution

The paper proposes a novel OD-CRL framework combining AOBO and NSDP to address basis sensitivity and subspace interference in conditional representation learning.

Findings

Achieves state-of-the-art performance across multiple tasks.

Demonstrates superior generalization capabilities.

Effectively suppresses inter-subspace interference.

Abstract

Conditional representation learning aims to extract criterion-specific features for customized tasks. Recent studies project universal features onto the conditional feature subspace spanned by an LLM-generated text basis to obtain conditional representations. However, such methods face two key limitations: sensitivity to subspace basis and vulnerability to inter-subspace interference. To address these challenges, we propose OD-CRL, a novel framework integrating Adaptive Orthogonal Basis Optimization (AOBO) and Null-Space Denoising Projection (NSDP). Specifically, AOBO constructs orthogonal semantic bases via singular value decomposition with a curvature-based truncation. NSDP suppresses non-target semantic interference by projecting embeddings onto the null space of irrelevant subspaces. Extensive experiments conducted across customized clustering, customized classification, and customized retrieval tasks demonstrate that OD-CRL achieves a new state-of-the-art performance with superior generalization.