Sharpness-aware Second-order Latent Factor Model for High-dimensional and Incomplete Data

TL;DR

This paper introduces a novel sharpness-aware second-order latent factor model (SSLF) that leverages Hessian information and sharpness terms to improve representation learning on high-dimensional, incomplete data, demonstrating superior performance.

Contribution

The paper proposes a new SSLF model that incorporates sharpness-aware optimization and second-order information to enhance low-rank representation learning for complex data.

Findings

Consistently outperforms state-of-the-art baselines on industrial datasets.

Effectively captures node-to-node interaction patterns in HDI data.

Improves generalization through sharpness-aware optimization.

Abstract

Second-order Latent Factor (SLF) model, a class of low-rank representation learning methods, has proven effective at extracting node-to-node interaction patterns from High-dimensional and Incomplete (HDI) data. However, its optimization is notoriously difficult due to its bilinear and non-convex nature. Sharpness-aware Minimization (SAM) has recently proposed to find flat local minima when minimizing non-convex objectives, thereby improving the generalization of representation-learning models. To address this challenge, we propose a Sharpness-aware SLF (SSLF) model. SSLF embodies two key ideas: (1) acquiring second-order information via Hessian-vector products; and (2) injecting a sharpness term into the curvature (Hessian) through the designed Hessian-vector products. Experiments on multiple industrial datasets demonstrate that the proposed model consistently outperforms state-of-the-art baselines.