TRKM: Twin Restricted Kernel Machines for Classification and Regression

TL;DR

TRKM introduces a twin restricted kernel machine framework that enhances classification and regression by leveraging duality and kernel methods, addressing challenges of generalization and scalability in complex, large-scale datasets.

Contribution

The paper proposes TRKM, a novel twin RKM model that uses Fenchel-Young conjugate duality and kernel tricks to improve performance and scalability in classification and regression tasks.

Findings

TRKM outperforms baseline models on UCI and KEEL datasets.

TRKM demonstrates robustness in handling complex and uneven data distributions.

TRKM effectively predicts brain age in a real-world dataset.

Abstract

Restricted kernel machines (RKMs) have considerably improved generalization in machine learning. Recent advancements explored various techniques within the RKM framework, integrating kernel functions with least squares support vector machines (LSSVM) to mirror the energy function of restricted Boltzmann machines (RBM), leading to enhanced performance. However, RKMs may face challenges in generalization when dealing with unevenly distributed or complexly clustered data. Additionally, as the dataset size increases, the computational burden of managing high-dimensional feature spaces can become substantial, potentially hindering performance in large-scale datasets. To address these challenges, we propose twin restricted kernel machine (TRKM). TRKM combines the benefits of twin models with the robustness of the RKM framework to enhance classification and regression tasks. By leveraging the Fenchel-Young inequality, we introduce a novel conjugate feature duality, allowing the formulation of classification and regression problems in terms of dual variables. This duality provides an upper bound to the objective function of the TRKM problem, resulting in a new methodology under the RKM framework. The model uses an energy function similar to that of RBM, incorporating both visible and hidden variables corresponding to both classes. Additionally, the kernel trick is employed to map data into a high-dimensional feature space, where the model identifies an optimal separating hyperplane using a regularized least squares approach. Experiments on UCI and KEEL datasets confirm TRKM's superiority over baselines, showcasing its robustness and efficiency in handling complex data. Furthermore, We implemented the TRKM model on the brain age dataset, demonstrating its efficacy in predicting brain age.