Unitary-Group Invariant Kernels and Features from Transformed Unlabeled Data

TL;DR

This paper introduces theoretically grounded invariant kernels and features for data transformations, addressing practical challenges like lack of transformed labeled data and incomplete transformation observation, validated through experiments.

Contribution

It proposes a new invariant kernel SVM and a kernel extension for invariant feature extraction, solving key issues in practical invariant learning.

Findings

The invariant kernel SVM addresses unavailability of transformed labeled data.

The kernel extension effectively extracts invariant features.

Experiments validate the theoretical guarantees and practical effectiveness.

Abstract

The study of representations invariant to common transformations of the data is important to learning. Most techniques have focused on local approximate invariance implemented within expensive optimization frameworks lacking explicit theoretical guarantees. In this paper, we study kernels that are invariant to the unitary group while having theoretical guarantees in addressing practical issues such as (1) unavailability of transformed versions of labelled data and (2) not observing all transformations. We present a theoretically motivated alternate approach to the invariant kernel SVM. Unlike previous approaches to the invariant SVM, the proposed formulation solves both issues mentioned. We also present a kernel extension of a recent technique to extract linear unitary-group invariant features addressing both issues and extend some guarantees regarding invariance and stability. We present experiments on the UCI ML datasets to illustrate and validate our methods.