Empirical Evaluation and Theoretical Analysis for Representation Learning: A Survey

TL;DR

This survey reviews current evaluation methods and theoretical analyses of representation learning, highlighting recent progress and discussing future research directions in the field.

Contribution

It provides a comprehensive overview of evaluation techniques and theoretical insights, extending previous work to guide future research in representation learning.

Findings

Evaluation methods vary depending on application

Representation learning algorithms achieve state-of-the-art performance

Discussion of future directions for the field

Abstract

Representation learning enables us to automatically extract generic feature representations from a dataset to solve another machine learning task. Recently, extracted feature representations by a representation learning algorithm and a simple predictor have exhibited state-of-the-art performance on several machine learning tasks. Despite its remarkable progress, there exist various ways to evaluate representation learning algorithms depending on the application because of the flexibility of representation learning. To understand the current representation learning, we review evaluation methods of representation learning algorithms and theoretical analyses. On the basis of our evaluation survey, we also discuss the future direction of representation learning. Note that this survey is the extended version of Nozawa and Sato (2022).