Improving Variational Autoencoder using Random Fourier Transformation: An Aviation Safety Anomaly Detection Case-Study

TL;DR

This paper enhances Variational Autoencoders with Random Fourier Transformation to improve anomaly detection, demonstrating better performance on synthetic and aviation safety datasets, and introduces a trainable RFT variant.

Contribution

It introduces the use of RFT in VAEs for anomaly detection and proposes a trainable RFT variant, advancing model training and inference techniques.

Findings

Models with RFT outperform conventional models in anomaly detection.

RFT enables simultaneous learning of low and high-frequency features.

Trainable RFT shows no conclusive benefits over random RFT.

Abstract

In this study, we focus on the training process and inference improvements of deep neural networks (DNNs), specifically Autoencoders (AEs) and Variational Autoencoders (VAEs), using Random Fourier Transformation (RFT). We further explore the role of RFT in model training behavior using Frequency Principle (F-Principle) analysis and show that models with RFT turn to learn low frequency and high frequency at the same time, whereas conventional DNNs start from low frequency and gradually learn (if successful) high-frequency features. We focus on reconstruction-based anomaly detection using autoencoder and variational autoencoder and investigate the RFT's role. We also introduced a trainable variant of RFT that uses the existing computation graph to train the expansion of RFT instead of it being random. We showcase our findings with two low-dimensional synthetic datasets for data representation, and an aviation safety dataset, called Dashlink, for high-dimensional reconstruction-based anomaly detection. The results indicate the superiority of models with Fourier transformation compared to the conventional counterpart and remain inconclusive regarding the benefits of using trainable Fourier transformation in contrast to the Random variant.