Detection of Malicious DNS-over-HTTPS Traffic: An Anomaly Detection Approach using Autoencoders

TL;DR

This paper introduces an autoencoder-based anomaly detection method for encrypted DNS-over-HTTPS traffic, enabling detection of unseen malicious activities without decrypting the data.

Contribution

It presents a novel autoencoder approach that detects malicious DoH traffic solely from encrypted data, outperforming existing anomaly detection algorithms.

Findings

Achieves a median F-1 score of 99% in detecting malicious traffic.

Outperforms local outlier factor, one-class SVM, isolation forest, and variational autoencoders.

Effective in identifying zero-day attacks in encrypted DNS traffic.

Abstract

To maintain the privacy of users' web browsing history, popular browsers encrypt their DNS traffic using the DNS-over-HTTPS (DoH) protocol. Unfortunately, encrypting DNS packets prevents many existing intrusion detection systems from using plaintext domain names to detect malicious traffic. In this paper, we design an autoencoder that is capable of detecting malicious DNS traffic by only observing the encrypted DoH traffic. Compared to previous works, the proposed autoencoder looks for anomalies in DoH traffic, and thus can detect malicious traffic that has not been previously observed, i.e., zero-day attacks. We run extensive experiments to evaluate the performance of our proposed autoencoder and compare it to that of other anomaly detection algorithms, namely, local outlier factor, one-class support vector machine, isolation forest, and variational autoencoders. We find that our proposed autoencoder achieves the highest detection performance, with a median F-1 score of 99\% over several types of malicious traffic.