ML-based tunnel detection and tunneled application classification

TL;DR

This paper develops a comprehensive machine learning pipeline for detecting and classifying encrypted tunneling protocols, including OpenVPN and Wireguard, analyzing feature effectiveness and generalization across different network conditions.

Contribution

It introduces a complete detection and classification pipeline for tunneling protocols and applications, addressing protocol coverage gaps and analyzing generalization and robustness.

Findings

Effective detection of OpenVPN and Wireguard tunnels.

Insights into feature importance for tunnel classification.

Analysis of domain generalization and adversarial robustness.

Abstract

Encrypted tunneling protocols are widely used. Beyond business and personal uses, malicious actors also deploy tunneling to hinder the detection of Command and Control and data exfiltration. A common approach to maintain visibility on tunneling is to rely on network traffic metadata and machine learning to analyze tunnel occurrence without actually decrypting data. Existing work that address tunneling protocols however exhibit several weaknesses: their goal is to detect application inside tunnels and not tunnel identification, they exhibit limited protocol coverage (e.g. OpenVPN and Wireguard are not addressed), and both inconsistent features and diverse machine learning techniques which makes performance comparison difficult. Our work makes four contributions that address these limitations and provide further analysis. First, we address OpenVPN and Wireguard. Second, we propose a complete pipeline to detect and classify tunneling protocols and tunneled applications. Third, we present a thorough analysis of the performance of both network traffic metadata features and machine learning techniques. Fourth, we provide a novel analysis of domain generalization regarding background untunneled traffic, and, both domain generalization and adversarial learning regarding Maximum Transmission Unit (MTU).