Attacks on Dynamic Protocol Detection of Open Source Network Security Monitoring Tools

TL;DR

This paper examines the vulnerabilities of dynamic protocol detection methods in open-source network security tools, revealing significant evasion risks and highlighting fundamental disambiguation challenges affecting operational reliability.

Contribution

It provides the first comprehensive analysis of evasion vulnerabilities in open-source dynamic protocol detection mechanisms and discusses inherent trade-offs in their design.

Findings

All analyzed tools are vulnerable to evasion attacks.

Two out of three tools do not adequately address protocol disambiguation.

Fundamental issues in protocol disambiguation remain unaddressed.

Abstract

Protocol detection is the process of determining the application layer protocol in the context of network security monitoring, which requires a timely and precise decision to enable protocol-specific deep packet inspection. This task has proven to be complex, as isolated characteristics like port numbers are not sufficient to reliably determine the application layer protocol. Hence, more dynamic detection approaches have been developed. In this paper, we analyze the Dynamic Protocol Detection mechanisms employed by popular and widespread open-source network monitoring tools. We show on the example of HTTP that all analyzed detection mechanisms are vulnerable to evasion attacks, which pose a serious threat to real-world monitoring operations. We find that the underlying fundamental problem of protocol disambiguation is not adequately addressed in two of three monitoring systems that we analyzed. To enable adequate operational decisions, this paper highlights the inherent trade-offs within Dynamic Protocol Detection.