FlowDNS: Correlating Netflow and DNS Streams at Scale

TL;DR

FlowDNS presents a real-time system that correlates DNS responses with network traffic to accurately identify customer services, improving network management and security at scale.

Contribution

This work introduces a novel correlation system that combines DNS and network-layer data, deployed at a large ISP, achieving high accuracy in service identification.

Findings

Correlates 81.7% of traffic with services without data loss.

Detects 0.5% of traffic as malicious or spam domains.

Enables detailed traffic attribution using DNS and BGP data.

Abstract

Knowing customer's interests, e.g. which Video-On-Demand (VoD) or Social Network services they are using, helps telecommunication companies with better network planning to enhance the performance exactly where the customer's interests lie, and also offer the customers relevant commercial packages. However, with the increasing deployment of CDNs by different services, identification, and attribution of the traffic on network-layer information alone becomes a challenge: If multiple services are using the same CDN provider, they cannot be easily distinguished based on IP prefixes alone. Therefore, it is crucial to go beyond pure network-layer information for traffic attribution. In this work, we leverage real-time DNS responses gathered by the clients' default DNS resolvers. Having these DNS responses and correlating them with network-layer headers, we are able to translate CDN-hosted domains to the actual services they belong to. We design a correlation system for this purpose and deploy it at a large European ISP. With our system, we can correlate an average of 81.7% of the traffic with the corresponding services, without any loss on our live data streams. Our correlation results also show that 0.5% of the daily traffic contains malformatted, spamming, or phishing domain names. Moreover, ISPs can correlate the results with their BGP information to find more details about the origin and destination of the traffic. We plan to publish our correlation software for other researchers or network operators to use.