Dynamic Cluster Analysis to Detect and Track Novelty in Network Telescopes

TL;DR

This paper presents a dynamic clustering approach for analyzing network telescope traffic to detect and track new and evolving cybersecurity threats, simplifying manual analysis for security experts.

Contribution

It introduces a three-stage pipeline combining self-supervised embeddings, clustering, and temporal tracking to identify novel patterns in noisy network data.

Findings

Identified 50-70 clusters per day, with 10-20 being previously unseen.

Successfully tracked the evolution of clusters over 20 days.

Highlighted activity changes and new incidents in network traffic.

Abstract

In the context of cybersecurity, tracking the activities of coordinated hosts over time is a daunting task because both participants and their behaviours evolve at a fast pace. We address this scenario by solving a dynamic novelty discovery problem with the aim of both re-identifying patterns seen in the past and highlighting new patterns. We focus on traffic collected by Network Telescopes, a primary and noisy source for cybersecurity analysis. We propose a 3-stage pipeline: (i) we learn compact representations (embeddings) of hosts through their traffic in a self-supervised fashion; (ii) via clustering, we distinguish groups of hosts performing similar activities; (iii) we track the cluster temporal evolution to highlight novel patterns. We apply our methodology to 20 days of telescope traffic during which we observe more than 8 thousand active hosts. Our results show that we efficiently identify 50-70 well-shaped clusters per day, 60-70% of which we associate with already analysed cases, while we pinpoint 10-20 previously unseen clusters per day. These correspond to activity changes and new incidents, of which we document some. In short, our novelty discovery methodology enormously simplifies the manual analysis the security analysts have to conduct to gain insights to interpret novel coordinated activities.