Benchmarking Various ML Solutions in Complex Intent-Based Network Management Systems

TL;DR

This paper presents a benchmarking approach for evaluating the computational performance of various machine learning techniques in intent-based network management systems, especially on heterogeneous hardware platforms, using collaborative filtering methods.

Contribution

It introduces a novel benchmarking method based on collaborative filtering to estimate ML performance in complex IBN systems with minimal initial data.

Findings

Accurately estimates ML performance with few benchmarks

Effective on heterogeneous hardware platforms

Applicable to complex ICT supply chain systems

Abstract

Intent-based networking (IBN) solutions to managing complex ICT systems have become one of the key enablers of intelligent and autonomous network management. As the number of machine learning (ML) techniques deployed in IBN increases, it becomes increasingly important to understand their expected performance. Whereas IBN concepts are generally specific to the use case envisioned, the underlying platforms are generally heterogenous, comprised of complex processing units, including CPU/GPU, CPU/FPGA and CPU/TPU combinations, which needs to be considered when running the ML techniques chosen. We focus on a case study of IBNs in the so-called ICT supply chain systems, where multiple ICT artifacts are integrated in one system based on heterogeneous hardware platforms. Here, we are interested in the problem of benchmarking the computational performance of ML technique defined by the intents. Our benchmarking method is based on collaborative filtering techniques, relying on ML-based methods like Singular Value Decomposition and Stochastic Gradient Descent, assuming initial lack of explicit knowledge about the expected number of operations, framework, or the device processing characteristics. We show that it is possible to engineer a practical IBN system with various ML techniques with an accurate estimated performance based on data from a few benchmarks only.