Towards Optimal VPU Compiler Cost Modeling by using Neural Networks to Infer Hardware Performances

TL;DR

This paper introduces VPUNN, a neural network-based cost model that improves the efficiency of neural network compiler scheduling by accurately predicting hardware performance, outperforming existing models on Intel VPU processors.

Contribution

The paper proposes a novel neural network approach for cost modeling in neural network compilers, enhancing schedule optimization accuracy and efficiency.

Findings

VPUNN outperforms state-of-the-art cost models on Intel VPU hardware.

Neural network-based cost modeling improves scheduling decisions.

The approach enables more efficient compiler performance across diverse configurations.

Abstract

Calculating the most efficient schedule of work in a neural network compiler is a difficult task. There are many parameters to be accounted for that can positively or adversely affect that schedule depending on their configuration - How work is shared between distributed targets, the subdivision of tensors to fit in memory, toggling the enablement of optimizations, etc. Traditionally, neural network compilers determine how to set these values by building a graph of choices and choosing the path with minimal 'cost'. These choices and their corresponding costs are usually determined by an algorithm crafted by engineers with a deep knowledge of the target platform. However, when the amount of options available to a compiler is large, it is very difficult to ensure that these models consistently produce an optimal schedule for all scenarios, whilst still completing compilation in an acceptable timeframe. This paper presents 'VPUNN' - a neural network-based cost model trained on low-level task profiling that consistently outperforms the state-of-the-art cost modeling in Intel's line of VPU processors.