Hierarchical Mixture of Experts: Generalizable Learning for High-Level Synthesis

TL;DR

This paper introduces a hierarchical Mixture of Experts model for high-level synthesis in FPGA design, improving domain generalization and performance prediction across diverse kernels using a multi-level learning approach.

Contribution

It proposes a novel hierarchical MoE structure with a two-stage training method, enhancing domain generalization in performance prediction for FPGA high-level synthesis.

Findings

Hierarchical MoE improves prediction accuracy across new kernels.

Two-stage training stabilizes expert specialization.

Model effectively captures multi-granularity program features.

Abstract

High-level synthesis (HLS) is a widely used tool in designing Field Programmable Gate Array (FPGA). HLS enables FPGA design with software programming languages by compiling the source code into an FPGA circuit. The source code includes a program (called "kernel") and several pragmas that instruct hardware synthesis, such as parallelization, pipeline, etc. While it is relatively easy for software developers to design the program, it heavily relies on hardware knowledge to design the pragmas, posing a big challenge for software developers. Recently, different machine learning algorithms, such as GNNs, have been proposed to automate the pragma design via performance prediction. However, when applying the trained model on new kernels, the significant domain shift often leads to unsatisfactory performance. We propose a more domain-generalizable model structure: a two-level hierarchical Mixture of Experts (MoE), that can be flexibly adapted to any GNN model. Different expert networks can learn to deal with different regions in the representation space, and they can utilize similar patterns between the old kernels and new kernels. In the low-level MoE, we apply MoE on three natural granularities of a program: node, basic block, and graph. The high-level MoE learns to aggregate the three granularities for the final decision. To train the hierarchical MoE stably, we further propose a two-stage training method to avoid expert polarization. Extensive experiments verify the effectiveness of the proposed hierarchical MoE. We publicized our codes at https://github.com/weikai-li/HierarchicalMoE.