IOMMU Support for Virtual-Address Remote DMA in an ARMv8 environment

TL;DR

This paper demonstrates the successful testing and utilization of ARM's IOMMU (SMMU) for virtual-address remote DMA in a complex, multi-node system, enabling efficient and correct memory coherence.

Contribution

It provides a custom kernel module implementation to test and use the ARM SMMU, validating its operation for virtual address translation in a multi-node environment.

Findings

SMMU correctly translates virtual to physical addresses during DMA transfers

Enabled dynamic address translation without explicit pre-mapping

Validated SMMU operation across different transaction scenarios

Abstract

In complex systems with many compute nodes containing multiple CPUs that are coherent within each node, a key challenge is maintaining efficient and correct coherence between nodes. The Unimem system addresses this by proposing a virtualized global address space that enables such coherence, relying on the I/O Memory Management Unit (IOMMU) in each node. The goal of this thesis is to support this approach by successfully testing and using the IOMMU of a single node. For this purpose, we used ARM's IOMMU, known as the System Memory Management Unit (SMMU), which translates virtual addresses to physical addresses. Because Linux documentation for the SMMU is limited and unclear, we implemented custom kernel modules to test and use its functionality. First, we tested the SMMU in the Processing System (PS) of the Xilinx Zynq UltraScale+ MPSoC by developing a module that inserted virtual-to-physical address mappings into the SMMU. We then triggered a DMA transfer to a virtual address and observed that the request passed through the SMMU for address translation. We repeated this experiment by initiating DMA transactions from the Programmable Logic (PL) and similarly confirmed that the transactions were translated by the SMMU. Finally, we developed a module that enables transactions from the PL without requiring explicit pre-mapping of virtual and physical address pairs. This was achieved by configuring the SMMU with the page table pointer of a user process, allowing it to translate all relevant virtual addresses dynamically. Overall, we successfully demonstrated the correct operation of the SMMU across all tested scenarios. Due to time constraints, further exploration of advanced SMMU features is left for future work.