Tensor Memory Engine: On-the-fly Data Reorganization for Ideal Locality

TL;DR

This paper introduces a hardware/software co-designed Tensor Memory Engine that reorganizes data on-the-fly to improve memory locality for data-intensive applications on FPGA platforms.

Contribution

It presents a novel approach that decouples data reorganization from computation, enhancing memory locality without redesigning applications.

Findings

Provides ideal cache locality for applications with poor locality

Seamlessly integrates with existing CPU data paths on FPGA platforms

Reduces need for costly application redesigns and tuning

Abstract

The shift to data-intensive processing from the cloud to the edge has introduced new challenges and expectations for the next generation of intelligent computing systems. As the memory wall continues to grow, modern systems can only meet these performance expectations by displaying data access patterns that exhibit ideal layouts in memory and ideal spatiotemporal locality in caches. However, only a few data-intensive applications are characterized by ideal locality. Instead, most applications exhibit either (i) poor locality when naively implemented and must undergo costly redesigns and tuning or (ii) inflated memory footprint to offer proper locality. To address the aforementioned challenges, we propose a hardware/software co-designed approach that can be implemented on commercially available SoC/FPGA platforms. Our approach seamlessly inserts in the CPUs' data path a Tensor Memory Engine that provides data with an ideal cache locality to running applications by (i) accessing the memory on behalf of the CPUs and (ii) composing a re-organized view of the memory layout. Unlike in- and near-memory computing approaches, it sets itself apart by clearly decoupling computing and memory accesses; computation is still performed on CPUs while the data re-organization is delegated to the Tensor Memory Engine.