TerEffic: Highly Efficient Ternary LLM Inference on FPGA

TL;DR

TerEffic introduces a specialized FPGA architecture for ternary quantized LLM inference, achieving high throughput and energy efficiency on edge devices, significantly outperforming existing hardware like NVIDIA Jetson and A100.

Contribution

The paper presents a novel FPGA-based system optimized for ternary quantized LLM inference, enabling flexible, high-performance, and energy-efficient deployment on edge hardware.

Findings

Achieves 16,300 tokens/sec on 370M-parameter model

Provides 192x throughput improvement over NVIDIA Jetson Orin Nano

Attains 8x power efficiency gain over NVIDIA A100

Abstract

Deploying Large Language Models (LLMs) efficiently on edge devices is often constrained by limited memory capacity and high power consumption. Low-bit quantization methods, particularly ternary quantization, have demonstrated significant potential in preserving model accuracy while substantially decreasing memory footprint and computational costs. However, existing general-purpose architectures and accelerators have not fully exploited the advantages of low-bit quantization due to insufficient specialized hardware support. We introduce TerEffic, an FPGA-based architecture tailored for ternary-quantized LLM inference. The proposed system offers flexibility through reconfigurable hardware to meet various system requirements. We evaluated two representative configurations: a fully on-chip design that stores all weights within on-chip memories, scaling out using multiple FPGAs, and an HBM-assisted design capable of accommodating larger models on a single FPGA board. Experimental results demonstrate significant performance and energy efficiency improvements. For single-batch inference on a 370 M-parameter model, our fully on-chip architecture achieves 16,300 tokens/second, delivering a throughput 192 times higher than NVIDIA Jetson Orin Nano with a power efficiency of 455 tokens/second/W, marking a 19-fold improvement. The HBM-assisted architecture processes 727 tokens/second for a larger 2.7B-parameter model, which is 3 times of the throughput of NVIDIA A100, while consuming only 46W, resulting in a power efficiency of 16 tokens/second/W, an 8-fold improvement over the A100.