Joint Architecture-Token-Bitwidth Multi-Axis Optimization of Vision Transformers for Semiconductor IC Packaging

TL;DR

This paper introduces a holistic multi-axis optimization framework for Vision Transformers, jointly tuning architecture, token processing, and bit-width to significantly improve efficiency for industrial applications without sacrificing accuracy.

Contribution

It is among the first to jointly optimize architecture, token, and bit-width in ViTs, specifically targeting resource-efficient deployment in semiconductor manufacturing.

Findings

Achieves over 10x throughput improvement and 10x reductions in parameters, FLOPs, and energy consumption.

Maintains accuracy on industrial defect classification tasks despite aggressive compression.

Demonstrates the effectiveness of combined multi-axis optimization in real-world industrial scenarios.

Abstract

Vision Transformers (ViTs) have achieved strong performance in visual recognition, yet their deployment in resource-constrained industrial environments remains limited. Some main challenges are their high computational cost, memory requirement, and energy consumption. While individual efficiency techniques such as neural architecture search (NAS), token compression, and low-precision inference have been extensively studied, most prior work targets only a single optimization axis, limiting overall deployment gains while preserving accuracy. In this paper, we present one of the first holistic frameworks that jointly optimizes three complementary axes: architecture, token, and bit-width. Specifically, the framework identifies compact backbones via Neural Architecture Search (AutoFormer), reduces information processing via token merging (ToMe), and accelerates per-operation execution via fp16 mixed-precision inference. Starting from a DeiT-B/16 baseline, we first analyze accuracy-efficiency trade-offs on ImageNet-1K under aggressive compression. Then, we apply the selected configurations to a real-world in-house 3D X-ray semiconductor defect classification dataset for IC chip packaging inspection. Results show that the proposed multi-axis framework achieves more than 10 times improvement in throughput along with over 10 times reductions in parameter count, FLOPs, and energy consumption, while maintaining the required accuracy on the downstream industrial task. To the best of our knowledge, this is among the earliest works to jointly optimize architecture, token, and bit-width dimensions in ViTs and the first such resource-efficient, deployment-focused study tailored to semiconductor manufacturing.