Smaller, Faster, Cheaper: Architectural Designs for Efficient Machine Learning

TL;DR

This paper explores architectural principles to develop smaller, faster, and more resource-efficient machine learning models, focusing on data handling, modified attention mechanisms, and leveraging normalizing flows.

Contribution

It introduces novel architectural strategies including data ingress/egress optimization, restricted attention in vision transformers, and leveraging normalizing flows for improved efficiency.

Findings

Enhanced data utilization improves model performance.

Restricted attention increases neural expressivity.

Leveraging normalizing flows aids in better model distillation.

Abstract

Major advancements in the capabilities of computer vision models have been primarily fueled by rapid expansion of datasets, model parameters, and computational budgets, leading to ever-increasing demands on computational infrastructure. However, as these models are deployed in increasingly diverse and resource-constrained environments, there is a pressing need for architectures that can deliver high performance while requiring fewer computational resources. This dissertation focuses on architectural principles through which models can achieve increased performance while reducing their computational demands. We discuss strides towards this goal through three directions. First, we focus on data ingress and egress, investigating how information may be passed into and retrieved from our core neural processing units. This ensures that our models make the most of available data, allowing smaller architectures to become more performant. Second, we investigate modifications to the core neural architecture, applied to restricted attention in vision transformers. This section explores how removing uniform context windows in restricted attention increases the expressivity of the underlying neural architecture. Third, we explore the natural structures of Normalizing Flows and how we can leverage these properties to better distill model knowledge. These contributions demonstrate that careful design of neural architectures can increase the efficiency of machine learning algorithms, allowing them to become smaller, faster, and cheaper.