Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustness

TL;DR

This study compares different large language models to understand the trade-offs between efficiency, accuracy, and robustness against adversarial attacks, revealing that simplified architectures can offer a good balance for resource-constrained, resilient applications.

Contribution

It introduces a framework for evaluating the trade-offs among efficiency, performance, and adversarial robustness in LLMs and provides extensive experimental insights on three models.

Findings

Simplified models achieve higher efficiency and comparable robustness.

Gated Linear Attention and MatMul-Free LM outperform in robustness on adversarial datasets.

Trade-offs exist between model complexity, accuracy, and adversarial resilience.

Abstract

With the increasing demand for practical applications of Large Language Models (LLMs), many attention-efficient models have been developed to balance performance and computational cost. However, the adversarial robustness of these models remains under-explored. In this work, we design a framework to investigate the trade-off between efficiency, performance, and adversarial robustness of LLMs and conduct extensive experiments on three prominent models with varying levels of complexity and efficiency -- Transformer++, Gated Linear Attention (GLA) Transformer, and MatMul-Free LM -- utilizing the GLUE and AdvGLUE datasets. The AdvGLUE dataset extends the GLUE dataset with adversarial samples designed to challenge model robustness. Our results show that while the GLA Transformer and MatMul-Free LM achieve slightly lower accuracy on GLUE tasks, they demonstrate higher efficiency and either superior or comparative robustness on AdvGLUE tasks compared to Transformer++ across different attack levels. These findings highlight the potential of simplified architectures to achieve a compelling balance between efficiency, performance, and adversarial robustness, offering valuable insights for applications where resource constraints and resilience to adversarial attacks are critical.