Can We Trust LLMs on Memristors? Diving into Reasoning Ability under Non-Ideality

TL;DR

This paper investigates how memristor non-idealities affect LLM reasoning and evaluates strategies like thinking mode and in-context learning to enhance robustness in memristor-based CIM architectures.

Contribution

It provides a comprehensive analysis of memristor non-idealities on LLM reasoning and systematically evaluates training-free strategies to improve robustness.

Findings

Reasoning capability decreases significantly with memristor non-idealities.

Shallow layer redundancy improves robustness effectively.

Thinking mode is better at low noise levels but degrades at high noise.

Abstract

Memristor-based analog compute-in-memory (CIM) architectures provide a promising substrate for the efficient deployment of Large Language Models (LLMs), owing to superior energy efficiency and computational density. However, these architectures suffer from precision issues caused by intrinsic non-idealities of memristors. In this paper, we first conduct a comprehensive investigation into the impact of such typical non-idealities on LLM reasoning. Empirical results indicate that reasoning capability decreases significantly but varies for distinct benchmarks. Subsequently, we systematically appraise three training-free strategies, including thinking mode, in-context learning, and module redundancy. We thus summarize valuable guidelines, i.e., shallow layer redundancy is particularly effective for improving robustness, thinking mode performs better under low noise levels but degrades at higher noise, and in-context learning reduces output length with a slight performance trade-off. Our findings offer new insights into LLM reasoning under non-ideality and practical strategies to improve robustness.