TMUAD: Enhancing Logical Capabilities in Unified Anomaly Detection Models with a Text Memory Bank

TL;DR

TMUAD introduces a novel three-memory framework combining text and image memory banks to improve logical and structural anomaly detection in images, achieving state-of-the-art results across multiple datasets.

Contribution

The paper proposes a three-memory framework that unifies logical and structural anomaly detection using a text memory bank and image memory banks, enhancing detection capabilities.

Findings

Achieves state-of-the-art performance on seven datasets.

Effectively captures logical descriptions and object contours.

Improves anomaly detection accuracy in industrial and medical images.

Abstract

Anomaly detection, which aims to identify anomalies deviating from normal patterns, is challenging due to the limited amount of normal data available. Unlike most existing unified methods that rely on carefully designed image feature extractors and memory banks to capture logical relationships between objects, we introduce a text memory bank to enhance the detection of logical anomalies. Specifically, we propose a Three-Memory framework for Unified structural and logical Anomaly Detection (TMUAD). First, we build a class-level text memory bank for logical anomaly detection by the proposed logic-aware text extractor, which can capture rich logical descriptions of objects from input images. Second, we construct an object-level image memory bank that preserves complete object contours by extracting features from segmented objects. Third, we employ visual encoders to extract patch-level image features for constructing a patch-level memory bank for structural anomaly detection. These three complementary memory banks are used to retrieve and compare normal images that are most similar to the query image, compute anomaly scores at multiple levels, and fuse them into a final anomaly score. By unifying structural and logical anomaly detection through collaborative memory banks, TMUAD achieves state-of-the-art performance across seven publicly available datasets involving industrial and medical domains. The model and code are available at https://github.com/SIA-IDE/TMUAD.

Tables

Table 1. TABLE I: Image-level AUROC comparison results on MVTec LOCO [ 6 ] dataset for unified logical and structural anomaly detection. The results marked “ ∗ \ast ”, “ ¶ \P ” and “ ‡ \ddagger ” are reported by  [ 25 ] ,  [ 29 ] and  [ 56 ] , respectively. “Individual CLIP” indicates CLIP trained individually on each dataset.

Category	Dream $\P$	GCAD$\ast$	PatchCore $\ast$	OmniAL $‡$	GeneralAD$\ast$	LogicAL	PBAS	UniVAD	LogSAD	Ours	Individual CLIP
	[57] CVPR21	[6] IJCV21	[2] CVPR22	[58] CVPR23	[59] ECCV24	[56] CVPR24	[21] TCSVT25	[30] CVPR25	[25] CVPR25	TMUAD-U	TMUAD-I	PBAS+$M_T$
Breakfast Box	80.3	83.9	77.1	75.9	-	85.4	88.4	88.4	95.7	99.5	99.5	94.5
Juice Bottle	94.3	99.4	94.6	99.5	-	98.5	89.1	90.9	95.2	94.9	94.9	93.1
Pushpins	68.6	86.2	74.1	79.6	-	87.4	76.5	74.9	83.6	93.1	93.1	91.3
Screw Bag	70.6	63.2	73.3	83.1	-	82.0	73.2	66.5	83.2	94.9	96.9	94.3
Splicing Connectors	85.4	83.9	86.0	88.1	-	89.0	80.0	88.8	93.5	91.2	91.8	86.8
$\Delta$AUROC	25.7	36.2	21.3	23.6	-	16.5	15.9	24.4	12.5	8.3	7.7	7.7
Average	79.8	83.3	81.0	85.0	84.9	88.5	81.4	81.9	90.2	94.6	95.2	92.0

Table 2. TABLE II: Image-level AUROC comparison results on industrial and medical anomaly detection datasets. The results marked “ ∗ \ast ” and “ ¶ \P ” are reported by  [ 38 ] and  [ 30 ] , respectively. “Individual CLIP” indicates CLIP trained individually on each dataset.

Dataset	WinCLIP $\ast$	VAND $\ast$	AnomalyGPT $\P$	AnomalyCLIP $\ast$	MVFA-AD $\ast$	AdaCLIP $\ast$	AA-CLIP	UniVAD	Ours	Individual CLIP
	[60] CVPR23	[61] CVPRW23	[35] AAAI24	[62] ICLR24	[63] CVPR24	[64] ECCV24	[38] CVPR25	[30] CVPR25	TMUAD-U	TMUAD-I	PBAS+$M_T$ / PBAS
MVTec AD [4]	91.8	86.1	94.1	90.9	86.6	90.0	90.5	99.0	99.0	99.1	99.5/99.5
VisA [5]	78.0	66.4	87.4	82.1	76.5	84.3	84.6	96.2	95.9	96.0	97.1/97.1
MPDD [51]	63.6	73.0	-	73.7	70.9	72.1	75.1	97.2	97.5	98.3	97.9/97.9
BrainMRI [52]	66.5	58.8	73.2	83.3	70.9	80.2	80.2	84.2	89.2	-	-
RESC [54]	42.5	65.6	82.4	75.7	77.3	82.7	82.7	89.4	89.6	-	-
LiverCT [53]	64.2	54.7	60.3	61.6	63.0	64.2	69.7	70.7	72.1	-	-

Table 3. TABLE III: Pixel-level AUROC comparison results on industrial and medical anomaly detection datasets.

Dataset	WinCLIP $\ast$	VAND $\ast$	AnomalyGPT $\P$	AnomalyCLIP $\ast$	MVFA-AD $\ast$	AdaCLIP $\ast$	AA-CLIP	UniVAD	Ours	Individual CLIP
	[60] CVPR23	[61] CVPRW23	[35] AAAI24	[62] ICLR24	[63] CVPR24	[64] ECCV24	[38] CVPR25	[30] CVPR25	TMUAD-U	TMUAD-I	PBAS+$M_T$ / PBAS
MVTec AD [4]	85.1	87.6	95.3	91.1	84.9	89.9	91.9	96.3	96.9	96.9	98.3/98.3
VisA [5]	79.6	94.2	96.2	95.4	93.4	95.5	95.5	98.5	98.8	98.8	94.9/94.9
MPDD [51]	95.2	94.9	-	96.2	94.5	96.6	96.7	97.7	98.5	98.6	97.8/97.8
BrainMRI [52]	86.0	94.5	96.0	96.2	95.6	93.9	95.5	96.7	97.6	-	-
LiverCT [53]	96.2	95.6	95.8	93.9	96.8	94.5	97.8	97.2	97.6	-	-

Table 4. TABLE IV: Image-level AUROC comparison results on the MVTec LOCO [ 6 ] dataset. “Average” represents the image-level average AUROC for unified logical and structural anomaly detection.

Methods	Logical	Structural	Average
SimpleNet [66]	71.5	83.7	77.6
PBAS [21]	76.6	88.5	82.6
UniVAD [30]	73.8	92.9	83.3
LogSAD [25]	89.3	93.1	91.2
Ours	91.8	98.5	95.2

Table 5. TABLE V: Ablation studies of three memory bank on the MVTec LOCO [ 6 ] dataset.

Memory banks			Structural		Logical		Average
$M_T$	$M_O$	$M_P$	Image	Pixel	Image	Pixel	Image	Pixel
✓			63.4	64.0	88.4	77.6	75.9	70.8
	✓		96.4	62.0	58.1	54.1	77.3	58.1
		✓	91.1	91.4	69.7	73.0	80.4	82.2
	✓	✓	98.5	94.2	70.4	76.1	84.5	85.2
✓	✓	✓	98.5	94.6	91.8	84.8	95.2	88.3

Table 6. TABLE VI: Ablation studies of the class-level text memory bank M T M_{T} on the MVTec LOCO [ 6 ] dataset in image-level AUROC.

Memory bank construction			Anomaly score calculation			AUROC
Grounded SAM [45]	VLMs [44] $\to T_{img}^j$	VLMs $\to O_{pos}^j\to T_{img}^j$	CLIP text encoder[43]	T5 encoder[67]	Class-level Text Matching	only $M_T$
	✓		✓			54.0
	✓			✓		55.9
✓	✓			✓		57.5
✓		✓		✓		81.7
✓		✓			✓	88.4

Table 7. TABLE VII: Plug-and-play class-level text memory bank M T M_{T} on the MVTec LOCO [ 6 ] dataset.

Methods	AUROC
UniVAD [30]	81.9
UniVAD [30] + $M_T$	91.2 ($\uparrow$ 9.3)

Table 8. TABLE VIII: Ablation studies of the object-level image memory bank M O M_{O} on the MVTec LOCO [ 6 ] dataset in image-level AUROC.

Encoder	Segmentation model	AUROC (only $M_O$)
DINOv2 [49]	Grounded SAM [45]	65.8
CLIP [43]	Grounded SAM [45]	67.7
DINOv2 [49]	SAM only [42]	74.9
CLIP [43]	SAM only [42]	75.1

Table 9. TABLE IX: Ablation studies of the patch-level image memory bank M P M_{P} in image-level AUROC.

Dataset	DINOv2 [49]	CLIP [43]	DINOv2+CLIP
MVTec LOCO [6]	74.8	76.0	78.9
MVTec AD [4]	97.2	98.7	98.9
VisA [5]	91.2	94.3	95.5

Table 10. TABLE X: Few shot (4-shot) image-level AUROC of unified anomaly detection on the MVTec LOCO [ 6 ] dataset.

Methods	PatchCore	WinCLIP+	PromptAD	UniVAD	LogSAD	Ours
	[2]	[60]	[3]	[30]	[25]
AUROC (%)	68.7	71.3	73.5	76.0	86.3	81.2$\to$85.3

Table 11. TABLE XI: Resource efficiency and performance analysis on the MVTec LOCO [ 6 ] dataset in image-level AUROC. “-I” indicates CLIP TRAINED INDIVIDUALLY on the MVTec LOCO [ 6 ] dataset, while “-U” denotes CLIP TRAINED uniformly across all datasets.

Methods	Inference Time (s)	Parameters (M)	AUROC
DeCo-Diff [7]	29.0	478.6	69.5
PBAS [21]	0.9	72.8	81.4
UniVAD [30]	1286.1	2402.1	83.3
TMUAD-U-L	131.8	11741.0	94.9
TMUAD-U-B (Ours)	81.7	11741.0	94.6
TMUAD-U-B (3 GPUs)	30.7	11741.0	94.6
TMUAD-U-S	32.9	3449	94.1
TMUAD-I-B	81.7	11741.0	95.2
PBAS-I-B	33.9	9534.8	92.0

Equations

$$s_{max}^{\hat{j}}={\max }_{T_{img}^i\in M_T}\left\{O_{size}^j|O_{class}^j=O_{class}^{\hat{j}}\right\},$$

$$s_{min}^{\hat{j}}={\min }_{T_{img}^i\in M_T}\left\{O_{size}^j|O_{class}^j=O_{class}^{\hat{j}}\right\},$$

$$S_O={\sum }_{\hat{s}=1}^{{\hat{n}}_s}(1-{\max }_{F_{seg}^s\in F_O^i,F_O^i\in M_O}(\frac{F_{seg}^s\cdot {\hat{F}}_{seg}^{\hat{s}}{}^{\top }}{\Vert F_{seg}^s\Vert \Vert {\hat{F}}_{seg}^{\hat{s}}\Vert }))\cdot {\hat{I}}_{seg}^{\hat{s}}.$$

$$S_P={\max }_{F_P^i\in M_P}(1-\text{upsample}(\frac{F_P^i\cdot {\hat{F}}_P^{\top }}{\Vert F_P^i\Vert \Vert {\hat{F}}_P\Vert })),$$

$$S_{pixel}={\lambda }_1{S}_T+{\lambda }_2{S}_I+{\lambda }_3{S}_P.$$

Full text

I Introduction

Industrial anomaly detection is critical for quality inspection and predictive maintenance, involving the identification of deviations from normal patterns with limited or no prior knowledge of potential anomalies [1, 2, 3]. Industrial anomaly images can be roughly categorized into two types: structural anomalies [4, 5] and logical anomalies [6]. Structural anomaly detection methods include reconstruction-based methods (autoencoder [7, 8, 9, 10] and diffusion [11, 12, 13, 14]), teacher-student architecture-based methods [15, 16, 17, 18], and memory bank-based methods [2, 19, 20, 21, 22, 23, 24]. These methods 1) typically focus on detecting local regions and surfaces, such as scratches, dents, stains, cracks, or contaminants in manufactured products [25], and 2) fail in logical anomaly detection, as such anomalies may appear normal in local details but manifest as object missing, incorrect position, or incorrect combinations of normal elements, requiring the capture of long-range and global dependencies [26, 27, 28, 29, 30].

To unify structural anomaly and logical anomaly detection, current memory-based anomaly detection methods [31, 29, 30, 25] determine the presence of anomalies (image-level anomalies) and detect anomalous regions (pixel-level anomalies) by extracting image features. As shown in Fig. 1(a), these methods aim to simultaneously extract global and local image features for detecting structural anomalies, as well as component and positional image features for identifying logical anomalies. Currently, techniques for extracting global and local features are similar, typically employing Vision Transformers to encode images into patch features. Recent approaches generally enhance performance by designing new strategies for extracting components and positional features to understand the relationships between components. For example, GCAD [6] effectively emphasizes causal relationships in images by extracting causal matrices from patch features and applying causality graph sparsification. UniVAD [30] uses a graph-enhanced component modeling, which abstracts inter-component relationships into graph structures. Logical relationships are captured through nodes and edges, enabling the extraction of relational features. SAM-LAD [31] initially employs a base model to perform zero-shot segmentation and subsequently models inter-component relationships by constructing a similarity matrix to extract logical dependencies. However, there are two issues in the anomaly detection process: 1) Both structural anomaly and logical anomaly detection rely on searching and comparing image features stored in the memory bank with those of the test image. 2) High-dimensional image features are often extracted to improve logical anomaly detection capabilities. This leads to poor performance in logical anomalies because most image feature channels are unrelated to logical anomalies.

In this paper, instead of an elaborately designed image feature memory bank, we introduce a text memory bank to enhance the detection of logical anomalies, as shown in Fig. 1(b). Text, as the primary medium of natural language, is a expression form rich in logical information, effectively encoding object categories, counts, locations, sizes, and combination relationships. Compared to images, text can present information in a more concise and structured manner, filtering out a large amount of redundant content unrelated to logic, which is crucial for improving the performance of logical anomaly detection.

Specifically, we propose a Three-Memory framework for Unified structural and logical Anomaly Detection (TMUAD), consisting of class-level text memory bank, object-level image memory bank, and patch-level image memory bank. We first propose a logic-aware text extractor to capture rich logical descriptions of all objects from input images, containing object categories, counts, locations, sizes, and combination relationships. These text descriptions compose a text memory bank for logical anomaly detection. We then construct an object-level image memory bank by extracting image features of segmented objects to retain complete object contours. Finally, we use visual encoders to extract patch-level image features for structural anomaly detection. These three complementary memory banks are used to retrieve and compare normal images that are most similar to the query image, compute individual anomaly scores, and fuse them into a final anomaly detection score. Our framework unifies structural anomalies and logical anomalies using three collaborative memory banks, enabling more accurate logical anomalies detection in object counts and locations through explicit text descriptions. The contributions of this paper are summarized as follows:

We propose a unified structural and logical anomaly detection framework, consisting of class-level text, object-level image, and patch-level image memory bank. Our class-level text memory bank can significantly enhance the logical anomaly detection performance. 2. 2.

We propose a text-based anomaly detection pipeline, including a plug-and-play logic-aware text extractor to capture rich logical information between objects for logical anomaly detection, and a compatible anomaly score calculation method for text by reverse-locating images from anomaly-related text. 3. 3.

Our unified anomaly detection framework achieves state-of-the-art performance in logical and structural anomaly detection across seven publicly available datasets involving industrial and medical domains.

II Related work

II-A Memory Bank-based Anomaly Detection

Since anomaly samples are very scarce in real-world environments, many methods focus on achieving unsupervised anomaly detection by constructing memory banks. PatchCore [2] uses patch features extracted by ImageNet pre-trained encoders to construct memory bank, and calculate the similarity between query samples and memory bank features to detect anomalies. PMB [32] combines partition memory bank Module to build storage templates for each channel to retain detailed features. PBAS [21] uses a feature projector layer based on PatchCore [2] to improve anomaly detection accuracy. OPFA [33] combines GMM [34] to model the distribution of memory bank data, thereby identifying abnormal data more effectively. These models perform very well on structural anomaly, but cannot effectively distinguish logical anomaly.

Along with the first logical anomaly dataset MVTec LOCO [6], many methods began to focus on a unified framework for detecting structural anomaly and logical anomalies. GCAD [6] removes noise from causal graphs through simple and effective causal graph sparsification, thereby improving the interpretability of causal relationships in images. UniVAD [30] constructs structured combinatorial relationships between objects by combining component segmentation techniques and graph-enhanced component model to achieve effective determination of combinatorial logic rules. These methods achieve the extraction of logical information effectively by designing specialized components, but the features lack interpretability and have a large amount of redundant information, resulting in limited performance of logical anomaly detection.

II-B Vision Language Models–based Anomaly Detection.

In recent years, Vision Language Models (VLMs)[35, 36, 37, 38, 25, 39, 40, 41] have provided new technological tools for anomaly detection due to their revolutionary reasoning capabilities in various vision tasks. SAM [42] has a powerful zero-sample image segmentation capability, which allows us to segment unseen objects without training. CLIP [43] uses contrastive learning to map images and texts to a unified vector space, supporting tasks such as image classification and retrieval. Qwen2-VL [44] possess advanced question-answering and fine-grained localization capabilities, offering new technical approaches for anomaly detection tasks.

AnomalyGPT [35] freezes the main parameters of the language model and adds a prompt learner to it, which embeds image features and text features into the original input latent space of the visual language model, and achieves fine-tuning by training the parameters of the cue embedding network. FocusCLIP [36] uses the anomaly capture module to refine these defect regions, improving its ability to identify detailed anomalies. AA-CLIP [38] achieves fine-tuning by introducing a residual adapter layer to the CLIP, giving the model the ability to distinguish between normal and abnormal. LogicQA [40] guides the VLMs through the designed prompts to summarize the patterns of the samples and to detect anomalies. LogSAD [25] employs GPT-4 to automatically induce rules and utilizes a matching mechanism to detect anomalies by comparing object types and verifying potential rule violations. Although this method incorporates textual information to assess the consistency of object categories, it still relies primarily on image features rather than textual descriptions when processing positional relationships, thereby limiting the model’s overall performance. Moreover, this approach lacks the ability to achieve pixel-level precise localization of detected text anomalies. To address these limitations, this paper proposes enhancing the extraction of textual information and introducing a text-query-based localization strategy to improve both detection accuracy and localization effectiveness.

III Method

III-A Overview

Our goal is to develop a multi-memory framework to unify logical and structural anomaly detection. For structural anomaly detection, current patch-based methods[2, 23, 30, 6] perform well but exhibit limitations in effectively detecting logical anomalies. To address these limitations, we propose three complementary strategies: 1) retain patch-level image features to construct a patch-level image memory bank $M_{P}$, thereby preserving the model’s capability to detect structural anomalies, 2) introduce a logic-aware textual description of images to significantly enhance logical anomalies detection of unified anomaly detection frameworks by constructing a class-level text memory bank $M_{T}$, and 3) introduce object-level image features from segmented images to build an object-level image memory bank $M_{O}$, which compensates for the inability of patch-level features to fully capture the shape and structure of individual objects. As shown in Fig. 2, Our TMUAD framework consists of memory bank construction in Sec. III-B and anomaly score calculation in Sec. III-C.

III-B Memory Bank Construction

Our TMUAD takes a normal image $I_{n}^{i}$ as input and extracts corresponding text descriptions $T_{img}^{i}$, object-level image features $F_{O}^{i}$, and patch-level image features $F_{P}^{i}$ to construct three distinct memory banks: $M_{T}$, $M_{O}$, and $M_{P}$. The class-level text memory bank $M_{T}$ contains information such as object categories, counts, locations, and sizes. By capturing the attributes of each object category, $M_{T}$ effectively represents the logical relationships between object categories, making it suitable for detecting logical anomalies, including missing objects, incorrect object combinations, or abnormal spatial distributions. The object-level image memory bank $M_{O}$ contains the visual features of individual objects, capturing their complete structural characteristics. The patch-level image memory bank $M_{P}$ stores multi-scale local visual features, enabling the representation of fine-grained details across the whole image. In addition, we use the K-means algorithm to cluster the features of $M_{O}$ and $M_{P}$ to reduce storage space and improve retrieval efficiency [30].

III-B1 Class-level Text Memory Bank

We construct a class-level text memory bank $M_{T}:=\{T_{img}^{i}\}_{i=1}^{n_{i}}$ from normal images $I_{n}^{i}$ in the training set, where $T_{img}^{i}$ represents text descriptions of $I_{n}^{i}$ for logical anomaly detection, $n_{i}$ is the total number of normal images. $T_{img}^{i}:=\{T_{class}^{j}\}_{j=1}^{n_{j}}$ comprises the textual descriptions of different object categories $T_{class}^{j}$ in $I_{n}^{i}$, where $T_{class}^{j}:=\{O_{class}^{j},O_{num}^{j},O_{pos}^{j},O_{size}^{j}\}$ contains the object category $O_{class}^{j}$, number $O_{num}^{j}$, position $O_{pos}^{j}$, size $O_{size}^{j}$. $n_{j}$ is the number of object categories in $I_{n}^{i}$.

As shown in Fig. 3 and Algorithm 1, we propose a logic-aware text extractor to capture rich logical information between objects for logical anomaly detection.

1. To obtain object categories $O_{class}^{j}$, we first use Grounded SAM [45] to segment a normal image $I_{n}^{i}$ into different objects $I_{seg}^{k}$, i.e., $I_{seg}^{k}=I_{m}^{k}\odot I_{n}^{i}$, $\{I_{m}^{k}\}=\mathrm{GSAM}(I_{n}^{i})$, where $n_{k}$ is the number of segmented objects in the normal image $I_{n}^{i}$ by Grounded SAM [45]. Then, we use CLIP [43] to obtain the categories $C_{k}$ of the segmented objects, i.e., $C_{k}=\mathrm{CLIP}(I_{seg}^{k},C)$, where $C$ is a list of predefined object categories.
2. To obtain object counts $O_{num}^{j}$, we merge the segmented images $I_{seg}^{k}$ of the same category and record their counts.
3. To obtain object position $O_{pos}^{j}$, we use VLM (i.e., Qwen2-VL [44]) to roughly describe the position of objects with prompts containing object categories, $\{O_{pos}^{j}\}=\mathrm{VLM}(\{O_{class}^{j}\},I_{n}^{i})$.
4. To obtain object size $O_{size}^{j}$, we calculate the area percentage of each segmented object relative to the whole normal image $I_{n}^{i}$, i.e., $\{O_{size}^{j}\}=\mathrm{Area}(\{I_{class}^{j}\})$, where $I_{class}^{j}$ is the merged mask of all objects belonging to category $O_{class}^{j}$. Unlike employing VLMs to generate a single textual description for the entire image [35, 46, 47, 48], which is difficult to accurately describe the number, size, and combination of objects, we propose logic-aware text descriptions based on segmented images, facilitating logical anomaly detection such as incorrect object combinations, incorrect positions, and missing objects.

III-B2 Object-level Image Memory Bank

We extract the visual features $F_{seg}^{s}$ of each segmented object $I_{seg}^{s}$ from the normal image $I_{n}^{i}$ by the pre-trained SAM [42] and the image encoder of CLIP [43], i.e., $\{I_{seg}^{s}\}_{s=1}^{n_{s}}=\mathrm{SAM}(I_{n}^{i})$, $F_{seg}^{s}=E_{clip}(I_{seg}^{s})$, where $n_{s}$ is the number of segmented objects in the normal image $I_{n}^{i}$ by SAM [42]. The object-level image memory bank can be denoted as $M_{o}:=\ {\textstyle\bigcup_{i=1}^{n_{i}}}F_{O}^{i}$, where $F_{O}^{i}=\left\{F_{seg}^{s}\right\}_{s=1}^{n_{s}}$ is the object-level image features of $I_{n}^{i}$. This memory bank offers several advantages. 1) By utilizing segmented objects, it effectively eliminates background interference since anomalies typically occur within the foreground objects of images. 2) In contrast to the class-level text memory bank or the patch-level image memory bank, the object-level image memory bank enables the retrieval and comparison of similar objects across different images.

III-B3 Patch-level Image Memory Bank

We employ CLIP [43] and DINOv2 [49] as vision encoders to extract multi-scale visual features $F_{P}^{i}$ from normal images $I_{n}^{i}$, i.e., $F_{P}^{i}=\{E_{clip}(I_{n}^{i}),E_{dino}(I_{n}^{i})\}$. These multi-scale features $F_{P}^{i}$ are obtained from multiple intermediate layers of the encoder: shallow layers primarily capture low-level visual cues such as edges, textures, and color, whereas deeper layers encode high-level semantic representations and global contextual information [50]. The patch-level image memory bank can be denoted as $M_{P}:=\left\{F_{P}^{i}\right\}_{i=1}^{n_{i}}$.

III-C Anomaly Score Calculation

Our TMUAD inputs a query image $I_{q}$, extracts corresponding text descriptions ($\hat{T}_{img}$), object-level image features $\hat{F}_{O}$, and patch-level image features $\hat{F}_{P}$, searches and compares the most similar features in the three memory banks, calculates three anomaly scores for each memory bank (class-level text anomaly score $S_{T}$, object-level image anomaly score $S_{O}$, and patch-level image anomaly score $S_{P}$), and finally fuses these anomaly scores for anomaly detection (i.e., ${S}_{pixel}$ in Fig. 4).

III-C1 Class-level Text Anomaly Score

Current anomaly detection methods [38, 12, 35] typically compute pixel-level anomaly scores to localize anomalous regions, while image-level anomaly scores are used to determine the presence of anomalies. The image-level anomaly scores can be derived from pixel-level scores, for example by taking their maximum or average value [30, 7, 21]. However, when anomaly detection is performed with text, only image-level anomaly scores are readily available, whereas pixel-level scores are challenging to obtain [46, 25, 40]. To address this limitation, we propose a pixel-level calculation method for class-level text anomaly score by reverse-locating images from anomaly-related text. This method is compatible with widely adopted anomaly score calculation pipelines, enabling the computation of both image-level and pixel-level anomaly scores using a text-based memory bank.

As shown in Fig. 5 and Algorithm 2, we first search the class-level text memory bank $M_{T}$ for the text $T_{sim}$ that is most similar to the text description $\hat{T}_{img}$ of the query image $I_{q}$ [55]. We then compute anomaly scores by comparing each object categories in $T_{sim}=\{O_{class}^{t},O_{num}^{t},O_{pos}^{t},O_{size}^{t}\}_{t=1}^{n_{t}}$ with those in $\hat{T}_{img}:=\{O_{class}^{\hat{j}},O_{num}^{\hat{j}},O_{pos}^{\hat{j}},O_{size}^{\hat{j}}\}_{{\hat{j}}=1}^{\hat{n}_{j}}$, where $n_{t}$ and $\hat{n}_{j}$ denote the total number of object categories in $T_{sim}$ and $\hat{T}_{img}$, respectively. For anomalies in object category $O_{class}^{\hat{j}}$, count $O_{num}^{\hat{j}}$, and location $O_{pos}^{\hat{j}}$, we use the merged mask $I_{class}^{\hat{j}}$ of all objects belonging to category $O_{class}^{\hat{j}}$ as the pixel-level anomaly score, i.e., $S_{T}=\mathrm{max}(S_{T},I_{class}^{\hat{j}})$, where $\mathrm{max}(\cdot)$ denotes a pixel-wise maximum operator. $S_{T}$ is initialized as a zero image with the same width and height as $I_{q}$. For abnormal in object sizes $O_{size}^{\hat{j}}$, we calculate a coefficient $\alpha$ that quantifies the deviation of $O_{size}^{\hat{j}}$ from the normal size range $\left[s_{min}^{\hat{j}},s_{max}^{\hat{j}}\right]$, i.e., $S_{T}=\mathrm{max}(S_{T},\alpha\cdot I_{class}^{\hat{j}})$. $s_{min}^{\hat{j}}$ and $s_{max}^{\hat{j}}$ denote the maximum and minimum sizes of the object category $O_{class}^{\hat{j}}$ in the memory bank $M_{T}$.

[TABLE]

[TABLE]

where $T_{img}^{i}=\{O_{class}^{j},O_{num}^{j},O_{pos}^{j},O_{size}^{j}\}_{j=1}^{n_{j}}$. To detect the missing object categories $O_{class}^{t}$ in the query image $I_{q}$, we record all possible locations where the categories $O_{class}^{t}$ could appear, denoted as $I_{c}$ in Algorithm 1. $I_{c}$ are used to compute the pixel-level anomaly score corresponding to the missing object categories, i.e., $S_{T}=\mathrm{max}(S_{T},I_{c})$.

III-C2 Object-level Image Anomaly Score

We use SAM [42] to segment the query image $I_{q}$ into different objects $\hat{I}_{seg}^{\hat{s}}$ and use CLIP [43] to extract the object-level image features $\hat{F}_{O}=\{\hat{F}_{seg}^{\hat{s}}\}_{\hat{s}=1}^{\hat{n}_{s}}$, $\hat{F}_{seg}^{\hat{s}}=E_{clip}(\hat{I}_{seg}^{\hat{s}})$, where $\hat{n}_{s}$ is the number of segmented objects in the query image $I_{q}$ by SAM [42]. The object-level image anomaly score $S_{O}$ can be calculated by

[TABLE]

III-C3 Patch-level Anomaly Score

We employ CLIP [43] and DINOv2 [49] to extract patch-level image features $\hat{F}_{P}$ from the query image $I_{q}$. The patch-level anomaly score $S_{P}$ can be calculated by

[TABLE]

where $\mathrm{upsample}(\cdot)$ denotes upsampling the patch feature map to the resolution of the query image $I_{q}$.

III-C4 Anomaly Score Fusion

Considering that $M_{P}$ and $M_{O}$ focus on structural anomaly detection, while $M_{T}$ are designed to detect logical anomalies, we perform a weighted fusion of $S_{T},S_{I},S_{P}$ to derive the pixel-level anomaly score map,

[TABLE]

The fusion weights are empirically set as $\lambda_{1}=0.05$, $\lambda_{2}=0.3$, and $\lambda_{3}=0.65$. Finally, we take the maximum value of ${S}_{pixel}$ as the image-level anomaly score ${S}_{image}$.

IV Experiments

IV-A Dataset setting and Evaluation Metrics

We evaluated our method on two anomaly detection tasks, including logical anomaly detection and structural anomaly detection on the widely used 7 anomaly detection datasets in industrial and medical domains. The area under the receiver operating characteristic curve (AUROC) is employed as the primary evaluation metric. For all datasets, the training set contains only normal samples, while the test set contains both normal and abnormal samples. Representative examples are shown in Fig. 6, where abnormal regions are marked with red bounding boxes.

For industrial logical anomaly detection and structural anomaly detection, we use the MVTec LOCO dataset [6], containing 1772 normal samples for training and 1568 samples (575 normal and 993 anomalous sample) for testing.

For the industrial structural anomaly detection, we employ three widely used datasets: MVTec AD [4], MPDD [51], and VisA [5]. These datasets primarily consist of structural anomalies such as scratches, dents, contaminations, and various structural changes. MVTec AD simulates real industrial production scenarios, containing 3629 normal samples for training and 1725 samples (467 normal and 1258 anomalous sample) for testing across 15 categories. MPDD focuses on metal part defect detection, with a training set of 888 normal samples and a test set comprising 176 normal and 281 anomalous samples. VisA introduces objects with complex structures and variations in pose or position, comprising 9621 normal samples for training and 1200 anomalous samples for test.

To evaluate the generalization ability of the proposed framework, we conducted experiments on three medical anomaly detection datasets: BrainMRI [52], LiverCT [53], and RESC [54]. The BrainMRI dataset is a human brain tumor segmentation benchmark commonly used in medical image analysis. It contains a training set with 7500 normal samples and a test set with 640 normal and 3,075 abnormal samples. The LiverCT dataset consists of abdominal CT scans. Its training set includes 1542 normal samples, while the test set comprises 833 normal and 660 abnormal samples. The RESC dataset targets ophthalmic disease detection. It provides 4,297 normal training samples and a test set with 1,041 normal and 764 abnormal samples.

IV-B Implementation Details

Before constructing the three memory banks, we employ Grounded SAM [45] to detect and segment objects in normal images. Grounded SAM integrates Grounding DINO [65] for object category detection and SAM [42] for object segmentation. The segmented objects from seven datasets are then organized by category, and both the categorized segmented images (from SAM [42]) and their corresponding text labels (from Grounding DINO [65]) are used to fine-tune CLIP [43] (ViT-L/14). In this fine-tuning process, 200 images per category are used to enhance the classification accuracy of our logic-aware text extractor.

Except for the CLIP model in the text extractor, which uses our fine-tuned weights, all other models use publicly available pre-trained weights, including VLM [44] in Fig. 3, as well as CLIP [43], DINOv2 [49], Grounding SAM [45], and SAM [42] in Fig. 2. For all datasets, input images for all models are uniformly resized to 448×448 pixels. The number of clusters in the K-means algorithm is set to 100 for each image category to build patch-level image memory bank $M_{P}$ and is set to 1000 for each object category to build object-level image memory bank $M_{O}$.

IV-C Comparison with State-of-the-art Methods

For fair comparisons, we retrained UniVAD [30] on full data and PBAS [21] using publicly available code on seven datasets containing industrial anomalies [6, 4, 51, 5] and medical anomalies [52, 53, 54], while the results of other methods are provided from the original papers as much as possible. Except for “TMUAD-I” and “PBAS+$M_{T}$” in Table I, II, and III, which use CLIP trained individually on each dataset, we use CLIP with the same weight in the text extractor is applied across all datasets to achieve unified logical and structural anomaly detection.

For unified logical anomaly detection and structural anomaly detection, we compare our TMUAD with the following methods on the MVTec LOCO [6] dataset: PatchCore [2], Dream [57], PBAS [21], UniVAD [30], GCAD [6], GeneralAD [59], OmniAL [58], LogicAL [56], LogSAD [25] in full-data evaluation results. Table I shows that our method substantially outperforms state-of-the-art methods in terms of image-level AUROC (94.6). Unlike existing methods, which exhibit large performance fluctuations across categories, our method maintains balanced performance across different image categories, demonstrating greater robustness in anomaly detection. Table IV further confirms that our method achieves the SOTA image-level AUROC for both logical and structural anomaly detection. Fig. 7 illustrates that TMUAD accurately localizes anomaly regions with minimal noise, consistently across both logical and structural anomalies.

For structural anomaly detection, we compare our TMUAD with the following method on 3 industrial structural datasets (MVTec AD [4] in Fig. 8, VisA [5] in Fig. 9, and MPDD [51] in Fig. 10) and 3 medical datasets (BrainMRI [52] in Fig. 11, RESC [54], and LiverCT [53]): WinCLIP [60], VAND [61], AnomalyGPT [35], AnomalyCLIP [62], MVFA-AD [63], AdaCLIP [64], AA-CLIP [38], and UniVAD [30]. We retrained UniVAD [30] on full data for fair comparisons. Table II and Table III show the image-level and pixel-level AUROC metrics for different datasets, respectively.

Table I, II, and III demonstrate that state-of-the-art anomaly detection performance can be achieved using unified weights. Furthermore, by applying dataset-specific weight of CLIP in the text extractor, we can obtain state-of-the-art results within a unified framework. For example, “PBAS+$M_{T}$” preserves its strong structural anomaly detection capability while simultaneously achieving competitive performance on logical anomaly detection.

IV-D Ablation Studies

We conducted ablation studies of our memory banks on the MVTec LOCO [6] dataset, as shown in Table V and Fig. 12. The image-level and pixel-level AUROC shown in Table V are calculated using $M_{T},M_{O},M_{P}$. We make three key observations:

1. $M_{O}$ is most effective for image-level AUROC in detecting structural abnormalities;
2. $M_{P}$ achieves the highest pixel-level AUROC for structural abnormalities;
3. $M_{T}$ is most effective for detecting logical abnormalities; 4) and unified detection of logical and structural anomalies requires integrating all three memory banks. These results align with the original motivation for constructing multiple complementary memory banks in Sec. III-B.

We conducted ablation studies on our class-level text memory bank $M_{T}$ using the MVTec LOCO dataset [6]. The image-level AUROC shown in Table VI are calculated using $M_{T}$ only. The first and second row represent directly inputting entire images into VLMs [44] to generate textual descriptions, with anomaly scores computed via cosine similarity between textual features encoded by the T5 encoder [67] and the CLIP text encoder [43]. Compared to the CLIP encoder, which primarily captures global text features of the entire image, the T5 encoder extracts richer semantic information, thereby improving logical anomaly detection. The third row represents that images are first segmented with Grounded SAM [45], followed by VLM-based textual descriptions of each segmented object. These descriptions are concatenated and encoded with T5 encoder [67] to compute similarity and anomaly scores. This validates the necessity of first segmenting and then describing the image. In the fourth row, the “VLMs $\to O_{pos}^{j}\to T_{img}^{j}$” indicates that Algorithm 1 is used to obtain class-level text descriptions. Using a T5 encoder to calculate text similarity and derive anomaly scores may lack sensitivity to the number, location, and size of objects, which are critical for logical anomaly detection. To address this limitation, our proposed class-level text matching strategy in Algorithm 2 significantly enhance the performance of logical anomaly detection, as shown in the fifth row of Table VI. In addition, we conducted an ablation study on the MVTec LOCO dataset [6] to evaluate object localization using vision-language models (VLMs), as shown in Fig. 13. Unlike the nine-grid method that estimates an object’s location by calculating its geometric center, VLMs leverage strong prior knowledge and semantic understanding to approximate object positions. This approach reduces errors that commonly arise when objects lie near equidistant boundaries. Notably, our proposed class-level text memory bank can be seamlessly integrated into other methods in a plug-and-play manner, e.g., UniVAD [30] for unified logical and structural anomaly detection, PBAS [21] for structural anomaly detection. Table VII demonstrate that incorporating our text memory bank substantially enhances the performance of logical anomaly detection. As shown in Fig. 14, we present the results of applying class-level textual memory bank to detect anomalies in various scenarios of the MVTec LOCO dataset [6]. The results demonstrate that our logic-aware textual descriptions effectively identify anomalies related to object location, number, size, missing, and other logical inconsistencies.

We conducted ablation experiments on the segmentation model, as shown in Fig. 15. The third and fourth columns present the results of SAM segmentation [42] and Grounding SAM segmentation [45], respectively. For logical anomaly detection (first row), Grounding SAM [45] produces semantically more complete segmented objects compared with SAM [42]. It effectively excludes most background regions unrelated to logical anomalies, enabling our TMUAD to focus more directly on the relevant objects. For structural anomaly detection (second row), SAM [42] outperforms Grounding SAM [45] by capturing finer object details and comprehensively segmenting image regions that may contain anomalies. This detailed representation of SAM [42] is particularly beneficial for detecting structural anomalies.

We conducted ablation experiments on object-level image memory banks $M_{O}$ on the MVTec LOCO [6] dataset. The image-level AUROC shown in Table VIII are calculated using $M_{O}$ only. Table VIII indicate that the optimal configuration involves using SAM [42] for segmentation, extracting object-level features with the CLIP image encoder [43], computing text similarity, and deriving anomaly scores.

We conducted ablation experiments on the patch-level image memory bank $M_{P}$ on the MVTec LOCO [6] dataset. The image-level AUROC shown in Table IX are calculated using $M_{P}$ only. Table IX shows that combining DINOv2 [49] and CLIP encoders [43] yields superior performance compared to using either encoder alone. Fig. 16 further illustrates their complementary strengths: CLIP emphasizes global differences in foreground objects, which is advantageous for detecting anomalies in large objects, whereas DINO captures fine-grained pixel-level differences, enabling more accurate detection of small-object anomalies. This dual-encoder framework achieves effective complementarity in structural anomaly detection: 1) DINO suppresses noise in non-anomalous regions (first row), while 2) CLIP enhances the detection of large-object anomalies (second row).

V Discussion and Limitation

V-A Limitations

Our method sometimes fails to detect logical anomalies on the MVTec LOCO [6] dataset, as shown in Fig. 17. In the first row, although the number of pins and cells is consistent, one cell contains two pins. This anomaly is challenging to identify because cells are generally treated as background; incorporating background anomaly detection could improve performance in such cases. In the second row, the wires are misaligned when connecting to the ports. This issue may be mitigated by incorporating the concept of object contact detection [68, 69].

V-B Few-shot Anomaly Detection

We evaluated few-shot anomaly detection on the MVTec LOCO [6] dataset, as shown in Table X. The results indicate that our TMUAD (AUROC: 81.2) performs worse than LogSAD [25] (AUROC: 86.3) in few-shot settings but TMUAD (AUROC: 94.6) outperforms LogSAD [25] (AUROC: 90.2) in full data (Table I). In addition, our TMUAD with a relaxed class-level text matching strategy (i.e., w/o $O_{num}^{j},O_{pos}^{j},O_{size}^{j}$) (AUROC: 85.3) demonstrates competitive performance with LogSAD [25] (AUROC: 86.3) in few-shot scenarios. We analyze that the reason may be that our strict text matching strategy is highly effective when sufficient normal samples are available, but overly restrictive in settings with limited data, leading to performance degradation in few-shot scenarios.

V-C Resource Efficiency Analysis

Fig. 18 shows that: 1) the AUROC metric improves as the number of clusters increases, but the gain saturates around 100 clusters; and 2) when the number of clusters exceeds 100, the average inference time rises sharply. Table XI shows that performance and efficiency comparisons between our TMUAD under various settings and the SOTA methods. We constructed our base model, TMUAD-U-B, using Qwen2-VL-7B with 100 K-means clusters in the memory bank $M_{P}$. We further introduce two variants: TMUAD-U-L, built on Qwen2-VL-7B with 300 clusters, and TMUAD-U-S, built on w/o Qwen2-VL-7B and with 25 clusters. Additionally, we report the inference time of TMUAD-U-B when accelerated by three A6000 GPUs in parallel. Except TMUAD-U-B (3 GPUs), all experiments in this paper are conducted on a single A6000 GPU. In future work, we will accelerate anomaly detection by applying product quantization [70] during the search for similar samples.

VI Conclusion

The proposed TMUAD is a memory-based anomaly detection framework that unifies logical and structural anomaly detection by leveraging multiple memory banks, consisting of class-level text, object-level image, and patch-level image memory bank. Our class-level text memory bank enhances logical anomaly detection by leveraging logic-aware textual descriptions, while our image memory bank is sensitive to structural anomalies through object-level features that preserve contours and patch-level features that capture global contextual information. Our text memory bank is plug-and-play for structural anomaly detection methods, enabling their extension into a unified framework for both logical and structural anomaly detection. Our future work will focus on accelerating visual anomaly detection models and deploying them to robots, designing more effective feature representations, and incorporating object contact detection to improve anomaly detection.