Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search

TL;DR

This paper quantifies safety risks in AI-powered search engines using LLMs, revealing frequent harmful content and malicious URL citations, and proposes a defense mechanism to mitigate these risks effectively.

Contribution

It provides the first systematic safety risk assessment of production AIPSEs and introduces an agent-based defense with GPT-4.1 for risk mitigation.

Findings

AIPSEs often generate harmful content with malicious URLs even for benign queries.

Querying URLs increases risk responses; natural language queries slightly reduce risk.

The proposed GPT-4.1-based defense reduces risks with minimal information loss (~10.7%).

Abstract

Recent advancements in Large Language Models (LLMs) have significantly enhanced the capabilities of AI-Powered Search Engines (AIPSEs), offering precise and efficient responses by integrating external databases with pre-existing knowledge. However, we observe that these AIPSEs raise risks such as quoting malicious content or citing malicious websites, leading to harmful or unverified information dissemination. In this study, we conduct the first safety risk quantification on seven production AIPSEs by systematically defining the threat model, risk type, and evaluating responses to various query types. With data collected from PhishTank, ThreatBook, and LevelBlue, our findings reveal that AIPSEs frequently generate harmful content that contains malicious URLs even with benign queries (e.g., with benign keywords). We also observe that directly querying a URL will increase the number of main risk-inclusive responses, while querying with natural language will slightly mitigate such risk. Compared to traditional search engines, AIPSEs outperform in both utility and safety. We further perform two case studies on online document spoofing and phishing to show the ease of deceiving AIPSEs in the real-world setting. To mitigate these risks, we develop an agent-based defense with a GPT-4.1-based content refinement tool and a URL detector. Our evaluation shows that our defense can effectively reduce the risk, with only a minor cost of reducing available information by approximately 10.7%. Our research highlights the urgent need for robust safety measures in AIPSEs.