Fighting AI with AI: AI-Agent Augmented DNS Blocking of LLM Services during Student Evaluations

TL;DR

This paper presents AI-Sinkhole, an AI-agent framework that detects and blocks LLM chatbot services during exams to preserve academic integrity, using explainable classification and DNS-based blocking.

Contribution

The paper introduces AI-Sinkhole, a novel AI-agent system that dynamically identifies and blocks emerging LLM services during exams, with explainable classification and open-source code.

Findings

LLMs can be effectively classified across languages with F1 > 0.83

AI-Sinkhole successfully detects and blocks LLM services during proctored exams

Explainable classification enhances trust and robustness in detection

Abstract

The transformative potential of large language models (LLMs) in education, such as improving accessibility and personalized learning, is being eclipsed by significant challenges. These challenges stem from concerns that LLMs undermine academic assessment by enabling bypassing of critical thinking, leading to increased cognitive offloading. This emerging trend stresses the dual imperative of harnessing AI's educational benefits while safeguarding critical thinking and academic rigor in the evolving AI ecosystem. To this end, we introduce AI-Sinkhole, an AI-agent augmented DNS-based framework that dynamically discovers, semantically classifies, and temporarily network-wide blocks emerging LLM chatbot services during proctored exams. AI-Sinkhole offers explainable classification via quantized LLMs (LLama 3, DeepSeek-R1, Qwen-3) and dynamic DNS blocking with Pi-Hole. We also share our observations in using LLMs as explainable classifiers which achieved robust cross-lingual performance (F1-score > 0.83). To support future research and development in this domain initial codes with a readily deployable 'AI-Sinkhole' blockist is available on https://github.com/AIMLEdu/ai-sinkhole.