Protecting Small Organizations from AI Bots with Logrip: Hierarchical IP Hashing

TL;DR

This paper presents Logrip, a hierarchical IP hashing method that uses data visualization and statistical analysis of server logs to detect and mitigate AI bot traffic, helping small organizations protect their servers.

Contribution

The paper introduces a novel hierarchical IP hashing technique combined with data visualization to identify automated bot activity in server logs, addressing limitations of traditional detection methods.

Findings

80-95% of traffic from AI crawlers in real-world data

Effective differentiation between human and automated access patterns

Improved filtering of automated traffic to preserve server performance

Abstract

Small organizations, start ups, and self-hosted servers face increasing strain from automated web crawlers and AI bots, whose online presence has increased dramatically in the past few years. Modern bots evade traditional throttling and can degrade server performance through sheer volume even when they are well-behaved. We introduce a novel security approach that leverages data visualization and hierarchical IP hashing to analyze server event logs, distinguishing human users from automated entities based on access patterns. By aggregating IP activity across subnet classes and applying statistical measures, our method detects coordinated bot activity and distributed crawling attacks that conventional tools fail to identify. Using a real world example we estimate that 80 to 95 percent of traffic originates from AI crawlers, underscoring the need for improved filtering mechanisms. Our approach enables small organizations to regulate automated traffic effectively, preserving public access while mitigating performance degradation.