Pay-Per-Crawl Pricing for AI: The LM-Tree Agent

TL;DR

The paper introduces the LM Tree, an adaptive pricing agent that uses large language models to dynamically segment content and optimize pay-per-crawl revenue for AI content access.

Contribution

It presents a novel LLM-based segmentation approach that automatically discovers content distinctions for pricing, outperforming static and manual categorization methods.

Findings

LM Tree achieves 65% revenue increase over static pricing.

Outperforms publisher’s 8-segment taxonomy by 40%.

Effective in real-world content and query data.

Abstract

As AI systems shift from directing users to content toward consuming it directly, publishers need a new revenue model: charging AI crawlers for content access. This model, called pay-per-crawl, must solve a problem of mechanism selection at scale: content is too heterogeneous for a fixed pricing framework. Different sub-types warrant not only different price levels but different pricing rules based on different unstructured features, and there are too many to enumerate or design by hand. We propose the LM Tree, an adaptive pricing agent that grows a segmentation tree over the content library, using LLMs to discover what distinguishes high-value from low-value items and apply those attributes at scale, from binary purchase feedback alone. We evaluate the LM Tree on real content from a major German technology publisher, using 8,939 articles and 80,451 buyer queries with willingness-to-pay calibrated from actual AI crawler traffic. The LM Tree achieves a 65% revenue gain over a single static price and a 47% gain over two-category pricing, outperforming even the publisher's own 8-segment editorial taxonomy by 40% -- recovering content distinctions the publisher's own categories miss.