UrzaGPT: LoRA-Tuned Large Language Models for Card Selection in Collectible Card Games

TL;DR

UrzaGPT is a domain-adapted large language model fine-tuned with Low-Rank Adaptation to improve real-time card drafting decisions in Magic: The Gathering, demonstrating promising accuracy and adaptability over zero-shot models.

Contribution

This work introduces UrzaGPT, the first LLM fine-tuned for CCG drafting, showing how LLMs can be adapted for complex, evolving game decision tasks.

Findings

UrzaGPT achieves 66.2% accuracy after 10,000 fine-tuning steps.

Large LLMs like GPT-4o perform better than smaller models in zero-shot drafting.

Fine-tuning LLMs makes them viable for real-time CCG drafting tasks.

Abstract

Collectible card games (CCGs) are a difficult genre for AI due to their partial observability, long-term decision-making, and evolving card sets. Due to this, current AI models perform vastly worse than human players at CCG tasks such as deckbuilding and gameplay. In this work, we introduce UrzaGPT, a domain-adapted large language model that recommends real-time drafting decisions in Magic: The Gathering. Starting from an open-weight LLM, we use Low-Rank Adaptation fine-tuning on a dataset of annotated draft logs. With this, we leverage the language modeling capabilities of LLM, and can quickly adapt to different expansions of the game. We benchmark UrzaGPT in comparison to zero-shot LLMs and the state-of-the-art domain-specific model. Untuned, small LLMs like Llama-3-8B are completely unable to draft, but the larger GPT-4o achieves a zero-shot performance of 43%. Using UrzaGPT to fine-tune smaller models, we achieve an accuracy of 66.2% using only 10,000 steps. Despite this not reaching the capability of domain-specific models, we show that solely using LLMs to draft is possible and conclude that using LLMs can enable performant, general, and update-friendly drafting AIs in the future.