Zero-Shot ATC Coding with Large Language Models for Clinical Assessments

TL;DR

This paper presents a hierarchical, LLM-based approach for automating ATC code assignment in healthcare, achieving high accuracy while preserving data privacy, and demonstrating the effectiveness of smaller models.

Contribution

It introduces a hierarchical information extraction method for ATC coding using open-source LLMs, enabling privacy-preserving automation with competitive accuracy.

Findings

Achieves 78% exact match accuracy with GPT-4o

60% accuracy with Llama 3.1 70B

Fine-tuned smaller models match larger zero-shot models

Abstract

Manual assignment of Anatomical Therapeutic Chemical (ATC) codes to prescription records is a significant bottleneck in healthcare research and operations at Ontario Health and InterRAI Canada, requiring extensive expert time and effort. To automate this process while maintaining data privacy, we develop a practical approach using locally deployable large language models (LLMs). Inspired by recent advances in automatic International Classification of Diseases (ICD) coding, our method frames ATC coding as a hierarchical information extraction task, guiding LLMs through the ATC ontology level by level. We evaluate our approach using GPT-4o as an accuracy ceiling and focus development on open-source Llama models suitable for privacy-sensitive deployment. Testing across Health Canada drug product data, the RABBITS benchmark, and real clinical notes from Ontario Health, our method achieves 78% exact match accuracy with GPT-4o and 60% with Llama 3.1 70B. We investigate knowledge grounding through drug definitions, finding modest improvements in accuracy. Further, we show that fine-tuned Llama 3.1 8B matches zero-shot Llama 3.1 70B accuracy, suggesting that effective ATC coding is feasible with smaller models. Our results demonstrate the feasibility of automatic ATC coding in privacy-sensitive healthcare environments, providing a foundation for future deployments.