Delta-KNN: Improving Demonstration Selection in In-Context Learning for Alzheimer's Disease Detection

TL;DR

This paper introduces Delta-KNN, a novel demonstration selection method that significantly improves in-context learning performance for Alzheimer's Disease detection using large language models, surpassing existing baselines and achieving state-of-the-art results.

Contribution

Delta-KNN is a new demonstration selection strategy that enhances ICL by using a delta score and KNN retrieval, addressing the limitations of similarity-based methods in complex tasks.

Findings

Delta-KNN outperforms existing ICL baselines across datasets and models.

The method achieves state-of-the-art results with Llama-3.1.

Demonstrates the effectiveness of dynamic, relative scoring in demonstration selection.

Abstract

Alzheimer's Disease (AD) is a progressive neurodegenerative disorder that leads to dementia, and early intervention can greatly benefit from analyzing linguistic abnormalities. In this work, we explore the potential of Large Language Models (LLMs) as health assistants for AD diagnosis from patient-generated text using in-context learning (ICL), where tasks are defined through a few input-output examples. Empirical results reveal that conventional ICL methods, such as similarity-based selection, perform poorly for AD diagnosis, likely due to the inherent complexity of this task. To address this, we introduce Delta-KNN, a novel demonstration selection strategy that enhances ICL performance. Our method leverages a delta score to assess the relative gains of each training example, coupled with a KNN-based retriever that dynamically selects optimal "representatives" for a given input. Experiments on two AD detection datasets across three open-source LLMs demonstrate that Delta-KNN consistently outperforms existing ICL baselines. Notably, when using the Llama-3.1 model, our approach achieves new state-of-the-art results, surpassing even supervised classifiers.