AI Competency: Current State and Challenges

TL;DR

This paper explores the evolving concept of AI competency in medical education, emphasizing the need to focus on choosing the right AI tools and addressing related challenges.

Contribution

Introduces a hierarchical framework of AI competency domains and highlights the need for new performance indicators in clinical AI use.

Findings

Optimizing AI tools' output may yield diminishing returns as tools improve.

A hierarchical framework of cognitive, operational, and meta-AI domains is proposed for AI competency.

Developing performance indicators for AI tools in clinical contexts poses significant challenges.

Abstract

As artificial intelligence (AI) develops, the medical education community has begun defining the relevant forms of competency. Many experts emphasize the importance of optimizing AI tools’ output or understanding the relevant technical and normative considerations around using AI tools. A recent publication in this journal showed that optimizing instructions for large language models may yield diminishing returns as such tools improve. This suggests the need for a new competency—one that focuses on choosing the appropriate AI tools. I briefly summarize the current competency domains and examples to contextualize the current state of AI competency development, highlighting the need for further synthesis. I then introduce a hierarchical framework of competencies that might assist with priority-setting around subsequent competency development work. It consists of cognitive, operational, and meta-AI domains, which respectively correspond with the knowledge around understanding, using, and choosing AI tools. The final section describes the potential challenges associated with the development of AI competency. These include traditional concerns around competency-based medical education: deciding whether and which competencies are meaningful for measuring the targets of interest; adjusting the relevant measurements to reflect the necessary temporal and institutional context; and setting up the relevant organizational support to encourage measurement of competency. This section also discusses the challenges of developing the relevant performance indicators for AI tools across different clinical contexts. Such indicators will be necessary for guiding the choice of AI tools for the clinical context, but medical educators may not have the skills to develop them. In addition to identifying potential sources for relevant indicators, the medical education community may shape physicians’ norms of practice to drive the AI industry into producing the relevant indicators. The potential for physicians to incur higher medical liability from poor choice of AI may lead them to demand more nuanced performance indicators from AI suppliers. Physicians are also in a position to do so, since the competitive AI market may provide them more bargaining power.

Figures