Developer Insights into Designing AI-Based Computer Perception Tools

TL;DR

This study explores how developers of AI-based computer perception tools balance clinical utility, user trust, and ethical considerations, emphasizing design priorities like explainability, workflow integration, and stakeholder customization.

Contribution

It provides insights from interviews with developers, highlighting key design priorities and ethical considerations for integrating AI tools into clinical practice.

Findings

Developers prioritize explainability and context-awareness.

Alignment with clinical workflows is essential.

Ethical stewardship influences design choices.

Abstract

Artificial intelligence (AI)-based computer perception (CP) technologies use mobile sensors to collect behavioral and physiological data for clinical decision-making. These tools can reshape how clinical knowledge is generated and interpreted. However, effective integration of these tools into clinical workflows depends on how developers balance clinical utility with user acceptability and trustworthiness. Our study presents findings from 20 in-depth interviews with developers of AI-based CP tools. Interviews were transcribed and inductive, thematic analysis was performed to identify 4 key design priorities: 1) to account for context and ensure explainability for both patients and clinicians; 2) align tools with existing clinical workflows; 3) appropriately customize to relevant stakeholders for usability and acceptability; and 4) push the boundaries of innovation while aligning with established paradigms. Our findings highlight that developers view themselves as not merely technical architects but also ethical stewards, designing tools that are both acceptable by users and epistemically responsible (prioritizing objectivity and pushing clinical knowledge forward). We offer the following suggestions to help achieve this balance: documenting how design choices around customization are made, defining limits for customization choices, transparently conveying information about outputs, and investing in user training. Achieving these goals will require interdisciplinary collaboration between developers, clinicians, and ethicists.