Autonomous Computer Vision Development with Agentic AI

TL;DR

This paper demonstrates that an agentic AI system can autonomously interpret a computer vision task, plan the workflow, configure tools, and achieve high accuracy in medical image segmentation without human intervention.

Contribution

It introduces a method for autonomous computer vision system development using agentic AI with LLMs to plan and configure workflows from natural language prompts.

Findings

Successfully generated and executed a computer vision workflow from a natural language prompt.

Achieved high mean dice scores of 0.96, 0.82, 0.83 on chest X-ray segmentation tasks.

Showed potential for autonomous development of computer vision applications.

Abstract

Agentic Artificial Intelligence (AI) systems leveraging Large Language Models (LLMs) exhibit significant potential for complex reasoning, planning, and tool utilization. We demonstrate that a specialized computer vision system can be built autonomously from a natural language prompt using Agentic AI methods. This involved extending SimpleMind (SM), an open-source Cognitive AI environment with configurable tools for medical image analysis, with an LLM-based agent, implemented using OpenManus, to automate the planning (tool configuration) for a particular computer vision task. We provide a proof-of-concept demonstration that an agentic system can interpret a computer vision task prompt, plan a corresponding SimpleMind workflow by decomposing the task and configuring appropriate tools. From the user input prompt, "provide sm (SimpleMind) config for lungs, heart, and ribs segmentation for cxr (chest x-ray)"), the agent LLM was able to generate the plan (tool configuration file in YAML format), and execute SM-Learn (training) and SM-Think (inference) scripts autonomously. The computer vision agent automatically configured, trained, and tested itself on 50 chest x-ray images, achieving mean dice scores of 0.96, 0.82, 0.83, for lungs, heart, and ribs, respectively. This work shows the potential for autonomous planning and tool configuration that has traditionally been performed by a data scientist in the development of computer vision applications.