FlexiFly: Interfacing the Physical World with Foundation Models Empowered by Reconfigurable Drone Systems

TL;DR

FlexiFly is a reconfigurable drone platform that enhances foundation models' ability to sense, analyze, and act in physical environments by enabling targeted, high-granularity interactions, significantly improving task success rates.

Contribution

The paper introduces a novel reconfigurable drone platform and segmentation technique that allow foundation models to better interface with the physical world by focusing on relevant areas.

Findings

Enables FMs to complete tasks up to 85% more successfully.

Introduces a modular drone system for targeted sensing and actuation.

Demonstrates effectiveness in real smart home deployments.

Abstract

Foundation models (FM) have shown immense human-like capabilities for generating digital media. However, foundation models that can freely sense, interact, and actuate the physical domain is far from being realized. This is due to 1) requiring dense deployments of sensors to fully cover and analyze large spaces, while 2) events often being localized to small areas, making it difficult for FMs to pinpoint relevant areas of interest relevant to the current task. We propose FlexiFly, a platform that enables FMs to ``zoom in'' and analyze relevant areas with higher granularity to better understand the physical environment and carry out tasks. FlexiFly accomplishes by introducing 1) a novel image segmentation technique that aids in identifying relevant locations and 2) a modular and reconfigurable sensing and actuation drone platform that FMs can actuate to ``zoom in'' with relevant sensors and actuators. We demonstrate through real smart home deployments that FlexiFly enables FMs and LLMs to complete diverse tasks up to $85\%$ more successfully. FlexiFly is critical step towards FMs and LLMs that can naturally interface with the physical world.