Under review

Prompt2Fly

An LLM and Vision-Language Behavior Tree Framework for Autonomous Aerial Transportation

Miguel Gil-Castilla*, Mrunal Sarvaiya*, and Giuseppe Loianno

* Equal contribution

Paper — coming soon Video — coming soon Code — coming soon
01 / Tomato sauce can

“Pick up the tomato sauce can.”

Behavior Front VLM Action

Agentic Robotics Framework with Shared Autonomy
01 / Abstract

Aerial robots are versatile platforms for efficient package transportation in safety-critical scenarios, but reliable long-horizon mission execution requires effective collaboration with human supervisors. Enabling shared autonomy for these aerial systems requires natural-language communication, interpretable task decomposition, and explicit mechanisms for human oversight of safety-critical decisions. To address these challenges, we present Prompt2Fly, a hierarchical framework that combines large language models (LLMs) and vision-language models (VLMs) to translate natural-language commands into executable aerial payload-transportation missions using egocentric robot observations. Prompt2Fly employs an LLM to generate reactive behavior trees grounded in validated aerial transportation behaviors, while extracting task-relevant context to condition VLM-based navigation skills. This architecture combines the semantic reasoning and generalization capabilities of pretrained models with the interpretability and modularity of established planning and control algorithms. We demonstrate Prompt2Fly on cable-suspended payload transportation, where the system's underactuated and coupled payload dynamics introduce multiple challenges. We validate the framework through extensive real-world experiments using onboard cameras and a magnetic pickup mechanism. The system identifies and transports payloads in various environments and under different task specifications. To the best of our knowledge, Prompt2Fly is the first system to demonstrate vision-language-based autonomous payload transportation using slung-load aerial robots.

Overview of Prompt2Fly: natural-language input, grounding in robot skills, LLM behavior tree generation, and shared autonomy, with real-world transport of a quadrotor, a yellow cube, and a tomato sauce can.
Fig. 1 (A) Overview of Prompt2Fly. The user provides a natural-language (NL) instruction to the aerial robot. An LLM, grounded in a predefined set of robot capabilities such as visual navigation and position control, parses the instruction and generates an executable Behavior Tree (BT). The BT is then deployed on the robot with a human in the loop, allowing the operator to supervise safety-critical transitions, including payload-pickup/dropoff confirmation and emergency-stop execution. (B–D) Real-world zero-shot payload transportation with Prompt2Fly: We demonstrate generalization across objects with substantially different geometry, appearance, and dynamics by transporting (B) another quadrotor, (C) a yellow cube, and (D) a tomato sauce can.
02 / Contributions

We present Prompt2Fly, an agentic robotics framework with shared autonomy for aerial payload transportation. Our approach combines LLM-based task reasoning, vision-language perception, and reactive behavior trees to enable interpretable, language-guided robot autonomy with human oversight.

  1. 01

    Language-Guided Behavior Tree Generation for Robot Autonomy

    A language-to-behavior-tree framework that uses pretrained LLMs without task-specific fine-tuning to compose aerial transportation behaviors into executable BTs and generate behavior-specific VLM context, supporting real-world deployment.

  2. 02

    VLM-Based Suspended-Payload Transportation Planner

    A vision-language-conditioned planner tailored to aerial transportation systems that leverages VLMs to generate robot commands for the autonomous transportation of random objects from onboard camera observations.

  3. 03

    Shared Autonomy Through Human-In-The-Loop Supervision

    A collaborative human-in-the-loop architecture built on reactive BTs, allowing an operator to respond to safety-critical behavior transitions.

  4. 04

    Real-World Validation

    Extensive real-world experiments, demonstrating end-to-end execution of diverse payload-transportation missions using multiple onboard cameras and a magnetic pickup mechanism. To the best of our knowledge, Prompt2Fly is the first system to demonstrate vision-language-based autonomous transportation tasks with cable-suspended payloads.

03 / System
Quadrotor with cameras, cable, and magnet, next to the four payloads: a tomato sauce can, a quadrotor, a yellow cube, and a red cube.
Fig. 2 Left: Quadrotor with a suspended payload and the magnetic pickup mechanism. Right: The four different payloads transported with our method. Each uses a flat metal plate for magnetic attachment, with varying appearances and optional bullseye markings to aid pickup localization.
04 / Semantic Perception
Two-stage perception pipeline: Grounding DINO initialization at 1 Hz, then CLIPSeg tracking on zoom-cropped frames at 10 Hz.
Fig. 3 Overview of the onboard two-stage semantic perception pipeline. Grounding DINO provides an accurate full-image initialization from the language query, after which CLIPSeg performs high-frequency tracking. The updated detection is recursively used to define the crop for the next frame, reducing the visual search space while maintaining robust target detection.
05 / Real-World Experiments
Operator prompt and eight sequential snapshots of the quadrotor locating the ladder, approaching and picking up the red cube, transporting it, and landing.
Fig. 4 A long-range payload transportation mission demonstrating the zero-shot deployment of Prompt2Fly. The operator prompt, shown on the left, instructs the robot to locate the red box next to the ladder and transport it to the specified destination. Sequential snapshots of the experiment are shown on the right.
LLM-generated behavior tree with a safety branch (emergency, pause) and a nominal mission (take off, pick up, transport to goal, land) with confirmation and contingency nodes.
Fig. 5 LLM-generated BT for a payload transportation mission. The nominal mission executes takeoff, visual payload pickup, transport, drop-off, and landing, with operator confirmation and contingency behaviors at mission-critical transitions. Reactive and resumable control nodes allow execution to transition between nominal, safety, and operator-intervention behaviors. The safety branch is triggered when any subtree in the nominal mission returns a failure.