RUA / Awesome Robot Use AgentResearch blogs, papers, benchmarks, open-source tools, and social demos for robot-use agents.
164 entries · all categories
No matching resources. Try a different keyword or clear the filters.
General-purpose AI agents using robots through sensor and actuator APIs, with discussion of deployment, latency, and reliability. Author post ; discusses agents including Claude.
Compares direct motor commands, generated controllers, pretrained-policy supervision, and RL training across control, locomotion, and manipulation tasks. Author-reported results show that robot embodiment and control interface strongly affect performance. Direct-control simulations pause between model calls; physical Go2 explorations should be distinguished from simulated benchmark results.
Introduces GRID’s agent-driven workflow for robot integration, simulation, skill creation, and deployment evaluation. Demonstrates laboratory manipulation tasks and describes how execution feedback helps agents diagnose failures, repair components, and retain reusable skills and engineering knowledge.
Describes agents that observe camera feeds, generate robot-control programs, and call action models as tools. Presents real-robot demonstrations of task decomposition, outcome verification, replanning, and multi-agent coordination, with reusable skills shared across agents.
Connects Nemotron reasoning and vision models, NeMo Agent Toolkit, and Pipecat to Reachy Mini for voice, camera input, and robot behaviors. Describes ReAct tool calling and Python interfaces for hardware or simulation; focuses on a desktop interactive robot rather than general manipulation.
Describes embodied reasoning for multi-step planning, tool use, success tracking, and multi-robot coordination. ER 2 provides high-level decisions while a connected VLA handles motor execution; the page presents developer-reported capabilities and evaluations.
Paper references are grouped by contribution. Dates refer to first arXiv release unless noted. Method summaries reflect author reports; code, weights, and complete reproduction are separate release claims. Cross-links connect papers with related implementations and benchmarks; the resource count reflects catalogue entries, including these separate listings.
Data for agent action selection, tool use, planning, and execution feedback, with broader robot demonstrations included as supporting resources. Dates follow the associated paper or linked announcement; public downloads, paper-described training data, data directories, and surveys are distinguished below.
This section focuses on benchmark papers and their research contributions; the Benchmarks section below compares environments, tasks, and evaluation focus to help you choose a benchmark.
Unifies computer-use and robot-use through PAPAV: Perceive, Anticipate, Plan, Act, and Verify; examines physical constraints and benchmark coverage.
VLMs select discrete, incremental action units grounded by robot-specific interpreters.
Keeps the base policy frozen while developing runtime critics and recovery skills through execution, diagnosis, and gated updates.
Keeps model parameters frozen while evolving reusable skills and a context-code harness from target-environment rollouts.
Separates world memory from task memory and grounds the active goal in recalled objects, state, and evidence before each action.
A planner selects one tool call at a time, executes it through a common interface, and uses fresh observations and results to revise plans and retain experience.
Wraps robot capabilities as callable tools, maintains symbolic scene context, and evaluates action termination, success, and failure causes.
An LLM controller coordinates training, evaluation, and policy-improvement workers with recorded tool calls, evidence-gated promotion, and recoverable execution state.
A closed-loop orchestrator decomposes goals, selects frozen VLA policies or parameterized skills, verifies outcomes from observations, and recovers from failures.
Uses execution memory to route among heterogeneous policies and a Memory Bridge to improve handoffs between policies.
Separates cognitive planning from physical execution through session state, explicit task verification, and reusable experience.
Uses test-time training fast weights to condition robot policies on long visuomotor histories, supporting adaptation from demonstrations and execution context.
Exposes a frozen VLA as a retryable contact-rich primitive, coordinated with fixed analytic primitives and memory of execution outcomes.
Compiles language instructions into executable graphs of modular robot skills, then uses simulation feedback to localize failures and refine the graph.
Refines code-as-policy programs using multimodal execution traces, failure diagnosis, repair validation, a reusable skill library, and evolutionary search.
Connects scene reset, policy execution, outcome verification, and experiment refinement so coding agents can improve policies through physical trials.
Proposes its own practice tasks, writes and executes robot code, diagnoses failures, and distills successful behavior into reusable skills before downstream tasks arrive.
Studies iterative perception–reasoning–action, semantic action abstractions, and multimodal observations; also describes distillation into a smaller agent model.
Studies high-level VLM planners and low-level VLA controllers, including subgoal switching, observation representation, and planner memory.
Externalizes embodied capabilities as callable tools through the Embodied Tool Protocol (ETP), with EmbodiedToolBench evaluating tool-need recognition, selection, execution, and composition.
Reflects on execution trajectories to distinguish defective skills from execution lapses, then revises skill content or reinforces valid guidance.
Improves generalist robot policies using fleet-scale deployment experience, offline–online reinforcement learning, and human interventions.
VAP-TAMP combines VLM-guided active view selection and situation assessment with scene graphs and task-and-motion planning to handle execution-time disruptions.
Studies embodied coding agents through CaP-Gym, CaP-Bench, CaP-Agent0, and CaP-RL, including execution feedback and skill synthesis.
A VLM controller coordinates data collection, policy learning, and long-horizon execution; forward and inverse action pairs enable self-resetting collection.
Combines online action-model adaptation with LLM-guided graph-based replanning to respond to navigation failures and changing conditions.
Uses real rollouts to improve an action-conditioned world model, then synthetic rollouts to improve the VLA policy in an iterative loop.
Broader multimodal-agent survey covering reasoning, reflection, memory, tool invocation, and interaction with physical environments, including embodied AI applications.
Combines demonstration learning, learned success and progress estimates, and autonomous reinforcement-learning practice to improve robot policies.
Classifies how LLM/VLM agents integrate with robot APIs, ROS middleware, and orchestration frameworks; covers planning, tool calling, agent roles, and practical robotics toolkits.
Long-horizon simulation demonstrations with subtask annotations, disturbances, and memory-dependent tasks.
A high-level VLM interprets complex instructions and situated user feedback, while a low-level VLA executes the selected subtask.
Vision-driven embodied-agent evaluation; see the benchmark catalogue .
Generates visual monitoring code to evaluate spatio-temporal constraints, detecting failures after they occur and anticipating constraint violations.
LLM agents read URDF descriptions and call kinematics tools to build robot capability profiles, then coordinate heterogeneous robots for planning and execution.
Evaluation of embodied decision-making modules; see the benchmark catalogue .
Produces natural-language failure diagnoses that can refine rewards, task plans, and subtask verification in robot manipulation systems.
Converts language instructions and RGB-D observations into Python functions over 3D keypoints; hierarchical optimization solves end-effector actions in a perception-action loop.
LLM integration across communication, perception, planning, and control.
Uses generated code and visual grounding to construct 3D value maps for motion planning; the generated program can be reevaluated with visual feedback.
Searches hierarchical 3D scene graphs, calls a classical path planner, and uses scene-graph simulator feedback to repair infeasible long-horizon plans.
Robot agents negotiate subtasks through dialogue and generate task-space waypoints, revising plans using collision checks and other environment feedback.
KnowNo uses conformal prediction to calibrate a language-model planner’s uncertainty and request human clarification when action choices remain ambiguous.
An LLM generates task plans and execution constraints, while a VLM continuously checks violations to trigger recovery when execution diverges from the plan.
Summarizes multisensory execution history, explains failures, and conditions a planner on those explanations to produce corrective actions.
Trains a visual goal-conditioned generalist policy across robot embodiments and incorporates self-generated experience into subsequent training.
Prompts an LLM with available actions, scene objects, and example programs to generate executable task plans grounded in robot capabilities and context.
Generates programs that compose perception outputs, control APIs, and feedback loops.
Feeds success detection, scene descriptions, and human feedback into language-based planning; demonstrates replanning and responses to changed goals.
Combines language-model skill scoring with affordance/value estimates to select feasible robot behaviors.
Code links and usage restrictions are recorded per entry. Research papers and their code links are listed together in Papers; this section focuses on standalone tools, frameworks, environments, and demos. Check each repository, model, and dataset license before reuse.
Feedback / workflow: Mission graphs pass step outputs and support failure branches.
Setup: ROS 2 capabilities and mission definitions.
Limits: The built-in natural-language mission compiler is rule-based, not an LLM planner.
Feedback / workflow: Replay, simulation, and hardware workflows.
Setup: Dependencies for the selected workflow and hardware integration.
Limits: Pre-release Beta; individual hardware integrations range from stable to experimental.
Feedback / workflow: Model, memory, and component events.
Setup: Relevant robot components for motion.
Limits: The quickstart demonstrates visual question answering; motion requires robot-specific components.
Feedback / workflow: Simulation integrations and rai_bench.
Setup: Robot-specific tools and configuration.
Limits: The README lists rai_finetune as unfinished.
Feedback / workflow: Uses execution traces and memory-guided planning to compose manipulation primitives; implements Harness VLA.
Setup: Linux, an NVIDIA GPU, a configured planner, model checkpoints, and environment assets.
Environments: LIBERO-PRO, RoboCasa, and RoboTwin; real-robot extensions have separate hardware setup instructions.
Feedback / workflow: Policy execution, recording, and training integrations.
Setup: MuJoCo by default; real hardware is opt-in.
Limits: Simulator asset coverage does not establish equivalent hardware support.
These provide tasks, execution environments, and scene reconstruction workflows for agent research. Visual replay and contact simulation are distinguished per entry. An environment's inclusion does not mean its standard protocol evaluates tool use, streaming interaction, interruption, or recovery.
Reusable tools and libraries for perception, grasp generation, motion planning, execution, and robot learning. Connect them through robot APIs to build an observation–action–feedback loop; each entry describes its capabilities and integration requirements.
Perception and spatial understanding
Grasp generation
Motion planning and control
Execution infrastructure
Supporting policies and learning infrastructure
These resources can supply action models, data workflows, or deployment components. They are listed as foundations rather than complete robot-use agents.
Other Tools
Interfaces and utilities for robot observations, actions, evaluation, execution feedback, and data workflows.
Feedback / workflow: Tool responses; capabilities depend on the configured robot.
Setup: Appropriate tools and context for custom robots.
Limits: Robot-specific task outcomes require the corresponding tools and context.
Feedback / workflow: Robot communication and introspection.
Setup: A connected agent and robot stack.
Limits: Planning and outcome evaluation depend on the connected agent and robot stack.
Feedback / workflow: Runs evaluations and records auditable trial logs; powers StationeryBench.
Setup: Matching policy, task, and robot adapters. The linked GPT-6 Astra report uses version 0.58.0 on YAM arms.
Limits: Alpha software. See the trial results and limitations.
Feedback / workflow: Parallel episode evaluation with task metrics, recordings, and saved configurations; supports synchronous and live execution.
Setup: A compatible model checkpoint and benchmark configuration, with the required compute and container runtime.
Limits: Model–benchmark compatibility and reproduced results vary by integration; the leaderboard also aggregates published results.
Feedback / workflow: Records success, task scores, and action corrections across 50 aligned cases per policy.
Setup: Isaac Sim 5.1, RoboDojo, and OpenPI; simulator assets and model checkpoints are obtained separately.
Limits: Official-model comparisons use reweighted public results, not same-seed reruns.
Feedback / workflow: JSON responses for topic, service, node, parameter, and action commands.
Setup: A configured robot and rosbridge.
Limits: Planning and outcome interpretation remain with the calling agent.
Feedback / workflow: Organizes demonstration data and connects reconstruction, retargeting, and policy-training stages.
Setup: GPU-enabled Docker, model weights, robot assets, and source datasets for the selected workflow.
Limits: Hardware compatibility varies by module. Its ingestion agent organizes demonstration data; it is not itself an online robot-control agent.
Selected X / Twitter and RedNote posts about GPT-6 Astra, with dates in UTC+8. Updated through 2026-09-14: 43 entries, with related follow-ups grouped together. Related articles are listed under Articles. Task labels describe the demonstrated activity, not task success or success rates; general manipulation is used when the exact task is unspecified. Results below are author-reported unless stated otherwise.
Real-robot demonstrations
Video unavailable here. Use the original post below.
Tasks: Pick & place, Articulated-object manipulation, Pouring & pipetting.
More: Benchmark catalogue · Evaluation report
Video unavailable here. Use the original post below.
Follow-up: Post
Video unavailable here. Use the original post below.
More: Related post
Video unavailable here. Use the original post below.
Tasks: Insertion, Stacking & rearrangement.
Video unavailable here. Use the original post below.
More: Author profile
Video unavailable here. Use the original post below.
More: Zhang post · ENPIRE
Video unavailable here. Use the original post below.
More: Related post
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
More: Control details
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
More: Author clarification
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Tasks: Pick & place, Stacking & rearrangement.
move_to. Medium reasoning, a 20-LLM-call budget, and a 25% speed cap. Video unavailable here. Use the original post below.
Tasks: Pick & place, Insertion.
Simulation demonstrations
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Follow-up: Post
Tasks: Mobile manipulation, Pick & place.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Follow-ups: Post 1 · Post 2 · Post 3 · Post 4 · Post 5 · Post 6
Tasks: Articulated-object manipulation, Pouring & pipetting.
Video unavailable here. Use the original post below.
Tasks: Locomotion & balance, Robot & simulator design.
More: Related post
Video unavailable here. Use the original post below.
More: Related post
Video unavailable here. Use the original post below.
Tasks: Navigation & driving, Perception & calibration, Reconstruction & retargeting.
More: Evaluation report
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Tasks: Navigation & driving, General manipulation.
More: Paper · Project · Evaluation logs
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
More: Related post
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Tasks: Mobile manipulation, Pick & place.
Video unavailable here. Use the original post below.
Perception and reconstruction
These are supporting capabilities for robot-use workflows, rather than direct evidence of a complete robot-control agent.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Earlier version (2026-09-10): constructs a simplified 1X-hand mechanism from video, with illustrative cable deformation; full tendon physics remains future work. Tasks include reconstruction, retargeting, and simulator design. Post · Video · Preview
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Tasks: Reconstruction & retargeting, Perception & calibration.
Video unavailable here. Use the original post below.
Datasets and resource directories for robot-use agents.
Five desk-stationery tasks built on Inspect Robots: uncap a marker, retrieve an eraser, extract a sticky pad, pour paper clips, and hand over a ruler.
The report presents progress scores from 0 to 100 in 25-point increments; progress is distinct from complete-task success. Grading is human and non-blind, with manual resets and different control budgets for the two models. The bundled mock has no physics and does not evaluate manipulation ability.
Related: Inspect Robots · Community report
Evaluates agent-written code for drone reconstruction, localization, navigation, detection, and following.
Uses ten runs per model and up to ten submissions per run, selecting the best. Tasks receive baseline upstream outputs, and the human baseline uses coding assistance. Multiplying per-task probabilities does not measure a continuous end-to-end flight.
Related: Original post · Community entry
Evaluates generalist manipulation policies on 42 simulation and 18 real-world tasks. Simulation covers generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following.
Provides parallel simulation and a standardized real-world evaluation system with scene reset and remote access. Policy integration uses XPolicyLab; official leaderboard submissions follow the cloud evaluation protocol.
Tracks coding agents’ policy-improvement progress over wall-clock time on Push-T and Pin Insertion, including heuristic and gradient-based learning.
Evaluates the policy research-and-improvement loop. Comparisons depend on robot, compute, token, and retry budgets.
Related: Related method
Evaluates reasoning, creative physical-tool use, and adaptation to unexpected geometric, material, and assembly constraints.
Compares robot policies, pretrained VLAs, and oracle-state planners. Physical-tool use here concerns manipulating objects as tools, not software API calls. Results depend on the evaluated task subset and access to privileged state.
Tests how agents choose and sequence perception, locomotion, and manipulation to gather evidence about hidden spatial and physical properties.
Covers 10 categories and 29 subcategories. Active exploration produces answers to spatial questions; answer accuracy is distinct from manipulation-task success. Passive single-view data is a separate evaluation condition.
Evaluates robot-control coding agents across API abstraction levels, single-/multi-turn interaction, and visual grounding
Preview: Figure 1 of the paper.
Evaluates planning, reflection, and memory in long-horizon manipulation through interaction between a high-level VLM planner and a low-level VLA controller.
Includes annotated simulation trajectories and structured high-level/low-level evaluation. Scores depend on the controller and planner interface; this is not direct motor-command generation by the high-level model.
Related: Dataset entry
Vision-driven embodied agents across high- and low-level tasks
EB-ALFRED and EB-Habitat emphasize high-level task planning; EB-Navigation and EB-Manipulation require finer-grained actions and spatial reasoning. Report the environment and action interface alongside scores.
Provides 100 task categories spanning implicit instructions, spatial reasoning, knowledge transfer, and long-horizon skill composition.
Includes 60 primitive and 40 composite task categories. Evaluates both action policies and language-model capabilities; distinguish VLA execution, VLM-based agent workflows, and reasoning-only protocols.
Evaluates planning, tool and skill selection, task tracking, and coordination under spatial, temporal, and heterogeneous-agent constraints.
Includes 100,000 natural-language tasks across 60 houses. Released planners use perception and action tools backed by low-level skills. Report the perception, skill, and coordination configuration; the benchmark runs in simulation.
Goal interpretation, subgoal decomposition, action sequencing, and transition modeling
Useful for identifying decision-making errors; symbolic module evaluation should be distinguished from end-to-end physical execution.
Benchmarks code generation for free-form manipulation instructions using perception and motion-planning APIs shared across simulation and robots.
Integrates ROS, Gazebo, perception, and motion planning; studies physical constraints and module-level failures. Includes simulation benchmark evaluations and separate real-robot demonstrations on Franka and UR5 arms.
Automatically evaluates language-based household task planners, including execution feedback and replanning after action failures.
Uses ALFRED with AI2-THOR and WAH-NL with VirtualHome. Focuses on high-level task planning, with task sources shared with ALFRED.
Evaluates generated Python programs that compose mobility, perception, and human-interaction skills for service robots.
Contains 16 tasks with five prompt paraphrases each. Checks execution traces from multiple initial states against temporal-logic specifications. The benchmark measures program correctness in a symbolic simulator; CodeBotler robot demonstrations are separate deployment evidence.
Six collaborative manipulation tasks testing robot coordination, subtask planning, and task-space waypoint generation with environment feedback.
RoCo revises plans using feedback including collision checks; a separate motion planner executes the waypoints. The text-only reasoning dataset and real-world demonstrations are distinct from simulation benchmark runs.
Related: Related method
Evaluates household task execution from dialogue and interaction history, including collaboration between a Commander and a Follower.
Distinguishes Execution from Dialogue History, Trajectory from Dialogue, and Two-Agent Task Completion; the first two use recorded dialogue, while the third models both agents.
Aligns text-based household tasks with ALFRED environments to study high-level planning and transfer to embodied execution.
Offers text, visual, and hybrid environments. Results depend on the environment and observation/action interface; text-only runs evaluate language-based task execution.
Maps natural-language instructions and egocentric observations to navigation and interaction actions for multi-step household tasks.
Includes non-reversible object-state changes. Evaluates visual task execution in simulation, not real-robot motor control. ALFWorld provides aligned text environments for these tasks.
This collection takes inspiration from Awesome Computer Use Agent in its curation, organization, and presentation, and from Phillip Isola’s Robot-Use Agents blog in its focus on agents that interact with the physical world. We thank the authors and maintainers of the resources featured here for sharing their research, code, and demonstrations, and everyone who contributes suggestions and corrections.
@misc{awesome-robot-use-agent,
title = {Awesome Robot Use Agent},
author = {Kairun Wen and {Awesome Robot Use Agent Contributors}},
journal = {GitHub repository},
url = {https://github.com/kairunwen/Awesome-Robot-Use-Agent},
year = {2026},
note = {A curated collection of resources for robot-use agents.}
}
If you find this collection helpful for your work, please consider giving it a ⭐ Star.
This site uses Google Analytics to measure visits. How Google uses data.
Total views · —