RUA / Awesome Robot Use AgentResearch blogs, papers, benchmarks, open-source tools, and social demos for robot-use agents.
322 entries · all categories
No matching resources. Try a different keyword or clear the filters.
General-purpose AI agents using robots through sensor and actuator APIs, with discussion of deployment, latency, and reliability. Author post ; discusses agents including Claude.
Compares direct motor commands, generated controllers, pretrained-policy supervision, and RL training across control, locomotion, and manipulation tasks. Author-reported results show that robot embodiment and control interface strongly affect performance. Direct-control simulations pause between model calls; physical Go2 explorations should be distinguished from simulated benchmark results.
Introduces GRID’s agent-driven workflow for robot integration, simulation, skill creation, and deployment evaluation. Demonstrates laboratory manipulation tasks and describes how execution feedback helps agents diagnose failures, repair components, and retain reusable skills and engineering knowledge.
Describes agents that observe camera feeds, generate robot-control programs, and call action models as tools. Presents real-robot demonstrations of task decomposition, outcome verification, replanning, and multi-agent coordination, with reusable skills shared across agents.
Connects Nemotron reasoning and vision models, NeMo Agent Toolkit, and Pipecat to Reachy Mini for voice, camera input, and robot behaviors. Describes ReAct tool calling and Python interfaces for hardware or simulation; focuses on a desktop interactive robot rather than general manipulation.
Describes embodied reasoning for multi-step planning, tool use, success tracking, and multi-robot coordination. ER 2 provides high-level decisions while a connected VLA handles motor execution; the page presents developer-reported capabilities and evaluations.
RoboDojo’s report on evaluating GPT-6 Astra for robot manipulation.
Argues that coding agents can bridge model-based engineering and robot learning by writing, testing, and refining modular robot programs offline. Discusses GaP, inverse physics, and real-to-simulation-to-real development, alongside remaining challenges in simulation accuracy, verification, and generalization.
Paper references are grouped by contribution. Dates refer to first arXiv release unless noted. Method summaries reflect author reports; code, weights, and complete reproduction are separate release claims. Cross-links connect papers with related implementations and benchmarks; the resource count reflects catalogue entries, including these separate listings.
Data for agent action selection, tool use, planning, and execution feedback, with broader robot demonstrations included as supporting resources. Dates follow the associated paper or linked announcement; public downloads, paper-described training data, data directories, and surveys are distinguished below.
This section focuses on benchmark papers and their research contributions; the Benchmarks section below compares environments, tasks, and evaluation focus to help you choose a benchmark.
Reconstructs interactive digital twins from robot videos and recorded trajectories by refining executable world programs with tool orchestration and verification feedback.
Equips coding agents with a brick-assembly API and validation feedback to generate physically feasible LEGO builds from text or image prompts.
Schedules language-model reviews of a frozen VLA policy using accumulated execution experience, retaining a recovery path for unfamiliar or failed actions.
Unifies computer-use and robot-use through PAPAV: Perceive, Anticipate, Plan, Act, and Verify; examines physical constraints and benchmark coverage.
Lets a frozen multimodal model compose executable robot skills, hierarchical memory, and execution feedback in changing physical and social environments.
Wraps a fixed VLA executor with an editable planning harness that manages task history and selects improvements from rollout feedback.
Converts streaming visual observations into semantic events and an executable subtask list that guides a low-level VLA policy.
Jointly generates skill programs, future video, and robot actions within a learned world action model, connecting reusable code with predictive control.
Evaluates frontier models as closed-loop LIBERO-10 controllers through visual observations, robot state, and end-effector pose and gripper tool calls.
Turns human and robot manipulation videos into timestamped action instructions, process rewards, and outcome scores for data curation, robot learning, and execution monitoring.
Uses one compact high-level model to orchestrate manipulation and navigation skills, track progress, and recover from execution failures.
Post-trains a robot harness's planner with supervised fine-tuning and multi-turn reinforcement learning while keeping its execution components fixed.
Reviews physical AI through a perception–modeling–action loop, covering world models, embodied foundation models, robot control, and deployment challenges.
Introduces PhysRSI-Eval to study how public model capabilities and early robot feedback relate to the speed, extent, and generalization of self-improvement.
Evaluates long-horizon physical, strategic, and spatial reasoning through robot interaction across manipulation, puzzles, and household activities.
Uses a strong model to write, diagnose, repair, and qualify robot skills offline, then reuses the frozen files as code alone or through a cheaper caller.
Compares nine frontier multimodal models controlling physical robot arms on six manipulation tasks under multiple reasoning and service settings.
Gives a coding agent a thin robot CLI and calibrated RGB observations, while a second agent turns failed episodes into reusable code tools.
Composes robot capabilities into collaboration behavior trees and retains feedback-driven revisions, handoff checks, and shared multimodal memory for later tasks.
Revises executable multi-robot coordination over fixed motor skills using transition verification, checkpoint recovery, and reusable cooperative experience.
Builds task-specific laboratory digital twins, robot skills, and experimental workflows from scientific goals, refining them through simulation and physical feedback.
Uses a coding agent to design and refine spatial memory construction and retrieval for robot localization, navigation, and question answering.
Augments a frozen VLA policy with persistent user and scene memory, geometric action shielding, and recovery skills refined through reflection for personalized robot assistance.
Organizes research on agentic robots into four domains—Model, Data, Environment, and Harness—with a 17-category taxonomy and research agenda.
Composes reusable manipulation units, selects compatible policies from a capability registry, and uses visual verification to advance, retry, or switch executors after local failures.
Uses a learned critic to decide when a fixed VLA should continue or recover, retracting along its visited trajectory to a promising state before resuming execution.
Augments a fixed subtask VLA with demonstration-based planning guidance, adaptive visual memory, and geometric corrections for mobile-base positioning and precise manipulation.
Turns interaction feedback in a Playground scene into reusable code Skills and diagnostic Lessons, then freezes them for fresh agents to solve unseen scenes without model-weight updates.
Groups robot-tool primitives into reusable action chunks, reducing repeated LLM deliberation while returning control to the agent when execution encounters a failure.
Coordinates ordered subtasks, visual grounding, and working memory around an adapted action policy, using a separately trained progress evaluator to manage subtask transitions.
Studies whether 3D Gaussian Splatting reconstructions preserve real-world VLA policy rankings under dual fisheye wrist views.
Studies whether robot agents retain useful skills and memories beyond extra retries, and whether improvements transfer across tasks, simulators, and real robots.
Uses a frozen frontier VLM as a direct waypoint-and-gripper policy, keeping the full episode in context for memory-dependent manipulation.
Benchmarks navigation that requires deciding whether to open doors or containers, with an agent that verifies interactions and updates a persistent graph.
Benchmarks how action, perception, and experience skills change agents' manipulation success and execution cost across 100 perturbed LIBERO-Pro tasks.
Unifies navigation, instruction generation, and spatial understanding in one VLM; filters generated instructions by navigation success and path efficiency, then updates the policy after each trajectory shard with spatial-data replay.
Refines agent harnesses through contrastive Propose–Probe–Compose trials without ground-truth feedback, then retains adopted harnesses and reusable lessons for future tasks while keeping model weights fixed.
Combines onboard language-model task decisions with behavior-tree skills that enforce local execution conditions and return structured failure feedback.
Generates diagnostic programs that actively change viewpoints and probe physical interactions in simulation to improve aerial manipulation code.
Uses an LLM to write and revise articulated USD asset programs from partial mesh geometry, visual feedback, and physical authoring constraints.
Introduces Code-Only-as-Policy (COAP), a shared stateful code library that improves from execution feedback and controls robot tasks without model inference at test time.
Supplies a frozen multimodal robot policy with complementary virtual camera views from an online scene synchronized with physical execution, improving visibility of task-critical geometry.
Equips a pretrained multimodal model with navigation skills, tools, and task-progress memory, using image-point destinations and execution feedback to handle diverse navigation requests.
Selects informative robot trials to test competing hypotheses and jointly refine executable code, reusable skills, and memory around frozen robot foundation models.
Links task-relevant scene reconstruction with execution-grounded policy development, simulation testing, and transfer to a real robot.
Studies whether multimodal agents that directly control a mobile manipulator can gather hidden task information by searching, inspecting, and testing, then decide when the task is complete.
Studies how multimodal agents combine perception, reasoning, and tool use across robot embodiments, linking task outcomes to feedback use, recovery, and completion assessment.
Couples asset, scene, and task generation through an agentic refinement loop, using simulation feedback to revise scenes and robot plans for scalable demonstration generation.
Co-evolves a VLM orchestrator and robot policy during deployment by curating execution data, selectively fine-tuning skills, and verifying candidate policies before adoption.
Lets agents choose visual targets and poses while a backend handles geometry and control; subgoal-local contexts and a persistent playbook retain execution experience.
Frames video-to-simulation reconstruction as a coding-agent task, testing whether reconstructed worlds reproduce both scene geometry and executable robot interactions.
Gives coding agents terminal access to native ROS 2 interfaces, treating sensor observations as file I/O and robot manipulation as programs developed with execution feedback.
RPG develops reusable robot skills and a shared system prompt through dataset-guided simulation practice, using failure diagnosis and cross-task validation before physical deployment.
PyRUA-Lean composes robot primitives and frozen VLA policies in persistent Python cells, using local checks, retries, and selective observations to reduce repeated model calls.
Uses a coding agent to discover and verify recovery skills in digital twins, then orchestrates task and recovery policies with progress checks during deployment.
Studies how VLM agents turn geometric estimates, spatial reasoning, and interaction grounding into complete robotic tasks across five capability domains.
Couples semantic planning with runtime command grounding, monitoring, refusal, and recovery, using execution evidence and regression checks to validate revisions of reusable robot capabilities.
Connects coding agents to robot sensing, supervised execution, and persistent experience; an optional learned navigation policy can be called as a tool.
Coordinates goal setting, visuomotor execution, outcome verification, and memory agents for navigation, compressing multimodal history at verified goal-completion boundaries.
URAI pairs a programming agent that builds and refines reusable robot tools with an execution agent that selects tools from feedback between complete local motions.
Lets coding agents develop robot tools through simulated experiments, then jointly adapt the tools and simulator from a small number of physical trials.
Studies how rollout batch size and incumbent-versus-candidate selection affect automatic harness evolution for visual-interface robot agents.
Evaluates agents that choose observations and issue native robot commands in an interactive manipulation environment, separating perception, short-horizon execution, and long-horizon composition.
Improves robot manipulation by retrieving and evolving hierarchical task and action knowledge from execution feedback while keeping the base model and executor fixed.
Connects VLM-proposed mid-level motions to deterministic robot control, with asynchronous monitoring, outcome verification, and background memory updates for replanning.
Connects general-purpose LLMs to visual-geometric control tools, using visual clicks, tracked keypoints, and spatial constraints to execute manipulation without robot-policy training.
Converts an egocentric human video and task prompt into dexterous robot training data through simulation reconstruction, trajectory optimization, and a growing library of skills and verifiers.
Distills a strong agent’s robot-intervention experience into a playbook for a lighter agent, then refines that guidance from the recipient’s execution feedback.
Evaluates general-purpose multimodal coding agents on 25 manipulation tasks using a shared robot API and hidden physical-outcome checks.
Turns execution traces into validated updates to context, persistent memory, and hierarchical skills, promoting reusable improvements across tasks and robot embodiments.
Builds reusable robot programs from a single visual human demonstration, inferring task specifications, manipulation primitives, and a simulation environment for iterative verification and refinement.
Coordinates perception, action primitives, and execution checks through a fixed task program, then uses feedback between rollouts to revise code and retain reusable repair skills.
Exposes calibrated depth, visual anchors, spatial measurements, and grasp hypotheses as tools so a VLM can select robot motions from geometric evidence.
Evolves typed robot Policy APIs, an agent harness, and experience memory before deployment, then uses visual ReAct decisions to call the frozen APIs and recover from failures.
Uses coding agents to synthesize robot programs through simulator interaction, then freezes the programs and evaluates their generalization to unseen task instances.
Uses coding agents to solve long-horizon dexterous tasks, then diversifies successful programs into simulation demonstrations for training general robot policies.
Adds obstacle-aware route planning, verification, replanning, and contact-position selection to coding agents performing robot manipulation under collision constraints.
Uses a graph world model to check action preconditions, repair model-detectable errors locally, and reserve LLM replanning for semantic failures during long-horizon tasks.
Uses a general-purpose agent as a real-robot policy that interprets visual observations, writes executable programs, commands motions, and adapts to physical feedback with fixed model parameters.
Keeps task memory in a high-level agent and steers a stateless action model with language and optional 2D manipulation hints for long-horizon execution.
VLMs select discrete, incremental action units grounded by robot-specific interpreters.
Rewrites robot-control code, tools, memory, and skills from execution feedback, evaluates candidate agents, and retains improved harnesses.
Parses indoor video into per-object evidence, generates complete assets, and uses the GizmoAct VLM agent to align them through 3D-editor actions for real-to-sim scene modeling.
Keeps the base policy frozen while developing runtime critics and recovery skills through execution, diagnosis, and gated updates.
Keeps model parameters frozen while evolving reusable skills and a context-code harness from target-environment rollouts.
Separates world memory from task memory and grounds the active goal in recalled objects, state, and evidence before each action.
A planner selects one tool call at a time, executes it through a common interface, and uses fresh observations and results to revise plans and retain experience.
Wraps robot capabilities as callable tools, maintains symbolic scene context, and evaluates action termination, success, and failure causes.
An LLM controller coordinates training, evaluation, and policy-improvement workers with recorded tool calls, evidence-gated promotion, and recoverable execution state.
A closed-loop orchestrator decomposes goals, selects frozen VLA policies or parameterized skills, verifies outcomes from observations, and recovers from failures.
Uses execution memory to route among heterogeneous policies and a Memory Bridge to improve handoffs between policies.
Separates cognitive planning from physical execution through session state, explicit task verification, and reusable experience.
Uses test-time training fast weights to condition robot policies on long visuomotor histories, supporting adaptation from demonstrations and execution context.
Combines RGB-based exploration, MCP robot tools, and interactive intent clarification for open-world mobile manipulation, with a shared interface across simulation and real robots.
Exposes a frozen VLA as a retryable contact-rich primitive, coordinated with fixed analytic primitives and memory of execution outcomes.
Coordinates a task planner, per-robot manipulation agents, and visual verification agents, using local retries or global task reallocation when execution fails.
Compiles language instructions into executable graphs of modular robot skills, then uses simulation feedback to localize failures and refine the graph.
Refines code-as-policy programs using multimodal execution traces, failure diagnosis, repair validation, a reusable skill library, and evolutionary search.
Connects scene reset, policy execution, outcome verification, and experiment refinement so coding agents can improve policies through physical trials.
Proposes its own practice tasks, writes and executes robot code, diagnoses failures, and distills successful behavior into reusable skills before downstream tasks arrive.
Studies iterative perception–reasoning–action, semantic action abstractions, and multimodal observations; also describes distillation into a smaller agent model.
Uses tool-enabled coding agents and execution feedback to search multi-file robot policy repositories offline, then deploys the selected code or optimized agent harness.
Uses a VLM to write iterative Python cells in a persistent kernel, combining perception and geometry tools with intermediate visual feedback for spatial reasoning.
Studies high-level VLM planners and low-level VLA controllers, including subgoal switching, observation representation, and planner memory.
Automates robot reinforcement-learning workflows through specialized agents, persistent artifacts, executable stage gates, and experience-driven parallel trials.
Reconstructs interactive scenes from semantic maps so skill preconditions and an LLM judge can provide feedback for iterative robot-plan refinement before execution.
Externalizes embodied capabilities as callable tools through the Embodied Tool Protocol (ETP), with EmbodiedToolBench evaluating tool-need recognition, selection, execution, and composition.
Reflects on execution trajectories to distinguish defective skills from execution lapses, then revises skill content or reinforces valid guidance.
Improves generalist robot policies using fleet-scale deployment experience, offline–online reinforcement learning, and human interventions.
VAP-TAMP combines VLM-guided active view selection and situation assessment with scene graphs and task-and-motion planning to handle execution-time disruptions.
Studies embodied coding agents through CaP-Gym, CaP-Bench, CaP-Agent0, and CaP-RL, including execution feedback and skill synthesis.
A VLM controller coordinates data collection, policy learning, and long-horizon execution; forward and inverse action pairs enable self-resetting collection.
Connects LLM reasoning to perception, navigation, grasping, and placement tools, with working context and retrieved episodic memory for household manipulation.
Combines online action-model adaptation with LLM-guided graph-based replanning to respond to navigation failures and changing conditions.
Uses real rollouts to improve an action-conditioned world model, then synthetic rollouts to improve the VLA policy in an iterative loop.
Pairs a ReAct task planner with either generated Python code or iterative tool calls, comparing two execution interfaces for language-conditioned robot manipulation.
Uses agents to turn 3D captures into simulation scenes with estimated physics, articulated objects, and object variations for robot-policy training and evaluation.
Broader multimodal-agent survey covering reasoning, reflection, memory, tool invocation, and interaction with physical environments, including embodied AI applications.
Combines demonstration learning, learned success and progress estimates, and autonomous reinforcement-learning practice to improve robot policies.
Classifies how LLM/VLM agents integrate with robot APIs, ROS middleware, and orchestration frameworks; covers planning, tool calling, agent roles, and practical robotics toolkits.
Long-horizon simulation demonstrations with subtask annotations, disturbances, and memory-dependent tasks.
Orchestrates a reasoning planner, a VLA executor, and a temporal visual verifier that chooses continuation or recovery during sequential manipulation.
A high-level VLM interprets complex instructions and situated user feedback, while a low-level VLA executes the selected subtask.
Vision-driven embodied-agent evaluation; see the benchmark catalogue .
Builds digital twins with 3D Gaussian Splatting and physical meshes to train visual navigation and locomotion policies for real-world transfer.
Generates visual monitoring code to evaluate spatio-temporal constraints, detecting failures after they occur and anticipating constraint violations.
LLM agents read URDF descriptions and call kinematics tools to build robot capability profiles, then coordinate heterogeneous robots for planning and execution.
Evaluation of embodied decision-making modules; see the benchmark catalogue .
Produces natural-language failure diagnoses that can refine rewards, task plans, and subtask verification in robot manipulation systems.
Converts language instructions and RGB-D observations into Python functions over 3D keypoints; hierarchical optimization solves end-effector actions in a perception-action loop.
LLM integration across communication, perception, planning, and control.
Improves imitation-learned manipulation policies through reinforcement learning in digital twins reconstructed from the target real-world environment.
Uses an LLM to generate reward code from environment source and task descriptions, then iteratively improves rewards using reinforcement-learning results and reflection.
Uses generated code and visual grounding to construct 3D value maps for motion planning; the generated program can be reevaluated with visual feedback.
Searches hierarchical 3D scene graphs, calls a classical path planner, and uses scene-graph simulator feedback to repair infeasible long-horizon plans.
Robot agents negotiate subtasks through dialogue and generate task-space waypoints, revising plans using collision checks and other environment feedback.
KnowNo uses conformal prediction to calibrate a language-model planner’s uncertainty and request human clarification when action choices remain ambiguous.
An LLM generates task plans and execution constraints, while a VLM continuously checks violations to trigger recovery when execution diverges from the plan.
Summarizes multisensory execution history, explains failures, and conditions a planner on those explanations to produce corrective actions.
Trains a visual goal-conditioned generalist policy across robot embodiments and incorporates self-generated experience into subsequent training.
Prompts an LLM with available actions, scene objects, and example programs to generate executable task plans grounded in robot capabilities and context.
Generates programs that compose perception outputs, control APIs, and feedback loops.
Feeds success detection, scene descriptions, and human feedback into language-based planning; demonstrates replanning and responses to changed goals.
Combines language-model skill scoring with affordance/value estimates to select feasible robot behaviors.
Code links and usage restrictions are recorded per entry. Research papers and their code links are listed together in Papers; this section focuses on standalone tools, frameworks, environments, and demos. Check each repository, model, and dataset license before reuse.
Feedback / workflow: Mission graphs pass step outputs and support failure branches.
Setup: ROS 2 capabilities and mission definitions.
Limits: The built-in natural-language mission compiler is rule-based, not an LLM planner.
Feedback / workflow: Replay, simulation, and hardware workflows.
Setup: Dependencies for the selected workflow and hardware integration.
Limits: Pre-release Beta; individual hardware integrations range from stable to experimental.
Feedback / workflow: Model, memory, and component events.
Setup: Relevant robot components for motion.
Limits: The quickstart demonstrates visual question answering; motion requires robot-specific components.
Feedback / workflow: Each motion returns fresh camera images, measured displacement, remaining error, and a stop reason; tool calls and states support live monitoring and replay.
Setup: An MCP-capable host such as Claude Code or Codex, plus the selected simulator and assets. Remote backends support SSH or trusted TCP.
Limits: Alpha release. LIBERO and RoboDojo are supported; RoboTwin and real SO-101 support are listed as forthcoming. Apache-2.0.
Feedback / workflow: Searches for subjects, adjusts viewpoint and composition, captures photos, and reviews camera feedback.
Setup: Python 3.12+, a vision model API with tool-calling support, and DJI Tello hardware or Webots R2025a.
Evidence: Separate real DJI Tello flight demos and Webots simulation demos.
Limits: The v0.1 release focuses on photography; broader general-purpose capabilities remain roadmap items.
Feedback / workflow: Agents use skills and executable tools to access configured devices through their drivers.
Setup: Node.js 22.19+; default Python tools require uv and Python 3.11+. Supports Linux and macOS.
Evidence: The README documents a RoboEnv/RoboTwin simulator workflow and a separate real-world robot onboarding guide.
Limits: The harness does not supply simulator or hardware drivers; each integration needs its own setup. Kinex v1.0.0 was released on 2026-10-15.
Feedback / workflow: A model reasons, calls a tool, and reads the resulting observation; recorded requests, actions, and receipts support run replay and an evidence viewer.
Setup: A model endpoint and the chosen LIBERO / LIBERO-Pro, RoboTwin 2.0, or RoboDojo environment; full perception and planning stacks require their GPU dependencies.
Limits: Released on 2026-10-15 under Apache-2.0. The public benchmark integrations run in simulation; reported benchmark results are author-provided.
Feedback / workflow: Simulation integrations and rai_bench.
Setup: Robot-specific tools and configuration.
Limits: The README lists rai_finetune as unfinished.
Feedback / workflow: The agent explores from observations, records successful preconditions, actions, and effects, then compiles and tests a controller that runs without the agent.
Setup: Codex CLI or Claude Code with a supported simulation branch or real-robot backend.
Evidence: Includes RoboDojo and BEHAVIOR-1K simulation branches plus Franka FR3 and Rainbow RB-Y1 real-robot branches.
Skill library: Companion Rainbow RB-Y1 and Franka FR3 branches store executable task skills with preconditions, checks, and saved run evidence.
Limits: Each branch has separate setup and evaluation conditions. The companion skill-library repository has no declared license.
Feedback / workflow: Coordinates rollout collection, policy optimization, and evaluation; supports supervised fine-tuning and reinforcement learning for VLA policies.
Setup: A supported accelerator backend, model checkpoints, and environment-specific dependencies.
Environments: Includes LIBERO, ManiSkill, RoboCasa, RoboTwin, and Isaac Lab integrations; real-robot training requires the corresponding hardware setup.
Feedback / workflow: Tracks execution ownership, progress, cancellation, and bounded recovery across skill consumers and backends.
Setup: Build the core and configure a supported backend; documented robot-motion profiles use ROS 2 Humble and Gazebo.
Limits: Early-stage software validated in limited simulation profiles. No physical-robot integration, stable SDK/API/ABI, or hard-stop guarantee. Original material is MIT OR Apache-2.0; third-party notices remain separate.
Feedback / workflow: Agents inspect cameras at program checkpoints and refine or abort execution. Successful procedures become draft skills; promotion requires repeated runs on different layouts and operator approval.
Setup: Claude Code or Codex and uv; the simulated rig supports Linux or Apple-silicon macOS. Hardware operation needs the configured dual SO-101 arms and cameras.
Limits: Early development. The repository provides an author-recorded real-rig demonstration and a simulated rig; support for additional hardware is an architectural goal. Apache-2.0.
Feedback / workflow: Uses execution traces and memory-guided planning to compose manipulation primitives; implements Harness VLA.
Setup: Linux, an NVIDIA GPU, a configured planner, model checkpoints, and environment assets.
Environments: LIBERO-PRO, RoboCasa, and RoboTwin; real-robot extensions have separate hardware setup instructions.
Feedback / workflow: Records execution traces and native task feedback; the RSI workflow revises and retests harness components while keeping model weights fixed.
Setup: Python 3.10–3.12, a vision/tool-calling model endpoint, and benchmark-specific assets; use Linux benchmark images or an existing HTTP benchmark service.
Release: The runtime and LIBERO-family integrations are public. BEHAVIOR-1K, Bench2Dex, and DexJoCo guides were added on 2026-10-20, followed by RoboCasa365 Target50 and SenseNova-RoboRSI-27B-Preview on 2026-10-22. The public 27B model repository contains 12 safetensors weight shards and inference guidance; its technical report remains pending. Benchmark-specific images, experience packages, and upstream assets are separate downloads.
Feedback / workflow: Policy execution, recording, and training integrations.
Setup: MuJoCo by default; real hardware is opt-in.
Limits: Simulator asset coverage does not establish equivalent hardware support.
These provide tasks, execution environments, and scene reconstruction workflows for agent research. Visual replay and contact simulation are distinguished per entry. An environment's inclusion does not mean its standard protocol evaluates tool use, streaming interaction, interruption, or recovery.
U F L sequence. Pen policy execution requires Python 3.11, Isaac Lab 2.3.2, and rsl-rl-lib 3.0.1; each task lists its dependencies.Reusable tools and libraries for perception, grasp generation, motion planning, execution, and robot learning. Connect them through robot APIs to build an observation–action–feedback loop; each entry describes its capabilities and integration requirements.
Perception and spatial understanding
Grasp generation
Motion planning and control
Execution infrastructure
Supporting policies and learning infrastructure
These resources can supply action models, data workflows, or deployment components. They are listed as foundations rather than complete robot-use agents.
Other Tools
Interfaces and utilities for robot observations, actions, evaluation, execution feedback, and data workflows.
Feedback / workflow: Agents build and audit stations, generate controller programs, and use a dedicated hardware-control skill with operator confirmations.
Setup: RoboDK, an agent that reads skills, and Python for bundled scripts; physical operation also requires a matching robot driver.
Limits: Most skills are MIT, but the shape-builder skill is proprietary and the bundled RoboDK SDK is Apache-2.0. Only the real-robot-control skill is intended to move hardware; the others work in simulation.
Feedback / workflow: Tool responses; capabilities depend on the configured robot.
Setup: Appropriate tools and context for custom robots.
Limits: Robot-specific task outcomes require the corresponding tools and context.
Feedback / workflow: Robot communication and introspection.
Setup: A connected agent and robot stack.
Limits: Planning and outcome evaluation depend on the connected agent and robot stack.
Feedback / workflow: Runs evaluations and records auditable trial logs; powers StationeryBench.
Setup: Matching policy, task, and robot adapters. The linked GPT-6 Astra report uses version 0.58.0 on YAM arms.
Limits: Alpha software. See the trial results and limitations.
Feedback / workflow: Parallel episode evaluation with task metrics, recordings, and saved configurations; supports synchronous and live execution.
Setup: A compatible model checkpoint and benchmark configuration, with the required compute and container runtime.
Limits: Model–benchmark compatibility and reproduced results vary by integration; the leaderboard also aggregates published results.
Feedback / workflow: Records success, task scores, and action corrections across 50 aligned cases per policy.
Setup: Isaac Sim 5.1, RoboDojo, and OpenPI; simulator assets and model checkpoints are obtained separately.
Limits: Official-model comparisons use reweighted public results, not same-seed reruns.
Feedback / workflow: JSON responses for topic, service, node, parameter, and action commands.
Setup: A configured robot and rosbridge.
Limits: Planning and outcome interpretation remain with the calling agent.
Feedback / workflow: Reconstructs and augments scenes from human or robot demonstrations, then uses perception, generated Python stages, and execution feedback for manipulation. The project presents simulation skill acquisition and real-robot transfer.
Setup: Blender, MuJoCo, scene assets, model access, and SAM3 / GraspNet services for the agent pipeline.
Code scope: Real2Sim and agent execution are released separately. The current agent release implements the continuous stage loop; persistent cross-task skill storage is outside this release.
Feedback / workflow: Organizes demonstration data and connects reconstruction, retargeting, and policy-training stages.
Setup: GPU-enabled Docker, model weights, robot assets, and source datasets for the selected workflow.
Limits: Hardware compatibility varies by module. Its ingestion agent organizes demonstration data; it is not itself an online robot-control agent.
Selected X / Twitter and RedNote posts about GPT-6 Astra, with dates in UTC+8. Updated through 2026-09-21: 61 entries, with related follow-ups grouped together. Related articles are listed under Articles. Task labels describe the demonstrated activity, not task success or success rates; general manipulation is used when the exact task is unspecified. Results below are author-reported unless stated otherwise.
Real-robot demonstrations
Video unavailable here. Use the original post below.
Tasks: Pick & place, Articulated-object manipulation, Pouring & pipetting.
More: Benchmark catalogue · Evaluation report
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
More: Project
Video unavailable here. Use the original post below.
More: Cost details
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Follow-up: Post
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
More: Related post
Video unavailable here. Use the original post below.
Tasks: Insertion, Stacking & rearrangement.
Video unavailable here. Use the original post below.
More: Author profile
Video unavailable here. Use the original post below.
More: Zhang post · ENPIRE
Video unavailable here. Use the original post below.
More: Related post
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
More: Control details
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
More: Author clarification
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Tasks: Pick & place, Stacking & rearrangement.
move_to. Medium reasoning, a 20-LLM-call budget, and a 25% speed cap. Video unavailable here. Use the original post below.
Tasks: Pick & place, Insertion.
Simulation demonstrations
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Tasks: Insertion.
More: Interactive demos
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Follow-up: Post
Tasks: Mobile manipulation, Pick & place.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
More: Alternate view
Video unavailable here. Use the original post below.
Follow-ups: Post 1 · Post 2 · Post 3 · Post 4 · Post 5 · Post 6
Tasks: Articulated-object manipulation, Pouring & pipetting.
Video unavailable here. Use the original post below.
Tasks: Locomotion & balance, Robot & simulator design.
More: Related post
Video unavailable here. Use the original post below.
More: Related post
Video unavailable here. Use the original post below.
Tasks: Navigation & driving, Perception & calibration, Reconstruction & retargeting.
More: Evaluation report
Video unavailable here. Use the original post below.
More: Alternate view
Video unavailable here. Use the original post below.
More: Project
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Tasks: Navigation & driving, General manipulation.
More: Paper · Project · Evaluation logs
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
More: Related post
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Tasks: Mobile manipulation, Pick & place.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
More: Environment
Perception and reconstruction
These are supporting capabilities for robot-use workflows, rather than direct evidence of a complete robot-control agent.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Earlier version (2026-09-10): constructs a simplified 1X-hand mechanism from video, with illustrative cable deformation; full tendon physics remains future work. Tasks include reconstruction, retargeting, and simulator design. Post · Video · Preview
Video unavailable here. Use the original post below.
More: Interactive scene
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
More: Code
Video unavailable here. Use the original post below.
Video unavailable here. Use the original post below.
Tasks: Reconstruction & retargeting, Perception & calibration.
Video unavailable here. Use the original post below.
Datasets and resource directories for robot-use agents.
Uses paired conditions to test perceptual robustness, disruption adaptation, task updates, human coordination, and intent understanding.
Compares frontier models that directly issue end-effector pose and gripper tool calls from RGB observations and robot proprioception.
Evaluates coding agents on frozen robot tasks across 32 benchmark suites through program execution and isolated verification.
The released core set has 449 tasks; the full collection contains 1,090 tasks with 2,128 task-mode variants. Privileged mode grades submitted trajectories or policies by fresh replay; standard and standard-realtime modes expose images and proprioception and grade the live episode. Results are grouped by matching configurations, with success recorded separately from partial progress.
Measures initial performance, gain timing, best task scores, interaction cost, and recovery from failure during repeated robot attempts.
Evaluates robotic value models on task-state understanding, temporal progress monitoring, failure and recovery reasoning, and value consistency.
Zero-shot and one-shot settings share a held-out test set, with separate rankings and evaluations under embodiment and environment shifts. Training demonstrations are public; organizers retain the private test set and evaluate submitted model services and adapters. Full-shot training data is available, but full-shot results are not reported in this release.
Evaluates long-horizon physical, strategic, and spatial reasoning through robot interaction across manipulation, puzzles, and household activities.
Aggregates solve rates for agent harnesses and VLA policies across robot-use tasks, with 95% Wilson intervals and separate rankings when systems run different task sets.
Evaluates language models in coding-agent harnesses on eight real-world tabletop tasks, using camera observations and an HTTP API to control a robot arm.
Evaluates experience-driven robot-agent improvement across 330 simulated and five physical tasks, comparing code policies, live agents, and agent-trained action models.
Budget-matched restarts and no-library baselines separate retained experience from extra attempts. Transfer tests span new initial states, conditions, tasks, simulators, and physical hardware, with independent judging and resource accounting.
The paper was first posted on arXiv on 2026-10-13. The project website and paper are public; executable code remains forthcoming. The public repository currently contains announcement assets and a release plan.
Tests real-to-sim policy-ranking agreement and sensitivity to 3DGS reconstruction artifacts under dual fisheye wrist views.
Benchmarks navigation that requires deciding whether to open doors or containers, with an agent that verifies interactions and updates a persistent graph.
Measures task success, robot actions, token usage, and wall-clock time under native control, skill-only execution, and selective skill reuse.
Evaluates multimodal agents on 10 tasks where object locations, hidden properties, or unfamiliar mechanisms must be discovered through interaction before the agent submits.
Simulation only. Agents act zero-shot through a common visuomotor interface and decide when to submit. Each model plays 50 held-out instances per task (500 episodes), scored by success rate and separately by mean progress. The release adds 5,000 scripted demonstrations (366 h at 20 Hz) with per-frame subtask captions; a π0.5 policy fine-tuned on them almost never succeeds.
Evaluates multimodal agents on 84 tasks from 20 source projects, with task-specific observation/action interfaces, interaction budgets, and executable success checks.
Physics pauses during reasoning and analysis. Native controllers and assistance vary by task; optional code control is disabled by default. The paper reports one retained outcome per task and model, alongside execution traces and resource use. Deployment requires simulator assets and a Linux/NVIDIA GPU stack.
Provides 20 simulated household tasks and 60 human VR demonstrations per task, comparing learned policies with coding-agent programs through shared observations, actions, and a frozen whole-body controller.
Coding agents receive one demonstration video and a 101k-step development budget. Frozen programs are scored on 100 hidden evaluation episodes; the main comparison covers nine representative tasks. GR00T-WBC handles balance and walking throughout demonstration collection and policy evaluation.
Provides 28 manipulation tasks for controlled studies of behavioral cloning and reinforcement learning, with scripted demonstrations and GPU-accelerated simulation and data generation.
Five task families cover blocks, chambers, switches, Hanoi, and bowling. Gymnasium environments use state observations by default, with optional visual, CPU, and shortened-horizon variants. Demonstrations mimic non-Markovian human behavior; pre-generated datasets cover selected configurations.
Benchmarks 222 reconstruction instances across 39 task families from 189 robot and human videos, measuring geometry, object dynamics, and task functionality.
Submissions run from fresh initialization against hidden object-centric annotations. V2WScore combines build validity, functionality, geometry, and dynamics, then averages over task families; human-assisted references use the same scoring. Evaluation requires the released dataset and matching simulator environments.
Evaluates VLM agents on 1,000 cases from 32 existing sources and GeoProbe, separating geometric precision, functional grounding, and complete task success.
GeoProbe adds 168 cases probing camera motion, object motion, depth, and scale. Source-native observations, helpers, and budgets are retained; interactive evaluations use one reported episode per assignment. Intermediate progress is scored separately from terminal goal completion. Physical-robot transfer is outside the reported evaluation.
Tests observation selection and native robot control across a 30-task primary suite with ten perception, ten short-horizon, and ten long-horizon tasks.
The primary evaluation uses three rollouts per task from the same initial state and a 1,800-second episode budget. Stable success requires all three rollouts to succeed; stable stage completion retains partial progress and is not task success. The paper describes a 200-task pool, while the current code release includes only 30 tasks. RGB, depth, and proprioceptive observation profiles are evaluated separately; external simulator assets are required.
Evaluates general-purpose multimodal coding agents on 25 manipulation tasks using a shared robot API and hidden physical-outcome checks.
Evaluates coding agents on 28 long-horizon tasks across six suites, including assembly, deformables, liquids, and loco-manipulation, with task progress, completion time, and inference cost.
Agents interact through generic scene-inspection and robot-control tools, without task-specific macro primitives. Hidden offline graders score physical progress; partial-credit scores are distinct from full task completion. The Data link contains simulator assets.
Evaluates coding agents as robot learning engineers across 48 tasks in nine families: interactive control, policy development, perception and estimation, and mechanical design.
Agents develop solutions within task-specific time, interaction, and compute budgets. Independent evaluation uses hidden scenes, seeds, or physical conditions. The RLE Index averages task scores within each workflow, then weights the four workflows equally; reported API cost excludes simulation and GPU infrastructure.
Five desk-stationery tasks built on Inspect Robots: uncap a marker, retrieve an eraser, extract a sticky pad, pour paper clips, and hand over a ruler.
The report presents progress scores from 0 to 100 in 25-point increments; progress is distinct from complete-task success. Grading is human and non-blind, with manual resets and different control budgets for the two models. The bundled mock has no physics and does not evaluate manipulation ability.
Related: Inspect Robots · Community report
Evaluates agent-written code for drone reconstruction, localization, navigation, detection, and following.
Uses ten runs per model and up to ten submissions per run, selecting the best. Tasks receive baseline upstream outputs, and the human baseline uses coding assistance. Multiplying per-task probabilities does not measure a continuous end-to-end flight.
Related: Original post · Community entry
Evaluates generalist manipulation policies on 42 simulation and 18 real-world tasks. Simulation covers generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following.
Provides parallel simulation and a standardized real-world evaluation system with scene reset and remote access. Policy integration uses XPolicyLab; official leaderboard submissions follow the cloud evaluation protocol.
Tracks coding agents’ policy-improvement progress over wall-clock time on Push-T and Pin Insertion, including heuristic and gradient-based learning.
Evaluates the policy research-and-improvement loop. Comparisons depend on robot, compute, token, and retry budgets.
Related: Related method
Evaluates reasoning, creative physical-tool use, and adaptation to unexpected geometric, material, and assembly constraints.
Compares robot policies, pretrained VLAs, and oracle-state planners. Physical-tool use here concerns manipulating objects as tools, not software API calls. Results depend on the evaluated task subset and access to privileged state.
Tests how agents choose and sequence perception, locomotion, and manipulation to gather evidence about hidden spatial and physical properties.
Covers 10 categories and 29 subcategories. Active exploration produces answers to spatial questions; answer accuracy is distinct from manipulation-task success. Passive single-view data is a separate evaluation condition.
Evaluates robot-control coding agents across API abstraction levels, single-/multi-turn interaction, and visual grounding
Preview: Figure 1 of the paper.
Evaluates planning, reflection, and memory in long-horizon manipulation through interaction between a high-level VLM planner and a low-level VLA controller.
Includes annotated simulation trajectories and structured high-level/low-level evaluation. Scores depend on the controller and planner interface; this is not direct motor-command generation by the high-level model.
Related: Dataset entry
Vision-driven embodied agents across high- and low-level tasks
EB-ALFRED and EB-Habitat emphasize high-level task planning; EB-Navigation and EB-Manipulation require finer-grained actions and spatial reasoning. Report the environment and action interface alongside scores.
Provides 100 task categories spanning implicit instructions, spatial reasoning, knowledge transfer, and long-horizon skill composition.
Includes 60 primitive and 40 composite task categories. Evaluates both action policies and language-model capabilities; distinguish VLA execution, VLM-based agent workflows, and reasoning-only protocols.
Evaluates planning, tool and skill selection, task tracking, and coordination under spatial, temporal, and heterogeneous-agent constraints.
Includes 100,000 natural-language tasks across 60 houses. Released planners use perception and action tools backed by low-level skills. Report the perception, skill, and coordination configuration; the benchmark runs in simulation.
Goal interpretation, subgoal decomposition, action sequencing, and transition modeling
Useful for identifying decision-making errors; symbolic module evaluation should be distinguished from end-to-end physical execution.
Benchmarks code generation for free-form manipulation instructions using perception and motion-planning APIs shared across simulation and robots.
Integrates ROS, Gazebo, perception, and motion planning; studies physical constraints and module-level failures. Includes simulation benchmark evaluations and separate real-robot demonstrations on Franka and UR5 arms.
Automatically evaluates language-based household task planners, including execution feedback and replanning after action failures.
Uses ALFRED with AI2-THOR and WAH-NL with VirtualHome. Focuses on high-level task planning, with task sources shared with ALFRED.
Evaluates generated Python programs that compose mobility, perception, and human-interaction skills for service robots.
Contains 16 tasks with five prompt paraphrases each. Checks execution traces from multiple initial states against temporal-logic specifications. The benchmark measures program correctness in a symbolic simulator; CodeBotler robot demonstrations are separate deployment evidence.
Six collaborative manipulation tasks testing robot coordination, subtask planning, and task-space waypoint generation with environment feedback.
RoCo revises plans using feedback including collision checks; a separate motion planner executes the waypoints. The text-only reasoning dataset and real-world demonstrations are distinct from simulation benchmark runs.
Related: Related method
Evaluates household task execution from dialogue and interaction history, including collaboration between a Commander and a Follower.
Distinguishes Execution from Dialogue History, Trajectory from Dialogue, and Two-Agent Task Completion; the first two use recorded dialogue, while the third models both agents.
Aligns text-based household tasks with ALFRED environments to study high-level planning and transfer to embodied execution.
Offers text, visual, and hybrid environments. Results depend on the environment and observation/action interface; text-only runs evaluate language-based task execution.
Maps natural-language instructions and egocentric observations to navigation and interaction actions for multi-step household tasks.
Includes non-reversible object-state changes. Evaluates visual task execution in simulation, not real-robot motor control. ALFWorld provides aligned text environments for these tasks.
Tutorials, courses, talks, and interviews for learning about robot-use agents.
This collection takes inspiration from Awesome Computer Use Agent in its curation, organization, and presentation, and from Phillip Isola’s Robot-Use Agents blog in its focus on agents that interact with the physical world. We thank the authors and maintainers of the resources featured here for sharing their research, code, and demonstrations, and everyone who contributes suggestions and corrections.
@misc{awesome-robot-use-agent,
title = {Awesome Robot Use Agent},
author = {Kairun Wen and {Awesome Robot Use Agent Contributors}},
journal = {GitHub repository},
url = {https://github.com/kairunwen/Awesome-Robot-Use-Agent},
year = {2026},
note = {A curated collection of resources for robot-use agents.}
}
If you find this collection helpful for your work, please consider giving it a ⭐ Star.
This site uses Google Analytics to measure visits. How Google uses data.
Total views · —