r/ROS • u/Existing-Cat-2655 • 5h ago
r/ROS • u/Accomplished-Line721 • 6h ago
Using SLAM as an agent's spatial memory: I wrote an agent-friendly 2D lidar SLAM
Repo: https://github.com/ThomasZB/EvergreenSLAM
I've always wanted a robot to help with chores at home, with an agent in control (Claude Code, Hermes, ...). Today's models understand "get me a cola from the fridge" perfectly well. The problem is how the agent gets to know the home: where's the fridge, how do you get to the kitchen, where was the cola last time? You can do it with rules and text, but spatial info like that has to be described in a lot of detail, eats context, and may not survive frequent compaction.
Traditional SLAM is too much hassle for this. From my years working with SLAM, the real-world flow is roughly: drive around to map (usually after tuning odometry, calibrating extrinsics, syncing timestamps), save, switch to localization, re-tune when the environment changes, re-map when it changes a lot. Newcomers often don't know what they did wrong and end up with ghosting or drift, sometimes found only later, and then every station has to be configured again. For an agent the bigger problem is memory: coordinates are fragile, the map shifts after a rebuild and the agent can hardly tell.
So I flipped it around: why not a SLAM the agent can drive by itself? Walk the robot around once, say "this is the kitchen" in the kitchen, and never touch mapping or localization again. See the scissors in the left desk drawer, jot it down, and a few days later "where are the scissors?" has an answer. Furniture moves, the map gets re-optimized, the robot reboots, and the memory is still there.
That's lifelong SLAM underneath. It's been discussed in papers for years, but I couldn't find an open-source 2D implementation I could just use, so I wrote one:
- No mode switching. It always maps into the current session while localizing against frozen ones; a session freezes itself once it has enough new area and good constraints.
- The pose graph doesn't grow with uptime. Trimmed submaps are marginalized and sparsified into a Chow-Liu tree; frozen poses are constant.
- Places are directories. A place is bound to a submap, so it moves with the map when it's re-optimized; each place is a directory where the agent keeps notes and a SKILL.md. The agent uses a small CLI,
egs(also MCP). - One 2D lidar. Vision is unreliable when it's dark, and robust methods need compute; 3D lidars are expensive and a bit heavier on compute too.
Test robot: a 215 CNY (~30 USD) lidar, an Orange Pi with an RK3588, mecanum wheels, two power banks. After several days of on-and-off running at home and a round of bug fixes, it's stable enough to hand to an agent; once set up, the agent hardly ever asked me anything. In one test Claude Code drove it to the kitchen to check for changes, got wedged between an air purifier and the door frame, backed out, re-planned, then compared a photo with the last one and logged what looked like a mop. I never typed a coordinate.
To try it without hardware: pixi run -e ros demo replays a 3 MB public bag with a browser view.
Still a preview: tested in my flat and on public datasets, and I'm logging long-run memory numbers on the robot now. There's a comparison with slam_toolbox in the README; I haven't used slam_toolbox much and wrote it from its source, so corrections welcome.
r/ROS • u/llori218 • 12h ago
Open-sourced a small checker for robot release / acceptance evidence gaps
I’ve been looking at a narrow problem that can happen late in robot deployments:
a project passes acceptance testing, then software or configuration changes occur, and the final delivered release may no longer be the exact release covered by the test evidence.
I built a small offline tool to make that problem concrete.
robot-release-check currently checks only three things:
- delivered release vs. latest qualifying PASS
- changes after the last qualifying PASS
- closed items without verification evidence
The included example is fully synthetic. The tool does not connect to a robot, production system, CI/CD system, or source repository.
GitHub:
https://github.com/llorix218/robot-release-check
I intentionally kept v0.1 very small because I’m mainly trying to learn whether this failure mode actually shows up in real robotics projects.
For people doing ROS / AMR / AGV deployment, commissioning, QA, or customer acceptance:
Have you seen cases where the system delivered to the customer changed after the last acceptance test? If so, how does your team currently catch that?
r/ROS • u/OpenRobotics • 23h ago
News ROS News for the Week of October 12th, 2026 - Community News
discourse.openrobotics.orgr/ROS • u/DueHearing1315 • 1d ago
Discussion Turning 3D Gaussian Splatting scenes into robot navigation maps — is manual annotation still useful?
I've been experimenting with turning indoor 3D Gaussian Splatting (3DGS) scenes into maps that could be used for robot navigation.
3DGS is great for visual reconstruction, but a visually accurate scene doesn't automatically give you a navigation-ready map.
So I've been working on a workflow that allows me to:
- Define walkable and non-walkable areas directly on a reconstructed 3DGS scene
- Generate a 2D navigation grid from those annotations
- Mark semantic targets and robot stopping positions
- Use A* for offline path planning
- Export the annotated map as JSON for integration with other systems
I've attached a few screenshots showing the original scene, the walkable-area overlay, the resulting navigation grid, and the annotations from a 3D perspective.
This is currently focused on offline map preparation and path planning, not a complete autonomous navigation system.
I'm curious about how other robotics developers approach this problem.
A few questions:
If you're working with 3DGS or other reconstructed 3D environments, how do you prepare maps for navigation?
Is manually defining walkable areas and target points still common in your workflow?
Would you find a tool for this useful, or would you rather have someone prepare the navigation data for you?
What output formats would be most useful — occupancy grids, ROS/Nav2-compatible maps, JSON waypoints, or something else?
I'm exploring whether this workflow is worth developing into a tool or a small service, but I'd like to understand whether this is a real problem for others first.
Any feedback, including reasons why this approach might not be useful, would be appreciated.
r/ROS • u/Unfair-Confusion-168 • 1d ago
I got tired of spending more time on setup than on the robot, so I built a browser sandbox
https://reddit.com/link/1x7mn0d/video/90crysb97vvh1/player
Hey everyone,
I'm a robotics engineer, and hiring for robotics has frustrated me for a long time. A resume can say "experience with navigation" or "built a manipulation pipeline," but it doesn't show whether the candidate can find a fault in a robot that's moving and fix it.
So I built SimuCode, an assessment where the candidate gets a broken robot and has to get it working.
What it does:
- The candidate works in a browser IDE. Nothing to install.
- Each run executes in its own isolated sandbox.
- The verdict comes from the running robot, not from a checklist of expected output. The system looks at where the robot moved, what it sensed, and what it touched.
- The company sees what the candidate probes first and whether the robot recovers.
Where it's at: it's early, and I'm looking for feedback from people who hire or interview robotics engineers. What would you need to see in an assessment before you trusted it? And what would make you distrust one?
r/ROS • u/Classic-Pass-658 • 1d ago
How do you get a fix onto a robot that’s already running at a customer site?
Practical question for people running ROS robots in production rather than the lab.
When something breaks in the field and you have a fix, what does getting it onto the live robot look like? OTA, someone on site with a laptop, staged rollout, sim first?
The part I’m really trying to understand is trust. Before a change goes live on a working robot, what has to be true for your team or your customer to allow it? Rollback, speed or workspace limits during a trial, a named person approving, a record of what changed, data not leaving site, a security review?
Disclosure: I’m working on tooling around this. No link, just want to know how teams really do it.
r/ROS • u/Chemical-Hunter-5479 • 1d ago
Project Omarchy's AI (Grok & Claude) Controlling ROS Robot via AgenticROS!
r/ROS • u/Hopeful_Account_3464 • 1d ago
Project I built and just released an early-stage Python toolkit for investigating robot telemetry — looking for honest technical feedback
Hey everyone,
I've been working on a project called KORDEX Core, and I've just published its first release candidate, "0.1.0rc1", on GitHub.
The idea behind KORDEX is to explore how we can make robot telemetry easier to investigate: not just detect an unusual reading, but organize observations, identify patterns, examine available evidence, and communicate what we can and cannot conclude.
The current version includes:
- Rule-based telemetry pattern analysis
- Data-quality checks and uncertainty reporting
- Investigation of recorded telemetry
- Optional read-only MQTT integration
- An adapter for supported ROS2-style diagnostic payloads
- Structured findings and recommendations for human review
I want to be transparent about the project's current stage. This is an early prototype, not a production-ready robotics platform. It hasn't been validated against physical robots or field telemetry, and its real-world predictive accuracy hasn't been established. It doesn't send robot-control commands, and the ROS2-style adapter doesn't subscribe to a live ROS2 graph.
I'm sharing it because I want to learn from people who understand robotics, industrial systems, telemetry, and reliability engineering.
I'd genuinely appreciate your thoughts on a few questions:
I'm not claiming to have solved robot reliability. I'm trying to build something useful, understand the engineering challenges properly, and improve it based on evidence rather than assumptions.
Repository: https://github.com/Wale394/Kordex-core
Critical feedback is welcome. If something is poorly designed, unnecessary, or based on a wrong assumption, I'd rather know now than build on top of it.
Thanks for reading.
r/ROS • u/roxyjazzy • 1d ago
How to secure an internship in india as a robotics software engineer as a final year BTech student.
r/ROS • u/roxyjazzy • 1d ago
How to secure an internship in india as a robotics software engineer as a final year BTech student.
Hi, I’m a final year BTech Robotics and Automation student in India. I never did any internship till now and has no idea about how the industry works or what is the work of a robotics software engineer.
Since I’m in the final year ie. the 7th semester - there is an option that for the 8th semester j can either do a project or an internship and my college only allows to go for paid internships. I have knowledge of python ,C++ , Ros2 basics and some basic electronics knowledge. Now it the month of October and by December end i want to secure an internship in india with is paid at least 15k a month. I am interested in doing Ros2 simulations and is willing to explore and learn more about this. Can someone tell me , someone who is currently working as a robotics software engineer in india, how is the opportunities coming up, how to secure an internship, what all should i prepare for, what is the work that a robotics software engineer does and any other tips to keep in mind. Also i am planning to apply for at-least 20 to 30 companies for the same role so which all companies in india are actively hiring for this role. I really want to get some industry exposure. Someone please help me.
r/ROS • u/teriyakichickenn • 1d ago
Project Mechanical Engineer in Uncharted territory
Hi reddit, as the tittle says I am quite unfamiliar with anything programming (except for Arduino IDE and basic C), and need to program a whole animatronic. For my senior design project we're making a semi autonomous animatronic, using an Arduino Uno, Raspberry Pi 5 and ROS2, animatronic has already been designed, structurally analyzed and manufacturing has begun, and all parts have been acquired, but... code. From what I've researched if I want to code it remotely I have to download Docker, which has built in Ubuntu (allows my windows to run a Linux interface which ROS2 needs), and use Visual Studio Code in conjunction. I've also heard the words Humble, Jazzy and Tier 1 support being thrown around. Also read there's another option to just use VS Code and an SSH-extension. Which would be the better option of the two, and what exactly are Humbly/Jazzy/Ubuntu and ROS2?
TL;DR : Need help understanding ROS2 programming interfaces, VS Code, SSH Extensions and how they work together, and what's the best option to coding my animatronic.
r/ROS • u/Technical-Force-6266 • 2d ago
Robotics System Test Engineer (How)
Hi all,
I want to specialize in Robotics System Test Engineering. Testing at System-level. Not Software Test in isolation.
But; I want to learn this by doing. Not to complete separate courses in ROS2, Linux, Git, etc. I need a complete project designed for this purpose that I can follow and learn through it.
Can anyone help?
Example to the role I mean;
Best,
r/ROS • u/Wooden_One_4001 • 2d ago
colcon build RAM for MoveIt 2 from source, measured: default settings fail at 8 GB, sequential peaks at 2.4 GB
I measured how much RAM a from-source MoveIt 2 build needs (Jazzy branch, 41 packages) by capping a Docker container's memory with swap disabled, on a 4 vCPU Linux box. Peak memory is the container's cgroup memory.peak.
| Mode | RAM cap | Result | Peak |
|---|---|---|---|
default colcon build (4 workers × make -j4) |
16 GiB | OK, 41/41 | 8.7 GiB |
| default | 8 GiB | failed: cc1plus killed after 24/41 packages | 8.0 GiB (cap) |
| default | 4 GiB | failed after 4/41 | 4.0 GiB (cap) |
sequential (--parallel-workers 1, MAKEFLAGS=-j1) |
8 GiB | OK, 41/41 | 2.4 GiB |
| sequential | 4 GiB | OK, 41/41 | 2.4 GiB |
For comparison, Nav2 (45 packages) with the same method built fine at 8 GiB with default settings (peak 8.1 GiB at 16 GiB cap, 2.7 GiB sequential). So "8 GB is enough for ROS 2" holds for Nav2 but not for MoveIt 2 on default settings.
Caveats:
- memory.peak includes page cache, so the 8.7 GiB at 16 GiB is an upper bound, but the 8 GiB run still OOM-killed the compiler.
- Default parallelism scales with cores, so an 8- or 16-core machine will use more RAM than this 4-core run.
- Build times (1h37m default vs ~1h56m sequential) are not reliable; the box was a burstable cloud instance.
If you're on an 8 GB laptop: MAKEFLAGS="-j1" colcon build --parallel-workers 1, or just sudo apt install ros-jazzy-moveit if you don't need to modify MoveIt.
Script, results.tsv and per-run build logs: https://github.com/sungpyo9053/blog/tree/main/experiments/colcon-ram-moveit2-real-build
Full write-up (with the Nav2 comparison): https://huntlab.app/en/colcon-build-moveit2-ram-8gb-fails-sequential-measured/
(disclosure: my blog)
Was bored, vibe-coded an RViz plugin to move the camera focal point with keyboard controls and publish navigation goals.
I've been working on an autonomous driving simulator using CARLA, Autoware, and ROS 2.
One small but annoying problem I encountered was having to reach for the mouse every time I wanted to move the RViz camera or select a new navigation goal, especially when operating the simulator with a steering wheel.
So I built a small RViz plugin to make this easier.
What it does:
- Arrow keys move the RViz focal point along the X/Y axes.
- A visual reticle indicates the selected position.
- Enter publishes the target as a
PoseStampedgoal. - R resets the camera to the vehicle's
base_link. - Other ROS 2 nodes can also control the focal point through topics.
It's implemented using a C++ RViz panel plugin and a Python ROS 2 keyboard node.
Currently tested with ROS 2 Jazzy. The keyboard input runs in a separate terminal, and the view needs to be Orbit or XYOrbit.
GitHub: https://github.com/vinayakaraju46/rviz_focal_control
I'd be interested to know how others handle navigation-goal selection in their teleoperation or simulation setups.
Is there an existing RViz feature or tool that solves this more elegantly? Any suggestions for improving the implementation?
r/ROS • u/JuggernautWhich9757 • 3d ago
Question What are the Current Bottlenecks on developing SWARM robots to perform real world tasks?
r/ROS • u/No_Quail6685 • 3d ago
From real OS0 body clusters to a walking human in my robot dojo — calibration and heading failures
I’m updating the office “robot dojo” from my earlier simulation posts with actual Ouster OS0 data and a fixed iPhone camera.
This episode follows the sensor coming back online, its new scans being registered to the existing room map, and the camera/LiDAR alignment going through three iterations. The first solution looked good on flat walls but put the camera about 0.5 m sideways and 0.4 m too high. Matching a rendered Gaussian splat to the video, then adding my moving silhouette, exposed the ambiguity.
The next experiment was a walking human inside the twin: segment my body from each LiDAR frame → fit a capsule skeleton → colour it from the phone footage → estimate heading and gait → send the fitted avatar along paths in the room.
Some useful failures:
• The body clusters ranged from 45 to nearly 3,000 points per frame.
• Front/back ambiguity made the heading flip 180°; I used a global track plus walking direction to stabilise it.
• 23 frames contained mostly legs, so parts of the upper body are guessed.
• The gait model comes from one recording and about 13 m of walking.
The sensor capture is real; the displayed replay is recorded; the robot and generated avatar walks are simulation. The reported fit residuals are internal model checks, not independently validated motion-capture accuracy.
My EP10 build diary: https://www.youtube.com/watch?v=YlriQfsrjhc
For testing robot behaviour around a person in this twin, what would you validate first: human pose/heading uncertainty, timing between the sensors, or collision geometry? I’m especially interested in simple tests that would expose an unsafe or misleading model.
Disclosure: this is my own project/video, developed with AI agents. The episode mixes my recorded voice with disclosed synthetic narration using a clone of my own voice.
r/ROS • u/OpenRobotics • 3d ago
ROS Office Hours October 23rd at 9am PT -- Ask questions and show off your project! Link in comment.
r/ROS • u/Kirito-San8432 • 4d ago
Question My chess-playing Franka Panda robot is getting better! ROS 2, MuJoCo, Octo VLA, MPC + WBC
Hey everyone!
I've been developing a chess-playing Franka Panda robot in simulation as part of my Mechatronics Engineering thesis. I also developed the chess AI myself!
The project combines ROS 2, MuJoCo, Octo VLA, computer vision, Model Predictive Control (MPC), and Whole-Body Control (WBC).
Recently, I've made some significant improvements to the robot's movements.
Interestingly, most of the improvement didn't come from major changes to the controllers. I discovered that my Octo model wasn't receiving the robot states correctly, and I also had issues with camera reliability and ROS 2 communication.
After fixing these problems and making a few adjustments to MPC + WBC, the robot started behaving much better!
I also recently migrated my custom Finite State Machine (FSM) to the YASMIN framework, which has helped me organize the robot's behavior and improve error handling and recovery.
There's still plenty to improve, and my next goal is to explore reinforcement learning to dynamically adjust the MPC and WBC parameters.
I've been developing this project independently, learning a lot through experimentation, debugging, and research.
I'd love to hear some recommendations from the ROS community!
Are there any other ROS 2 frameworks or tools you'd recommend for improving robotic system reliability, task coordination, error recovery, or overall architecture?
I'm particularly interested in tools that could complement YASMIN and help manage complex robotic manipulation systems.
If you've worked with similar architectures involving ROS 2, VLA models, or advanced controllers, I'd love to hear about your experiences!
Any feedback or suggestions are welcome!
r/ROS • u/NarinderBrar • 4d ago
Introducing RobotLens — a connected engineering workspace for ROS 2

Hi everyone!
I’ve been building RobotLens, a native robotics engineering workspace that brings visualization, debugging, diagnostics, simulation, and development workflows into one connected environment.
My background is in computer graphics and game development. When I started learning ROS 2, I found myself constantly switching between terminals, visualizers, logs, simulators, and other tools.
Each tool was useful on its own, but as a beginner, I often struggled to understand how everything connected and what was actually happening across the system.
That experience led me to start building RobotLens.
Coming from graphics and game development, I’ve always appreciated how tools like Unity and Unreal bring complex workflows into a single, interactive workspace. I wanted to explore a similar approach for robotics engineering.
Here’s what’s been built so far:
- ROS 2 or Zenoh Workspace: Live discovery, topic inspection, TF visualization, diagnostics, and communication backend support.
- Visualization: Native Vulkan rendering, robot visualization, and interactive scene manipulation, topic visualization.
- Simulation: Gazebo and MuJoCo integration, with support for different simulation workflows.
- Navigation & Motion Planning: Nav2, EasyNav, and MoveIt 2 integration.
- Debugging & Analysis: Recording, plotting, logs, diagnostics, and an interactive topology view to explore system relationships at a glance.
- Integrated Development Environment: Built-in code editing, scripting, terminal, and development tools.
- Procedural Environment Generation: Create simulation environments programmatically, with MCP-based extensibility and LLM-assisted workflows.
- Reinforcement Learning: An RL Cockpit for visualizing, monitoring, and debugging robot training.
- Extensibility: MCP support for extending workflows, debugging, executing commands and building additional tools.
- User Experience: Drag-and-drop interactions and visual workflows designed to reduce repetitive configuration and command-line operations.
The goal isn’t simply to put existing ROS tools into one application.
It’s to make robotics development more connected, intuitive, and accessible, while still providing the depth and flexibility experienced developers need.
RobotLens is still under active development, and I’m preparing it for early users.
I’d value feedback from people working with ROS and robotics every day.
I’m also open to collaborations, partnerships, and conversations with individuals, research labs, or robotics companies interested in exploring RobotLens.
Looking forward to hearing your thoughts!
Vidoes : https://www.youtube.com/watch?v=BwFGRrJkLpk&list=PLNVRrfh5q7Zo&index=1
Website : https://robotlens.dev/
LinkedIn : LinkedIn Login, Sign in | LinkedIn
r/ROS • u/Chemical-Hunter-5479 • 5d ago
News 🚀 RealSense SDK 2.59.1 beta is now available!
r/ROS • u/SomewhereCommon4493 • 5d ago
Question for robotics developers working with real-world environments
r/ROS • u/Accomplished-Team725 • 5d ago
Open-source drone simulator controlled by hand gestures
SkyHand is a 3D drone simulator using p5.js + ml5.js hand tracking.
Control the drone directly with hand gestures.
Demo:
https://alzin.github.io/skyhand
GitHub:
r/ROS • u/samyak1729 • 5d ago
Discussion Can World Models absorb Simulation Stack as some people claim?
Last week I went down rabbit hole researching world models for an article I was writing, and I kept coming across discussions (specifically from NVIDIA engineers) about whether they could eventually replace conventional robotics simulators.
One important part to note is that different researchers seem to be working toward quite different versions of that future.
LeCun's JEPA approach focuses on predicting representations for planning. NVIDIA and other groups are exploring action-conditioned video models that can act as interactive environments. World Labs, meanwhile, is working on generating and reconstructing physically grounded worlds for robot training and evaluation.
The broader idea is sometimes called Simulation 2.0 (popularised by Jim Fan), although I'm not sure there's a shared definition of what that actually involves.
Training policies using learned dynamics isn't new. Dreamer has been doing that for years. What seems different now is the attempt to absorb more of the robotics workflow, from environment creation and demonstration collection to policy evaluation and training.
I went through some recent research to understand how far these approaches have actually gotten.
1. Interactive World Simulator (RSS 2026)
This was one of the more interesting papers I found.
The researchers built an action-conditioned video simulator that supports over 10 minutes of continuous interaction at 15 FPS on a single RTX 4090.
They collected demonstrations entirely inside the learned environment and used them to train imitation policies. Those policies achieved real-world performance comparable to policies trained using an equivalent amount of real demonstrations.
They also evaluated policy checkpoints inside the simulator and compared their performance against real-world evaluations across four manipulation tasks: T-pushing, rope routing, mug grasping, and pile sweeping.
They also evaluated four policy architectures (DP, ACT, π₀, and π₀.₅) across four manipulation tasks, comparing performance in the learned simulator against real-world results.
The reported Pearson correlations were 0.855 for T-pushing, 0.846 for rope routing, 0.887 for mug grasping, and 0.991 for pile sweeping, based on evaluations across 20 initial configurations per task.
The simulator tended to overestimate performance slightly on three tasks, but it was remarkably consistent at distinguishing stronger policies from weaker ones. That's potentially useful for selecting policy checkpoints without running every evaluation on physical robots.
2. World Labs: Real-to-sim-to-real (July 2026)
This was something I initially overlooked when reading about Marble and Atlas.
World Labs acquired SceniX and published results from a real-to-sim-to-real system that reconstructs physical tasks as interactive simulations.
Their demonstrations include bimanual box packing, cable routing, cube handover, and manipulation involving deformable objects.
They report training policies entirely in simulation, with zero real-world training data, and transferring them to physical robots. They also show policies operating autonomously for extended periods.
Their policy-evaluation results were particularly interesting.
For a bimanual cube-handover task, they evaluated each checkpoint using:
- 2,000 simulated trials: 1,000 in-distribution and 1,000 out-of-distribution.
- 100 real-world trials: 50 in-distribution and 50 out-of-distribution.
Across different policy architectures and checkpoints, they report that simulated performance preserved relative real-world policy rankings, including similar regions of success and failure.
However, World Labs' approach combines generative world models with task-aligned robotic simulation. It's not a demonstration that a single neural model has replaced the physics engine.
3. World Labs: Marble and Atlas
World Labs is also approaching a different bottleneck: creating the environments themselves.
In robotics case studies using Marble, researchers generated 3D environments and imported them into Isaac Sim, MuJoCo, and RoboSuite.
World Labs reports over 90% less manual environment-curation effort, with workflows that previously took days or weeks reduced to hours or minutes.
Their newer Atlas model extends the spatial-generation capabilities, including reconstruction from visual observations and camera-controlled video generation.
Automating the creation of simulation environments could have a substantial impact even if the underlying physics simulation remains conventional.
There's also an interesting direction emerging around automating the creation of physics-accurate 3D assets. Tools like Rigyd are working on converting images, text descriptions and 3D models into simulation-ready assets with collision geometry, articulation, and physical properties for conventional simulators.
And unlike the R2S2R results, Atlas' visual-generation capabilities don't establish that robot policies trained in its generated worlds will transfer successfully.
4. DreamDojo (ICML 2026)
DreamDojo approaches the problem through large-scale video pretraining.
The model was pretrained on 44,000 hours of egocentric human video and subsequently adapted to robot-specific interactions.
After distillation, it reaches 10.93 FPS and supports interactive rollouts lasting over a minute.
The paper demonstrates live teleoperation, policy evaluation, and model-based planning, along with evaluations on out-of-distribution interactions.
But I haven't seen evidence from DreamDojo that establishes general-purpose replacement of a conventional simulator for large-scale robot RL.
5. What-If World: A useful reality check
What-If World examines something that I think gets overlooked in the impressive interactive demos: whether models respond correctly when physical conditions change.
Its benchmark uses 319 paired scenarios to test physical consistency across nine models.
The reported results were quite weak:
- Best paired consistency: approximately 51.7%
- Open-source models: approximately 28%
- Friction-related interventions: approximately 14.2%
A model can generate realistic-looking motion without correctly predicting what happens when friction, mass, or other physical properties change.
That's particularly relevant to domain randomization and contact-rich manipulation.
These results aren't a direct evaluation of IWS, DreamDojo, or World Labs, so I wouldn't use them to dismiss those systems. But they highlight a capability that a general-purpose learned simulator would need.
One thing I'm still unsure about is how much of this translates to actual policy optimization.
Generating demonstrations, evaluating checkpoints, and reconstructing environments are all useful. But they're different from training policies through millions of closed-loop interactions.
Traditional simulators provide explicit physical parameters, privileged state, reproducibility, and thousands of parallel environments.
There's also the problem of model exploitation. A learned simulator might accurately evaluate existing policies but become unreliable once an RL algorithm starts optimizing against its prediction errors.
I haven't found a convincing general-purpose video world model that can replace a physics simulator for large-scale. But I still hear discussions around the same
At the same time, results like IWS and World Labs' R2S2R work make it difficult to dismiss learned simulation as just impressive video generation.
I'm curious how people working on robot learning and simulation interpret this.
Would also be interested in papers that test learned simulators under sustained RL optimization rather than just demonstration generation or policy evaluation.
References: