Physical AI, Chapter 6: Bridging the Sim-to-Real Divide

This is Part 6 of a series walking through my book Physical AI: Engineering Embodied Intelligence for the Real World. In the previous chapter, we established why real data is unaffordable at scale. This one is about the consequence: most modern policy training happens in a simulator, and the simulator is wrong.

The gap between a policy’s success rate in simulation and its success rate on the hardware the simulation was meant to approximate is the reality gap, and closing it is the central engineering problem of simulation-based training. A policy that succeeds 95% of the time in the simulator and 60% of the time on the robot has a gap of thirty-five points, and every one of those points has a specific source.

The chapter enumerates them. Contact dynamics are approximated by a static friction model, while real friction depends on temperature, humidity and wear. Rendered camera images are clean; real ones carry motion blur, rolling-shutter artefacts, lens flare, sensor noise and colour shift. Simulated motor commands produce instantaneous joint motion; real ones are delayed by response time, distorted by backlash and modulated by load. Then there is the unmodelled-effect problem, which is harder because no enumeration of it is ever complete: ambient light changing over the day, dust on the lens, vibration from a neighbouring machine, a person walking through the workspace.

Four techniques, layered

Domain randomization is the one that changed practice. Instead of one simulator that matches reality, thousands that differ — friction, link masses, actuator gains, control latency, lighting, texture, sensor noise all resampled per episode, so the real world looks to the policy like one more draw from the distribution it trained on. OpenAI’s Rubik’s Cube result, with Automatic Domain Randomization widening the ranges as the policy succeeded, was the proof of concept. It is now standard enough to ship in the default Isaac Lab configs. The chapter makes the distinction that matters in practice: visual randomization and dynamical randomization work by different mechanisms and are not interchangeable, and a pipeline that applies only one produces a policy robust to only one kind of mismatch.

Domain adaptation learns the bridge to one specific environment rather than robustness to all of them — a small adapter trained to translate real observations into the feature space the policy knows. It is more sample-efficient than widening randomization, and it fails in a characteristic way: it works on the observations it was trained on and breaks on outliers, exactly where you most need it. Which is why it stays a refinement layer rather than a primary mechanism.

Real-to-sim-to-real starts from a 3D scan of the actual workspace — NeRF or Gaussian splatting from a few dozen photographs — loaded into the simulator as a backdrop, so the policy trains against the specific shelves and tables it will meet. The bottleneck, in practice, is not scanning. It is converting the scan into simulator-compatible assets with articulation and contact surfaces identified, which is still substantially manual and has become one of the operational moats separating the leading programs.

World foundation models — NVIDIA Cosmos being the canonical instance — invert the approach: rather than approximating the world from first principles, learn it from video. The output is often harder to distinguish from real data than a simulator’s is. Its physics, though, are implicit, and objects sometimes pass through each other. Current best practice keeps physics simulators for consistency and uses world models for visual realism and rare scenarios.

The chapter closes with a full Isaac Lab randomization configuration and the metric worth tracking: the sim-to-real success-rate ratio, around 0.85–0.95 for a well-tuned pipeline and as low as 0.5 for one that is not.

Tomorrow: the policy is trained. Chapter 7 asks whether the robot can actually carry the computer it needs to run it — the sensor fusion tax, the training-to-edge compute mismatch, and the thermal and energy ceiling nobody puts in the demo video.