Physical AI, Chapter 3: The Physical AI Stack

This is Part 3 of a series walking through my book Physical AI: Engineering Embodied Intelligence for the Real World. In the previous chapter, we traced how the field got here. This one opens the machine up and walks the stack from photon to torque.

From the outside a robot looks like one object that responds to commands. From the inside it is a stack of layers, each running at its own rate, each with its own failure modes. Chapter 3 makes that picture explicit by following a single perception-action cycle end to end: a warehouse mobile manipulator asked to move a cardboard box from a conveyor to a pallet, with a RealSense D455 on a pan-tilt head, a 6-axis arm, a Jetson AGX Orin, and a behavior-cloning policy trained on a few thousand demonstrations.

The trace is mostly mundane, which is the point. A photon lands on the sensor; rolling-shutter readout takes about 33 ms. The driver wraps the frame in a ROS 2 message and another 5 to 15 ms disappears into transport. A segmentation network isolates the box in 20 ms, a pose estimator turns the masked depth into a 6-DOF pose in 15 more, and a tf2 lookup puts that pose in the arm’s frame. The policy — a 450-million-parameter VLA at int8 — runs in 40 ms and emits eight joint waypoints. An interpolator smooths them to 250 Hz, a PID loop turns each position into a motor current at 1,000 Hz, and the arm moves. One cycle, five rate regimes, four data representations, three compute domains, and a latency contribution and a bug opportunity at every interface.

The catalogue the policy actually reads

Sensors define the upper bound of what a robot can know, and no amount of cleverness downstream recovers information that was never captured. The chapter works through the classes with their real characteristics and real failure modes — RGB and depth cameras and where stereo, time-of-flight and structured light each break; LiDAR and why an 80 mm Ouster is unproblematic on a vehicle roof and impossible on a humanoid head; IMUs and visual-inertial odometry; and the force-torque and tactile layer that the foundation-model era has most undersold. A wrist force-torque sensor is how a policy knows the gripper actually closed on the box, that the screwdriver is on the screw and not the housing, that the peg went into the hole.

Perception then folds those bytes into one of five recurring shapes: a 6-DOF pose, a point cloud, an occupancy grid, a semantic segmentation mask, or a world-model latent. A real stack produces several in parallel and routes each to whichever consumer needs it — point cloud to the collision checker, pose to the manipulation policy, mask to the operator display.

Where the budget goes, and the pattern everyone converged on

At 10 Hz the whole loop has 100 ms; at 50 Hz it has 20. The chapter lays out the line-by-line allocation — exposure and readout, driver and transport, perception inference, world-model inference, policy inference, interpolation, actuator response — and the four optimisations that buy the budget back: fuse the networks to remove intermediate serialization, quantize to int8 for a two-to-fourfold speedup, parallelize across GPU and CPU, and predict forward so the action is aimed at where the system will be rather than where it was.

The structural answer the field has settled on is the split-rate architecture: a slow policy producing an action chunk at around 10 Hz, and a fast correction layer at 50 to 200 Hz reacting to fresh perception inside that chunk. Figure’s Helix runs cognition at 7–9 Hz and control at 200 Hz; Pi-0 does something similar with a flow head. It is common enough in 2024–2026 to be considered the default. The chapter closes with a working ROS 2 node — best-effort QoS on the sensor subscriptions, 30 ms approximate time synchronization, int8 modules, an eight-waypoint chunk on a JointTrajectory topic, and p50/p95/p99 logging — because the mundane plumbing is where most of the bugs in real systems live.

Tomorrow: we have described the body. Chapter 4 describes the brain — LLMs, VFMs, VLMs and VLAs, the move from CNNs to transformers to state-space models, diffusion policies, and the liquid neural networks that run in a few thousand parameters at the edge.