Physical AI, Chapter 7: Hardware Constraints and Edge Computing

This is Part 7 of a series walking through my book Physical AI: Engineering Embodied Intelligence for the Real World. In the previous chapter, we got a trained policy off the cluster. This one asks whether the robot can carry the computer that runs it.

A policy that ran beautifully on an H100 — with eighty gigabytes of high-bandwidth memory and a three-hundred-watt thermal budget — has to be cut down to run on a single module bolted to the robot, inside a sixty-watt envelope, while competing for cycles with the perception pipeline, the motion controller, the safety monitor and the network stack. The cut is rarely small and never free.

The numbers make the shape of the problem clear. An H100 delivers roughly two thousand INT8 TOPS with three terabytes per second of memory bandwidth. A Jetson AGX Orin — the standard platform for ambitious 2025 and 2026 robots, and the implicit reference design at Tesla, Figure, Apptronik and 1X — delivers about 275 TOPS sparse, 150 dense, with 204 GB/s and 64 GB of unified memory. The compute gap is around seven to one; the bandwidth gap is fifteen to one. And every watt spent on inference is a watt not spent on locomotion.

The tax you pay before the policy runs

Before any of that, there is the sensor fusion tax. A camera at 30 Hz, depth at 10, an IMU at 1,000, joint encoders at 1,000 or 2,000, force-torque at 1,000 — each with its own latency, its own coordinate frame, and its own drifting clock. Pairing the most recent sample of each is wrong in general, because the camera frame that just arrived was captured tens of milliseconds ago while the IMU sample was captured one millisecond ago, and treating them as simultaneous introduces a misalignment the policy never saw in training. Host-side timestamping is the common remedy; hardware sync is the only way to get to single-digit microseconds. Then there is the question nobody plans for until it happens: what the policy does when a stream drops, which it can only handle gracefully if it was trained with the possibility of missing inputs.

Getting the model to fit

The techniques are individually well known and the order matters. Quantization first — INT8 as standard, INT4 for the weight-heavy linear layers, FP8 where the hardware supports it, with quantization-aware training reserved for the layers that lose too much. Then distillation, in two distinct modes: shrinking a generalist into a smaller generalist (SmolVLA is a 450M distillation of larger Open X-Embodiment models), or compressing a generalist into a task-specific network that is faster and more accurate on its one job and useless off it. Then structured pruning, which removes whole channels and heads rather than scattered weights, so the result is a smaller dense model rather than a sparse one needing hardware support.

Two architectural moves round it out. Mixed cloud-edge inference splits the system by rate — a large reasoning model in the cloud producing plans every few seconds, a fast policy on the robot producing actions at loop rate — and needs an explicit fallback for when connectivity drops. And action chunking reduces inference frequency rather than inference cost, letting a policy run at one-eighth the rate at the price of executing actions conditioned on an observation half a second old.

The chapter closes with a full deployment budget for a representative humanoid on an Orin AGX: 16–18 GB of application memory against 56 available, a compute load dominated by the VLA at roughly half to three-quarters of dense throughput, 75 watts of compute-side draw eating 12–20% of the battery over a shift, and a total perception-action loop latency of 150–200 ms. Just large enough, with little headroom — which is the honest summary of the platform.

Tomorrow: everything in this chapter was a safety question wearing engineering clothes. Chapter 8 takes it head on — what changes when a digital failure becomes a physical hazard, and what the EU AI Act, the US patchwork, ISO 10218 and ISO/TS 15066 actually require of you.