Physical AI, Chapter 4: Architectures of Intelligence

This is Part 4 of a series walking through my book Physical AI: Engineering Embodied Intelligence for the Real World. In the previous chapter, we walked the stack and left the policy as a black box at its centre. This one opens the box.

A policy is doing three things at once, usually implicitly: perception in the narrow sense, turning the current observation into a representation of the world; reasoning, using that representation plus history plus the instruction to decide what to do; and action generation, turning the decision into motor commands that will actually produce the behaviour. Every architecture in the field is a different way of apportioning compute across those three jobs, and a modern stack rarely gives all three to one network.

The chapter works through the roles in order. Large language models sit at the slow planning layer, decomposing “clean up the kitchen” into steps, carrying a prior about what humans mean that no amount of robot data supplies. They also hallucinate, propose actions the robot cannot execute, and take hundreds of milliseconds to seconds — which is why they live one or two levels above the control loop and never in it. Vision foundation models like DINOv2 and SAM 2 donate perception the policy would otherwise have to learn from scratch, amortising millions of dollars of pre-training; the catch is that they were pre-trained on internet images, and a wrist camera that sees the gripper in every frame is not an internet image. Vision-language models ground the instruction in pixels — but only so far.

Four primitives, and the head at the end

Underneath the roles sit the primitives: convolutional networks, which still win where the hardware accelerator is tuned for them; transformers, which are the core of almost every modern VLA; state-space models, which scale linearly in sequence length and have a recurrent formulation that fits the perception-action loop naturally; and diffusion, which is less a primitive than a training paradigm but matters too much in action generation to leave out.

Diffusion Policy denoises a chunk of future waypoints from pure noise, conditioned on the observation. Its three advantages are real — smooth trajectories with no discretisation artefacts, multi-modal action distributions that can represent “left hand or right hand” instead of averaging them into an unusable middle, and action chunking built in. Its cost is real too: ten to a hundred denoising steps, each a forward pass. The chapter includes a working sketch of a conditional 1D U-Net and the DDPM sampling loop precisely so the cost is visible in the code rather than asserted in prose.

At the far end of the frontier sit Liquid Neural Networks, the MIT CSAIL line inspired by the 302-neuron nervous system of C. elegans, in which each neuron’s time constant is itself a function of the input. Published LNN controllers run in hundreds to thousands of parameters. A 19-neuron network doing lane-keeping on real video, still working under perturbations that broke CNN-based controllers, is the result that made the field pay attention — though the conditions are narrower than the headlines suggest.

The trade-off surface has three axes: parameter count, inference latency, and adaptability. A 7B VLA at fp32 is too slow for 10 Hz on a Jetson Orin and at int8 is just fast enough. A 450M model fits comfortably and has headroom. And memory bandwidth, not capacity, is usually the binding constraint — which is why aggressive quantization is standard even on platforms with memory to spare.

Tomorrow: every architecture here is trained on data, and physical data costs three to four orders of magnitude more per sample than scraped text. Chapter 5 is about the bottleneck that follows from that, and the teleoperation rigs, pooled datasets and flywheels built to get around it.