Vision Language Action
Camera patches and a language instruction enter one transformer. The policy outputs steer, throttle, and brake on a forked road.
vision tokens
actions drive the car
take the second exit
text tokens