Vision-Language-Action (VLA) Model

A Vision-Language-Action (VLA) model is a neural network that maps what a robot sees and what a person says directly to what the robot does. You give it a camera image and an instruction such as “put the carrot on the plate”, and it returns motor commands. The defining trick is that a VLA is built on top of a pretrained vision-language model (VLM), so the robot starts out already knowing what a carrot, a plate, and the word “on” mean, rather than learning those concepts from a few thousand teleoperated demonstrations. The term was introduced by Google DeepMind’s RT-2 paper, which showed that co-training a VLM on web data and robot trajectories produces policies that follow commands never seen in the robot data. VLAs are now the standard “brain” for Physical AI systems.

Why VLAs Exist

Classic robotics stacks are pipelines: a perception module detects objects, a planner picks a grasp, a motion controller executes it, and each stage is hand-engineered for one task and one robot. They are precise but brittle; change the lighting, the object, or the instruction and something breaks.

Imitation-learning policies trained end to end on robot demonstrations removed some of the hand engineering, but they only generalize as far as their (small, expensive) robot dataset reaches. A VLA attacks the data problem from the other side. The internet already contains billions of image-text pairs that teach a model what things are and how language refers to them. If actions can be expressed in a form the VLM can produce, all of that knowledge transfers to control.

How a VLA Works

Every VLA has the same three-part shape: a vision encoder turns pixels into tokens, a language-model backbone fuses those tokens with the instruction and (often) the robot’s joint state, and an action head turns the backbone’s output into motor commands.

%%{init: {'theme': 'base', 'themeVariables': { 'edgeLabelBackground': '#FFFFFF', 'lineColor': '#818CF8' }}}%%
graph LR
    subgraph IN["Inputs"]
        A["Camera frames"]
        B["Instruction: 'put the carrot on the plate'"]
        C["Proprioception: joint angles, gripper"]
    end
    A --> D["Vision encoder<br>SigLIP / DINOv2"]
    D --> E["VLM backbone<br>Llama / PaliGemma / Gemma"]
    B --> E
    C --> E
    E --> F["Action head"]
    F --> G["Action chunk<br>next H steps of motor commands"]
    G --> H["Robot controller"]
    H -->|new observation| A

    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;
    class A,B,C data;
    class D,E process;
    class F,G output;

The interesting design question is the action head, because a language model emits discrete tokens and a robot needs continuous numbers. Three answers have emerged.

1. Actions as text tokens (RT-2, OpenVLA)

The simplest idea: discretize each action dimension into 256 bins and treat each bin index as a token. RT-2 wrote actions out as ordinary text so they could share a training format with captions and visual question answering; OpenVLA (a 7B model on a Llama 2 backbone with fused DINOv2 and SigLIP vision features, trained on 970k real-world robot demonstrations) does the same by overwriting rarely used vocabulary tokens.

# Scenario: turning a continuous end-effector command into tokens a VLM can emit.
# Each of the 7 dims (xyz, rotation, gripper) is clipped and quantized into 256 bins.
import numpy as np

N_BINS = 256

def action_to_tokens(action, low, high):
    """action, low, high: arrays of shape (7,). Returns 7 integer token ids."""
    norm = (np.clip(action, low, high) - low) / (high - low)   # 0..1
    return np.minimum((norm * N_BINS).astype(int), N_BINS - 1)

def tokens_to_action(tokens, low, high):
    """Inverse: bin centers back to continuous values."""
    return low + (tokens + 0.5) / N_BINS * (high - low)

This is easy to train with the standard next-token loss and reuses all existing LLM infrastructure. The cost is speed and precision: one action step is 7 tokens, so a 50-step chunk is 350 tokens generated one at a time, which is slow for a robot that wants to run at tens of hertz.

2. Compressed action tokens (FAST)

Physical Intelligence’s FAST paper observed that naive per-timestep binning performs poorly on dexterous, high-frequency data because consecutive actions are almost identical, so the tokens carry little information each. FAST applies a discrete cosine transform to a whole chunk of actions, keeps the significant coefficients, and compresses them into far fewer tokens. Paired with π0, the authors report matching diffusion-style VLAs while cutting training time by up to 5x.

# Scenario: compress a 1-second, 50 Hz trajectory chunk (50 steps x 7 dims)
# into a short token sequence instead of 350 per-step bin tokens.
import numpy as np
from scipy.fft import dct

chunk = np.load("chunk_50x7.npy")                  # (50, 7), normalized to [-1, 1]
coeffs = dct(chunk, axis=0, norm="ortho")          # frequency domain, per action dim
quantized = np.round(coeffs * 10).astype(int)      # coarse scale; small coefficients become 0
# Most high-frequency entries are now zero, so a byte-pair encoder over the
# flattened sequence yields a much shorter token string than raw binning.

3. Continuous actions via diffusion or flow matching (π0, GR00T, Gemini Robotics)

The third family attaches a separate, smaller action expert to the VLM and generates continuous action chunks by iterative denoising. π0 starts from a pretrained VLM and adds a flow-matching action expert that produces a whole chunk of continuous actions at once, which is what lets it control dexterous tasks such as laundry folding and box assembly at high frequency. NVIDIA’s GR00T N1 calls the same split a dual-system design: a vision-language module (“System 2”) interprets the scene and instruction, and a diffusion-transformer module (“System 1”) turns that into fluid motor actions in real time, with both trained jointly end to end.

# Scenario: sample one action chunk from a flow-matching action expert
# conditioned on the VLM's features for the current scene (a sketch of the idea,
# not any one library's API).
import torch

def sample_action_chunk(vlm_features, action_expert, horizon=50, action_dim=7, steps=10):
    x = torch.randn(1, horizon, action_dim)          # start from noise
    dt = 1.0 / steps
    for i in range(steps):
        t = torch.full((1,), i * dt)
        velocity = action_expert(x, t, vlm_features) # predicted direction toward real actions
        x = x + velocity * dt                        # one Euler integration step
    return x                                         # (1, 50, 7) continuous actions

Action Chunking and the Latency Budget

Whatever the action head, modern VLAs rarely output one action at a time. They predict an action chunk: the next H control steps. The robot executes the chunk while the model thinks about the next one. Chunking is what makes a multi-billion-parameter model usable on a robot that needs to move smoothly at 20 to 50 Hz, because the expensive forward pass is amortized across many steps.

It also creates a trade-off a reader would otherwise have to take on faith. A long chunk gives the model time to finish thinking, but the robot runs open loop for longer and cannot react to a slipping object. A short chunk is more reactive but demands that inference finish before the previous chunk runs out. The widget below makes the budget concrete.

A Working Control Loop

Putting the pieces together, a deployed VLA runs a loop that predicts a chunk, executes part of it, then re-observes. Executing only the first few steps of each chunk (a receding horizon) gives back some of the reactivity that long chunks lose.

# Scenario: a mobile manipulator in a kitchen, running a VLA at 10 Hz replanning
# while the low-level controller interpolates to a 50 Hz command stream.
import time

def run_episode(vla, robot, instruction, execute_steps=5, max_seconds=60):
    start = time.time()
    while time.time() - start < max_seconds:
        obs = robot.get_observation()                 # image(s) + joint state
        chunk = vla.predict_chunk(obs.images, obs.state, instruction)  # (H, action_dim)
        for action in chunk[:execute_steps]:          # act on the first few, then re-observe
            robot.apply_action(action)
        if robot.task_done():
            return True
    return False

Adapting a VLA to Your Robot

Out of the box, a VLA is trained on a mixture of robot embodiments, so the practical question is how to specialize it to yours. The common route is parameter-efficient fine-tuning: OpenVLA’s authors showed it can be adapted to new robot setups with LoRA, and the model can be run in lower precision using quantization so a 7B backbone fits on a single workstation GPU. Gemini Robotics reports that, with additional fine-tuning, it can learn new short-horizon tasks from as few as 100 demonstrations.

# Scenario: specialize a pretrained VLA to a new gripper using ~100 teleoperated demos.
from peft import LoraConfig, get_peft_model

lora_cfg = LoraConfig(r=32, lora_alpha=16, target_modules="all-linear", lora_dropout=0.0)
vla = get_peft_model(vla, lora_cfg)        # train adapters, keep the backbone frozen
for batch in demo_loader:                  # images, instruction, ground-truth action tokens
    loss = vla(**batch).loss               # same next-token loss used in pretraining
    loss.backward()
    optimizer.step(); optimizer.zero_grad()

The Landscape

What’s New (2025-2026)

• Steerable generalists. Physical Intelligence’s π0.7 (April 2026) reports that one model, steered with richer prompts, can match the dexterity of earlier task-specific reinforcement-learning specialists such as π*0.6, a sign that the specialist-then-generalist split is collapsing.
• Open and commercial. NVIDIA moved GR00T N1.7 from an open research release into commercial deployment by mid-2026, with N2 previewed for the end of the year, and released an open humanoid reference design built on Jetson Thor.
• On-device VLAs. Google DeepMind’s Gemini Robotics On-Device line puts the VLA on the robot itself, removing the network round trip from the latency budget above, and Gemini Robotics ER 2 (July 2026) added real-time video understanding and multi-robot collaboration.
• Unified omni models. In October 2026 Reka released a research preview of Rho-1, a 19B model that handles text, images, video, and robot actions inside one network, with no external planner or separate world model. Its pitch is that the same weights that predict the next camera frame also drive the joints, which blurs the line between a VLA and a world model. Reka reports it was trained from scratch on 320 H100s in roughly three months, and it is a research preview with known limits such as a 672x384 resolution cap and long-horizon drift.
• Learning from video. Several recent systems pretrain on human egocentric video and use inverse dynamics models to infer actions from ordinary internet footage, attacking the central bottleneck: robot action data is scarce, and web video is not.

Limitations and Open Problems

• Action data is the bottleneck. Language models train on trillions of tokens; the largest robot datasets are orders of magnitude smaller, and every robot body produces differently shaped data.
• Latency. As the widget shows, large backbones struggle to keep up with high-frequency control, which is why dual-system and on-device designs are popular.
• Safety and verification. A VLA is a black box that emits motor commands. Hard limits on force, speed, and workspace still have to live in a conventional safety layer outside the model.
• Long-horizon drift. Errors compound over many steps; models trained on short clips can wander off task during long sequences.
• Simulation gaps. Policies tuned in simulation still face the sim-to-real problem described under Physical AI, and platforms like NVIDIA Cosmos exist partly to generate the synthetic training data VLAs need.

# Scenario: a tabletop arm with one wrist/third-person camera. We hand the
# current frame plus a plain-English instruction to OpenVLA and get back a
# 7-D end-effector command (xyz delta, rotation delta, gripper).
import torch
from PIL import Image
from transformers import AutoModelForVision2Seq, AutoProcessor

processor = AutoProcessor.from_pretrained("openvla/openvla-7b", trust_remote_code=True)
vla = AutoModelForVision2Seq.from_pretrained(
    "openvla/openvla-7b",
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True,
).to("cuda:0")

image = Image.open("current_frame.png")
prompt = "In: What action should the robot take to put the carrot on the plate?\nOut:"

inputs = processor(prompt, image).to("cuda:0", dtype=torch.bfloat16)
# unnorm_key picks which robot dataset's statistics to de-normalize the output with
action = vla.predict_action(**inputs, unnorm_key="bridge_orig", do_sample=False)
print(action)  # numpy array: [dx, dy, dz, droll, dpitch, dyaw, gripper]