Visuomotor Policy Learning

Visuomotor policy learning trains a robot controller that maps camera images, usually together with the robot’s own joint or gripper state, directly to motor commands. The mapping, a policy $\pi_\theta(a_t \mid o_t)$, is learned as one network from demonstrations, from reward, or from both, instead of being assembled from a hand-built perception module that estimates object poses and a separate planner that acts on them. The technique is also called learning control from pixels, pixels-to-actions learning or end-to-end visuomotor control. Vision-based robotic manipulation introduces learned visuomotor policies among its formulations of the manipulation problem, and embodied AI places imitation and reinforcement learning in the wider setting of agents that act; this article covers how such policies are trained, how they represent actions, where they fail and how they are evaluated.

Purpose

A modular manipulation system first estimates the state of the world, for example a 6D object pose, and then plans a motion to it. Its accuracy is bounded by the perception step and by calibration between camera and robot. Levine, Finn, Darrell and Abbeel (2016) made the point with a pose-prediction baseline: its vision network located the target to about 1 cm, yet the authors write that a policy acting on those predictions succeeded only on hanging a coat hanger, the task needing the least accuracy (the manipulation article notes that their results table also gives it 8.9% and 18.3% on the toy hammer), while many of the tasks have tolerances of a few millimetres. They note that the PR2 robot they used had been reported to hold camera-to-end-effector accuracy of only about 2 cm in open-loop motion. A policy trained from pixels to torques can learn visual features for the task itself and correct its own motion from what it sees, closing the loop on the image.

The same reasoning drives later systems. QT-Opt learns closed-loop grasping from an over-the-shoulder RGB camera so that the robot can re-grasp, push objects apart and react to disturbances rather than execute one planned grasp (Kalashnikov et al., 2018). ACT is trained from images to joint targets on low-cost arms whose accuracy is 5–8 mm, for tasks such as opening a translucent condiment cup where modelling the contact and deformation precisely enough for planning would be hard (Zhao et al., 2023).

When to Use It

The technique suits tasks where the state that matters is hard to estimate or to specify by hand: transparent, deformable or cluttered objects, contact-rich motions, and tasks whose tolerances are tighter than the calibration of the camera to the robot. It needs a learning signal at scale: tens to hundreds of demonstrations per task in the imitation studies below, or hundreds of thousands of real attempts for reinforcement learning (580,000 grasps for QT-Opt).

It is a poor fit where an accurate model and reliable state estimates already exist. Visual servoing controls motion from image features with a control law whose behaviour can be analysed, and pose-based picking with a known object model, as in robotic grasping and the manipulation article, gives an interpretable intermediate result that can be checked. A learned policy offers neither: in our reading of the results below, its failures show up only in closed-loop trials, and the studies evaluate success rates over tens to hundreds of trials rather than guarantees. None of the sources draws conclusions about the safety of deployed systems, and this article draws none.

How It Works

Observations, actions and learning signals

The observation $o_t$ is one or more RGB images, from fixed cameras, a wrist camera or both, plus proprioception (joint angles, end-effector pose, gripper state). The action $a_t$ can be joint torques (Levine et al., 2016), a Cartesian displacement of the gripper with open, close and stop commands (Kalashnikov et al., 2018), absolute target joint positions tracked by a low-level PID controller (Zhao et al., 2023), or end-effector positions or velocities (Chi et al., 2023). The choice matters: Diffusion Policy performed better with position commands than with velocity commands, while the baselines it compared against did better with velocity commands (Chi et al., 2023), and ACT’s authors saw worse results with joint-position deltas than with absolute joint targets (Zhao et al., 2023).

The policy is trained by one of three signals. Behaviour cloning regresses or fits the expert’s action given the observation, from a fixed dataset of demonstrations. Reinforcement learning maximizes a reward from the robot’s own experience. Guided policy search uses a trajectory optimizer, which sees the full state during training, as the teacher of a policy that sees only images. The formal setting, a partially observed decision process, is set out in the embodied AI article.

Encoders that keep spatial information

Classification networks pool their feature maps and so discard where things are, which is what control needs. Levine et al. (2016) removed pooling and ended their convolutional encoder with a spatial soft-argmax. Each of the 32 response maps $a_{cij}$ of the last convolutional layer (109 × 109 in their network) is turned into a probability map over pixel positions,

$$s_{cij} = \frac{e^{a_{cij}}}{\sum_{i'j'} e^{a_{ci'j'}}},$$

and then into its expected image position, a feature point

$$f_{cx} = \sum_{ij} s_{cij}\, x_{ij}, \qquad f_{cy} = \sum_{ij} s_{cij}\, y_{ij}.$$

The feature points, concatenated with the robot’s configuration, pass through two fully connected layers of 40 units to the torques; the network has about 92,000 parameters, 86,000 of them in the convolutions. The expectation is a fixed linear layer, so the encoder is forced to summarize the image as a set of 2D points. On a pose-estimation task used as a proxy, the authors measured an average 3D error of 1.30 cm with soft-argmax feature points, against 2.59 cm when the expectation was replaced by a learned fully connected layer, 4.75 cm with a fully connected layer instead of softmax and expectation, and 3.71 cm with max pooling and a fully connected layer (their Table 3). The same idea survives in later policies: Diffusion Policy uses an untrained ResNet-18 whose global average pooling is replaced by spatial softmax pooling, with GroupNorm instead of BatchNorm (Chi et al., 2023), and ACT flattens ResNet-18 feature maps into tokens with 2D sinusoidal position embeddings for a transformer (Zhao et al., 2023).

End-to-end training with guided policy search

The authors note that a network of about 92,000 parameters is a major challenge for standard policy search methods, so Levine et al. (2016) split the problem. Trajectory optimization, run under full state information, fits local time-varying linear-Gaussian controllers $p_i(u_t \mid x_t)$ for each of several initial conditions; supervised learning then trains the image-based policy $\pi_\theta(u_t \mid o_t)$ to match them; and the two are alternated, with a constraint that pulls the trajectories towards behaviour the policy can reproduce. The authors frame the alternation as an instance of Bregman ADMM, from which it inherits convergence guarantees. The full state, here the pose of the target object, is available only during training (“instrumented training”): the target is held in the robot’s other gripper, which moves it through known positions, and that arm is covered with cloth so the policy cannot use its appearance as a cue for the target’s position.

Training is initialized in two parts. The convolutional layers are pretrained to regress the target’s 3D points from 1,000 images collected by moving the target, with first-layer filters taken from an ImageNet classifier; the trajectories are initialized by 15 iterations of guided policy search with a small full-state network in place of the visual policy. The motor layers are then trained alone before the whole network is trained end to end, which the authors found kept the convolutional layers from forgetting their pretrained features. The task results, including the comparison with modular baselines, are given in the manipulation article.

Behaviour cloning and compounding error

Behaviour cloning trains a policy to imitate the expert on the states the expert visited, $\hat\pi_{\text{sup}} = \arg\min_{\pi} \mathbb{E}_{s \sim d_{\pi^*}}[\ell(s, \pi)]$, where $d_{\pi}$ is the average distribution of states over $T$ steps when $\pi$ is executed and $\ell$ a surrogate loss such as the 0–1 loss with respect to the expert’s action. At test time, however, the policy meets $d_{\hat\pi}$, the distribution its own actions produce. Ross, Gordon and Bagnell (2011) state the consequence as their Theorem 2.1, citing Ross and Bagnell (2010): for any task cost bounded in $[0, 1]$ per step, if the policy errs with probability $\epsilon$ on the expert’s distribution ($\ell$ the 0–1 loss or an upper bound on it), its total cost satisfies

$$J(\hat\pi) \le J(\pi^*) + T^2 \epsilon,$$

and the bound is tight, since there are problems where the extra cost does grow quadratically in $T$. The embodied AI article gives the intuition; the quadratic term is what “compounding error” refers to.

Two earlier algorithms by Ross and Bagnell reduce the growth to linear or near-linear, on some classes of problems, by letting the learner influence the states it is trained on. Forward training learns a separate policy for each time step on the states produced by the policies before it, which is impractical for long or undefined horizons; SMILe trains a stochastic mixture of policies over many iterations. DAgger (Dataset Aggregation) keeps a single deterministic policy (Ross et al., 2011):

1. Start with an empty dataset $\mathcal{D}$ and any policy $\hat\pi_1$; with $\beta_1 = 1$ the first iteration simply collects expert demonstrations.
2. At iteration $i$, run the mixture $\pi_i = \beta_i \pi^* + (1 - \beta_i) \hat\pi_i$, which hands control to the expert a fraction $\beta_i$ of the time.
3. Ask the expert which action it would take in every state visited, add those pairs to $\mathcal{D}$, and train $\hat\pi_{i+1}$ on all of $\mathcal{D}$.
4. After $N$ iterations, return the best policy on validation.

The parameter-free special case $\beta_i = 1$ for $i = 1$ and 0 afterwards “often performs best in practice”, in the authors’ words. Their analysis treats the sequence of policies as a no-regret online learner (Follow-The-Leader) and shows, in the infinite-sample case, for a strongly convex, bounded surrogate loss that upper-bounds the 0–1 loss and mixing weights $\beta_i \le (1 - \alpha)^{i-1}$ for a constant $\alpha$, that with $N$ of order $uT$ up to logarithmic factors, some policy in the sequence satisfies $J(\hat\pi) \le J(\pi^*) + uT\epsilon_N + O(1)$ (Theorem 3.2), where $\epsilon_N$ is the loss of the best policy in hindsight on the aggregated distributions and $u$ bounds how much one deviation from the expert can increase the cost-to-go. The bound is linear in $T$ when $u$ is constant, for example when the expert recovers from mistakes within a few steps. In the authors’ Super Tux Kart experiment, a linear steering policy on downsampled images learned from a human, supervised training did not improve as more expert laps were added, SMILe still fell off the track about twice per lap after 20 iterations, and DAgger never fell off after 15 iterations.

DAgger’s price is an expert who can label arbitrary states the learner reaches. ACT’s authors note that such annotation can be time-consuming and unnatural through a teleoperation interface, and that injecting noise during demonstrations to collect corrections can make fine tasks fail (Zhao et al., 2023). Much of the later work below instead changes what the policy predicts.

Reinforcement learning from pixels

QT-Opt (Kalashnikov et al., 2018) learns grasping by reinforcement learning on real robots. Its observation is a 472 × 472 RGB image from an over-the-shoulder camera with the gripper’s open or closed status and height; its action is a gripper displacement in 3D, a change of azimuth encoded by sine and cosine, open and close commands and a termination command; and its reward is 1 when an object is held above a height at the end of the episode, checked automatically by background subtraction after the object is dropped, with a penalty of −0.05 per step and at most 20 steps. Rather than train an actor network, QT-Opt learns only a Q-function $Q_\theta(s, a)$, a convolutional network of 1.2 million parameters into which the action is fed midway, and chooses actions by maximizing it with the cross-entropy method: 64 samples, a Gaussian refitted to the best 6, two iterations. The same optimization computes the Bellman targets

$$V(s') = \min_{i=1,2} Q_{\bar\theta_i}\big(s', \arg\max_{a'} Q_{\bar\theta_1}(s', a')\big),$$

with two lagged target networks, under a cross-entropy loss because returns lie in $[0, 1]$. The authors report that actor–critic methods are often unstable and that action-convex Q-functions fit grasping poorly. The method is off-policy, so data from seven KUKA robots, collected over four months (about 800 robot hours) and bootstrapped by a scripted policy with 15–30% success, could be pooled; 1,000 “Bellman updater” jobs labelled targets asynchronously for 10 training workers.

On grasps of objects not seen in training, each of the seven robots made 102 attempts with objects returned to the bin. The policy trained on 580,000 off-policy grasps succeeded in 87%, and after fine-tuning on about 28,000 on-policy grasps in 96%; a prior self-supervised grasping method trained on 900,000 grasps from its own paper succeeded in 78%, and in 67% when trained on 400,000 grasps from QT-Opt’s data (their Table 1). In a bin-emptying protocol, five repetitions of 30 attempts on 28 test objects, the fine-tuned policy succeeded in 88% of the first 10 and 20 attempts and 76% of all 30, against 72% for the prior method over 30. The prior method is the large-scale grasping study of Levine, Pastor, Krizhevsky and Quillen, a different paper from the Levine, Finn, Darrell and Abbeel work above. The learned behaviour included regrasping, pushing objects apart and knocking a bottle over before picking it up; in dense clutter the policy sometimes regrasped repeatedly.

Image augmentation was proposed in 2020 as a simple remedy for the data inefficiency of reinforcement learning from pixels. RAD (Laskin et al., 2020) applies data augmentation to the image observations sampled for training, without changing the underlying algorithm, randomly across the batch but consistently across the stacked frames of one observation. The authors compared ten augmentations (crop, translate, window, grayscale, cutout, cutout-color, flip, rotate, random convolution and colour jitter); on the DeepMind Control walker task, random crop alone was the most effective by a large margin, and splitting crop into its parts showed that the benefit came from translation rather than from hiding image borders. On the six DeepMind Control environments of their Table 1 they report the best scores at both 100,000 and 500,000 environment steps against CURL, PlaNet, Dreamer, SAC+AE and SLAC, averaged over 10 seeds; RAD also improved generalization to unseen levels on three ProcGen games. DrQ (Data-regularized Q) is concurrent work that RAD’s authors describe as combining random cropping with regularized Q-functions in soft actor-critic. Its paper was not read for this article; in its official implementation each sampled observation is padded by 4 pixels with replicated borders and randomly cropped back to 84 × 84 (a random shift), two augmented copies of each transition are drawn, the critic target is averaged over the two copies, and the critic loss is applied to both.

Reinforcement learning can also be run offline on logged data. In robomimic’s study, BCQ performed well on data generated by reinforcement-learning agents, but both BCQ and CQL performed poorly on human demonstrations (Mandlekar et al., 2021).

Policy representations: explicit, implicit and diffusion

An explicit policy outputs actions directly, $\hat a = F_\theta(o)$, trained with mean squared error, as a mixture of Gaussians, or as a categorical distribution over discretized actions. When demonstrations contain several valid actions for the same observation, a regression to the mean averages them into an action that none of the demonstrators took.

Implicit behavioral cloning (Florence et al., 2021) represents the policy as the minimizer of a learned energy, $\hat a = \arg\min_{a} E_\theta(o, a)$. Training draws $N_{\text{neg}}$ counter-example actions $\tilde a^j$ for each sample and minimizes an InfoNCE-style loss, the negative log-likelihood of $p_\theta(a \mid o) \propto e^{-E_\theta(o, a)}$ with the counter-examples estimating the normalizer:

$$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{e^{-E_\theta(o_i, a_i)}}{e^{-E_\theta(o_i, a_i)} + \sum_{j=1}^{N_{\text{neg}}} e^{-E_\theta(o_i, \tilde a_i^j)}}.$$

Inference optimizes the action by derivative-free sampling, an autoregressive variant, or Langevin dynamics. Implicit models can fit discontinuities sharply and represent multi-valued functions, and the authors prove that an argmin over a continuous function can represent any set-valued function with a closed graph. In their real-robot pushing tasks from images, each scored over 60 trials (20 rollouts for each of 3 models), the implicit policy succeeded in 85.0%, 88.3%, 83.3% and 48.3% of the four tasks against 35.0%, 55.0%, 6.7% and 19.6% for an explicit MSE policy; on the insertion task with 1 mm tolerance the gap was an order of magnitude (their Table 6).

Diffusion Policy (Chi et al., 2023) instead learns the gradient of the action distribution. Following denoising diffusion models, a noise-prediction network $\varepsilon_\theta$ is trained with

$$\mathcal{L} = \operatorname{MSE}\big(\varepsilon^k,\ \varepsilon_\theta(O_t, A_t^0 + \varepsilon^k, k)\big),$$

where $A_t^0$ is a demonstrated action sequence, $\varepsilon^k$ noise of the variance for iteration $k$, and the observation features $O_t$ condition the network (through FiLM in the convolutional version, cross-attention in the transformer version) rather than being generated. Sampling starts from Gaussian noise and applies $K$ steps of $A_t^{k-1} = \alpha\big(A_t^k - \gamma\, \varepsilon_\theta(O_t, A_t^k, k) + \mathcal{N}(0, \sigma^2 I)\big)$. Because $\varepsilon_\theta$ approximates the negative of the score $\nabla_a \log p(a \mid o)$, which does not depend on the normalizing constant, training needs no negative samples. The authors attribute IBC’s training instability to that sampling: IBC’s evaluation success oscillated during training, which made checkpoint selection difficult. With 100 training iterations and 10 inference iterations of DDIM, inference took 0.1 s on their GPU. The manipulation article summarizes the method’s real-world Push-T result; the benchmark results are under Evidence and Impact below.

Action chunking

A policy can predict a sequence of future actions instead of one. Diffusion Policy takes the last $T_o$ observations, predicts $T_p$ actions and executes $T_a$ of them before predicting again, which the authors call receding-horizon control (Chi et al., 2023); its official configuration uses $T_o = 2$, $T_p = 16$ and $T_a = 8$, and the authors found an execution horizon of 8 steps best for most of their tasks, with too long a horizon hurting through slow reaction.

ACT (Action Chunking with Transformers; Zhao et al., 2023) makes chunking its central idea. The policy models $\pi_\theta(a_{t:t+k} \mid s_t)$ instead of $\pi_\theta(a_t \mid s_t)$: every $k$ steps it observes, predicts the next $k$ target joint positions and executes them, which divides the effective horizon of the task by $k$. Chunks also absorb behaviour that a single-step policy cannot explain from the current observation, such as pauses in human demonstrations. To avoid jerky switching between chunks, ACT can query the policy at every step and average all predictions made for the current step, a temporal ensemble with weights

$$w_i = e^{-m i}, \qquad a_t = \frac{\sum_i w_i A_t[i]}{\sum_i w_i},$$

where $A_t[i]$ are the stored predictions for step $t$ and $w_0$ weights the oldest. Unlike smoothing over neighbouring steps, it averages predictions of the same step, and it adds no training cost. The official code uses $m = 0.01$.

Human demonstrations of the same motion vary, so ACT trains its policy as the decoder of a conditional variational autoencoder. A transformer encoder, used only in training, compresses the joint positions and the demonstrated action chunk into a style variable $z$; the decoder, the policy, reads four 480 × 640 camera images through ResNet-18 backbones (1,200 tokens of 512 dimensions), the joint positions and $z$, and a transformer decoder produces a $k \times 14$ chunk of target joint positions for the two arms. The loss is an L1 reconstruction term plus a KL term weighted by $\beta$, and at test time $z$ is set to zero. The model has about 80 million parameters, is trained from scratch for each task, and controls the arms at 50 Hz on the ALOHA teleoperation system, built for under $20,000.

The ablations isolate each part, averaged over two simulated tasks with scripted or human data (their Fig. 9). Without temporal ensembling, success rose from 1% at $k = 1$ to 44% at $k = 100$ and dipped slightly towards open-loop control (42% at $k = 200$, 41% at $k = 400$); adding chunking to a CNN-MLP baseline and to the nearest-neighbour method VINN improved both in the same way. Temporal ensembling added 3.3 points for ACT and 4 for the CNN-MLP baseline but cost VINN 20 points, which the authors attribute to VINN retrieving actions from the data rather than predicting them. Removing the CVAE made almost no difference with scripted data (59% to 58%) but dropped success with human data from 35.3% to 2%.

How to Apply It

A practical recipe follows from the sources: collect demonstrations with the same cameras and action space the policy will use; choose an encoder that keeps spatial information; pick an action representation that matches the demonstrations (multimodal and paused human data call for a generative policy and chunks); train by behaviour cloning; and judge checkpoints by closed-loop rollouts, not by validation loss. The example below implements the core of this on a synthetic task small enough to train on a laptop.

Example: behaviour cloning from pixels with action chunks

The task is reaching on a 32 × 32 image: a red blob marks a point agent and a green blob its target, at continuous positions in the unit square. The expert is a proportional controller with a speed limit, $a = 0.25(g - p)$ clipped to length 0.04 per step; it is deterministic and Markovian, so a correct label exists for every state the learner can reach. Two networks with the same spatial soft-argmax encoder are trained on 200 noise-free expert episodes of 50 steps (10,000 images): a single-step policy predicting $a_t$, and a chunk policy predicting $a_{t:t+16}$. The chunk policy is executed with execution horizons $T_a$ of 1, 4, 8 and 16 steps and with ACT’s temporal ensemble, so that the prediction target and the execution scheme are varied separately. Each policy runs closed loop for 60 steps from 200 fixed test starts, scored by success (final distance below 0.04, about 1.3 pixels) and mean final distance, under three conditions: clean; execution noise of 0.01 per axis per step; and a latency of 3 steps, in which the policy sees the image from three steps earlier and the chunk variants execute the entries of the chunk that belong to the current step, as Diffusion Policy’s low-dimensional robomimic evaluation code does to simulate latency. A last block measures the distribution shift: the policy’s action error against expert labels on held-out expert states and on the states it visits itself, and its distance from the expert’s trajectory under the same noise. Three seeds (0, 1, 2), each with its own demonstrations and initialization, are looped over; ranges are minimum and maximum over the seeds.

The example uses PyTorch 2.14.1 and NumPy 2.5.4, and took about 4.5 minutes on a laptop CPU. Before the first run the planned network was resized after timing one epoch (a stride of 2 in the first convolution and 12 instead of 30 epochs). The first run crashed after training the first seed because two network calls in the distribution-shift block were outside torch.no_grad(); they were wrapped in it, with no other change. After the first complete run, one printed column was added, the share of test episodes that end within 0.05 of the workspace border, to show where failures end; data, training, rollouts and the other printed values are unchanged, and the run with the added column reproduced the earlier values exactly.

import numpy as np
import torch
import torch.nn as nn

torch.set_num_threads(4)
H, TP, T_TEST, VMAX = 32, 16, 60, 0.04
c = (np.arange(H) + 0.5) / H
YY, XX = np.meshgrid(c, c, indexing="ij")


def render(p, g):  # agent: red blob, target: green blob, 32x32 RGB
    img = np.zeros((len(p), 3, H, H), np.float32)
    for ch, q, s in [(0, p, 1.0 / H), (1, g, 1.5 / H)]:
        img[:, ch] = np.exp(-((XX - q[:, 0, None, None]) ** 2
                              + (YY - q[:, 1, None, None]) ** 2) / (2 * s * s))
    return img


def expert(p, g):  # proportional controller with a speed limit
    a = 0.25 * (g - p)
    n = np.linalg.norm(a, axis=1, keepdims=True)
    return a * np.minimum(1.0, VMAX / np.maximum(n, 1e-12))


def sample_pairs(rng, n):  # start and target at least 0.3 apart
    s, g = rng.uniform(0.1, 0.9, (2, n, 2))
    bad = np.linalg.norm(s - g, axis=1) < 0.3
    while bad.any():
        s[bad], g[bad] = rng.uniform(0.1, 0.9, (2, bad.sum(), 2))
        bad = np.linalg.norm(s - g, axis=1) < 0.3
    return s, g


def demos(rng, n_ep, T=50):  # noise-free expert episodes, with 16-step action chunks
    s, g = sample_pairs(rng, n_ep)
    P, A, p = [], [], s.copy()
    for t in range(T + TP):
        a = expert(p, g); P.append(p.copy()); A.append(a); p = p + a
    P, A = np.stack(P, 1), np.stack(A, 1)               # (episode, time, 2)
    chunks = np.stack([A[:, t:t + TP] for t in range(T)], 1)
    G = np.repeat(g[:, None], T, 1)
    return P[:, :T].reshape(-1, 2), G.reshape(-1, 2), chunks.reshape(-1, TP, 2)


class Policy(nn.Module):  # conv layers -> spatial soft-argmax keypoints -> MLP
    def __init__(self, k):
        super().__init__()
        self.k = k
        self.conv = nn.Sequential(nn.Conv2d(3, 16, 5, stride=2), nn.ReLU(), nn.Conv2d(16, 16, 3),
                                  nn.ReLU(), nn.Conv2d(16, 16, 3))
        n = (H - 5) // 2 + 1 - 4                        # 10x10 feature maps
        g = (torch.arange(n) + 0.5) / n
        self.register_buffer("gx", g.repeat(n)); self.register_buffer("gy", g.repeat_interleave(n))
        self.mlp = nn.Sequential(nn.Linear(32, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU(),
                                 nn.Linear(64, 2 * k))

    def forward(self, x):
        f = self.conv(x).flatten(2).softmax(-1)          # (batch, 16, 10*10)
        kp = torch.cat([f @ self.gx, f @ self.gy], 1)     # expected x, y per channel
        return self.mlp(kp).view(-1, self.k, 2) * VMAX


def train(k, P, G, A, seed, epochs=12):
    torch.manual_seed(seed)
    net, X = Policy(k), torch.from_numpy(render(P, G))
    Y = torch.from_numpy(A[:, :k].astype(np.float32))
    opt = torch.optim.Adam(net.parameters(), 1e-3)
    for ep in range(epochs):
        for idx in torch.randperm(len(X)).split(128):
            loss = (((net(X[idx]) - Y[idx]) / VMAX) ** 2).mean()
            opt.zero_grad(); loss.backward(); opt.step()
    return net.eval()


@torch.no_grad()
def rollout(policy, s, g, ta=1, ens=False, sigma=0.0, delay=0, m=0.01):
    """Closed loop. policy(image) -> (n, k, 2) actions; execute ta of them per query.
    ens: query every step and average all predictions for the current step (ACT)."""
    rn = np.random.default_rng(2000)                     # same noise for every method
    n, p, hist, visited = len(s), s.copy(), [s.copy()], []
    buf, plan, start, off = {}, None, 0, 0
    for t in range(T_TEST):
        seen = hist[max(0, t - delay)]                    # image from `delay` steps ago
        if policy == "expert":
            a = expert(p, g)
        else:
            k = 0 if plan is None else plan.shape[1]
            if ens or plan is None or t - start >= min(ta, k - off):
                plan = policy(torch.from_numpy(render(seen, g))).numpy()
                q = max(0, t - delay)                     # step the image belongs to
                off = (t - q) if plan.shape[1] > 1 else 0  # chunks skip the stale part
                if ens:
                    for j in range(plan.shape[1]):
                        buf.setdefault(q + j, []).append(plan[:, j])
                start = t
            if ens:
                preds = np.stack(buf[t])                  # all predictions for step t
                w = np.exp(-m * np.arange(len(preds)))[:, None, None]   # w_0: oldest
                a = (w * preds).sum(0) / w.sum()
            else:
                a = plan[:, t - start + off]
        visited.append(p.copy())
        p = np.clip(p + a + sigma * rn.standard_normal(p.shape), 0, 1)
        hist.append(p.copy())
    return np.linalg.norm(p - g, axis=1), np.stack(visited, 1), np.stack(hist, 1)


s_test, g_test = sample_pairs(np.random.default_rng(1000), 200)
conds = {"clean": dict(), "noise": dict(sigma=0.01), "latency": dict(delay=3)}
variants = ["single-step", "chunk Ta=1", "chunk Ta=4", "chunk Ta=8", "chunk Ta=16",
            "chunk + ensemble"]
res = {(c_, v): [] for c_ in conds for v in variants}
shift = {"single-step": [], "chunk Ta=8": []}
for seed in [0, 1, 2]:
    rng = np.random.default_rng(seed)
    P, G, A = demos(rng, 200)
    Ph, Gh, Ah = demos(rng, 50)                          # held-out expert states
    nets = {1: train(1, P, G, A, seed), TP: train(TP, P, G, A, seed)}
    for cname, kw in conds.items():
        for v in variants:
            k = 1 if v == "single-step" else TP
            ta = 1 if v in ("single-step", "chunk + ensemble") else int(v.split("=")[1])
            d, vis, traj = rollout(nets[k], s_test, g_test, ta=ta, ens=v.endswith("ensemble"), **kw)
            edge = np.mean((np.minimum(traj[:, -1], 1 - traj[:, -1]) < 0.05).any(1)) * 100
            res[cname, v].append((np.mean(d < 0.04) * 100, d.mean(), edge))
            if cname == "noise" and v in shift:          # distribution shift (DAgger's point)
                _, _, tex = rollout("expert", s_test, g_test, **kw)
                net, gg = nets[k], np.repeat(g_test[:, None], T_TEST, 1).reshape(-1, 2)
                vv = vis.reshape(-1, 2)
                with torch.no_grad():
                    on_exp = net(torch.from_numpy(render(Ph, Gh)))[:, 0].numpy()
                    on_own = net(torch.from_numpy(render(vv, gg)))[:, 0].numpy()
                shift[v].append((np.sqrt(((on_exp - Ah[:, 0]) ** 2).sum(1).mean()),
                                 np.sqrt(((on_own - expert(vv, gg)) ** 2).sum(1).mean()),
                                 [np.linalg.norm(traj[:, t] - tex[:, t], axis=1).mean()
                                  for t in (5, 10, 20, 40)]))

for cname, kw in conds.items():
    d, _, _ = rollout("expert", s_test, g_test, **kw)
    print(f"[{cname}]  expert (true state): success {np.mean(d < 0.04) * 100:5.1f}%  "
          f"final dist {d.mean():.4f}")
    for v in variants:
        r = np.array(res[cname, v])
        print(f"  {v:17s} success {r[:, 0].mean():5.1f}% [{r[:, 0].min():5.1f}, {r[:, 0].max():5.1f}]"
              f"  final dist {r[:, 1].mean():.4f}  ends at edge {r[:, 2].mean():5.1f}%")
print("[noise] action RMSE on expert states vs on own states; "
      "distance to expert trajectory at t = 5, 10, 20, 40")
for v, rows in shift.items():
    e, o = np.mean([r[0] for r in rows]), np.mean([r[1] for r in rows])
    dev = np.mean([r[2] for r in rows], 0)
    print(f"  {v:17s} {e:.4f} vs {o:.4f}   " + "  ".join(f"{x:.4f}" for x in dev))

[clean]  expert (true state): success 100.0%  final dist 0.0000
  single-step       success  39.7% [ 17.5,  57.5]  final dist 0.1846  ends at edge  57.3%
  chunk Ta=1        success  35.7% [ 21.5,  49.0]  final dist 0.1731  ends at edge  51.3%
  chunk Ta=4        success  45.3% [ 28.0,  62.5]  final dist 0.1414  ends at edge  42.5%
  chunk Ta=8        success  50.0% [ 27.5,  70.5]  final dist 0.1180  ends at edge  35.3%
  chunk Ta=16       success  62.0% [ 32.5,  86.5]  final dist 0.0761  ends at edge  18.8%
  chunk + ensemble  success  60.7% [ 36.0,  82.5]  final dist 0.0784  ends at edge  20.2%
[noise]  expert (true state): success  97.5%  final dist 0.0188
  single-step       success  21.7% [ 12.0,  32.0]  final dist 0.2533  ends at edge  74.8%
  chunk Ta=1        success  19.2% [ 14.0,  22.0]  final dist 0.2264  ends at edge  67.3%
  chunk Ta=4        success  20.2% [ 14.0,  23.5]  final dist 0.2030  ends at edge  60.8%
  chunk Ta=8        success  23.8% [ 16.5,  30.0]  final dist 0.1747  ends at edge  53.0%
  chunk Ta=16       success  17.8% [ 13.5,  22.0]  final dist 0.1344  ends at edge  36.3%
  chunk + ensemble  success  20.3% [ 15.0,  25.5]  final dist 0.1396  ends at edge  38.8%
[latency]  expert (true state): success 100.0%  final dist 0.0000
  single-step       success  21.5% [ 10.0,  36.0]  final dist 0.2645  ends at edge  77.5%
  chunk Ta=1        success  49.8% [ 28.5,  72.0]  final dist 0.1272  ends at edge  38.7%
  chunk Ta=4        success  52.8% [ 28.0,  76.0]  final dist 0.1099  ends at edge  33.2%
  chunk Ta=8        success  60.5% [ 36.5,  83.5]  final dist 0.0817  ends at edge  21.3%
  chunk Ta=16       success  66.3% [ 45.5,  88.0]  final dist 0.0641  ends at edge  14.3%
  chunk + ensemble  success  67.5% [ 49.0,  87.5]  final dist 0.0584  ends at edge  12.0%
[noise] action RMSE on expert states vs on own states; distance to expert trajectory at t = 5, 10, 20, 40
  single-step       0.0033 vs 0.0497   0.0191  0.0257  0.0430  0.1917
  chunk Ta=8        0.0045 vs 0.0387   0.0223  0.0366  0.0394  0.1082

The expert succeeds in every clean episode, and no learned policy comes close: the single-step policy reaches the target in 39.7% of episodes on average over the three seeds, and more than half of its episodes end within 0.05 of the border of the workspace. The demonstrations never go there, since starts and targets lie in $[0.1, 0.9]^2$ and the expert moves straight to the target. The last block shows the mechanism Ross et al. (2011) describe. On held-out expert states the single-step policy’s action error is 0.0033, under a tenth of the speed limit; on the states it visits itself under noise it is 0.0497, fifteen times larger and above the speed limit; in our reading, because those states lie outside what the demonstrations cover, and the policy’s errors there take it further away. Its distance from the expert’s trajectory grows slowly to 0.043 at step 20 and then to 0.19 at step 40, so most of the drift happens late, after the expert has finished or nearly finished its approach (a start 0.3 to 1.1 from the target takes 8 to 28 steps at the speed limit). The spread over seeds is wide, 17.5% to 57.5% for the single-step policy, so only large differences and consistent trends in the table mean anything.

The plan’s hypotheses were that chunking would have nothing to fix on this Markovian, noise-free task, and that under noise success would fall as the execution horizon grows, with $T_a = 16$ clearly worst. The output contradicts the first; of the second, only the lowest mean success for $T_a = 16$ under noise is borne out, and not clearly, since the seed ranges overlap. With clean execution, predicting chunks but re-planning every step ($T_a = 1$, 35.7%) is no better than the single-step policy, so the chunk target alone does not help; what helps is executing more of each chunk before looking again. Success rises from 35.7% at $T_a = 1$ to 62.0% at $T_a = 16$, and the share of episodes ending at the border falls from 51.3% to 18.8%; temporal ensembling, which queries every step but averages up to 16 predictions per step, matches the longest chunks. In our reading this is ACT’s effective-horizon argument appearing for a different reason than in the paper: a policy that is queried less often has fewer occasions to issue a drifting action from a state it handles poorly, and averaging overlapping predictions damps each one. It is not the non-Markovian, multimodal human data that motivated ACT, which this deterministic expert does not produce.

Under execution noise every variant falls to 18–24% success, against 97.5% for the expert under the same noise. Longer chunks again end at the border less often and closer to the target on average, but their success does not rise; in our reading, because an open-loop chunk cannot correct the noise near the target, where the success threshold of 0.04 is about twice the expert’s own mean final distance under noise. With 3 steps of latency the single-step policy, which can only act on a stale image, drops from 39.7% to 21.5%, while the chunk variants, which execute the entries meant for the current step, keep or exceed their clean results; the output does not explain why they exceed them, and it should not be read as latency helping. Because the expert is deterministic and Markovian and the networks are trained by mean squared error, the example cannot show what ACT’s CVAE and Diffusion Policy’s denoising add on human demonstrations: an MSE-trained chunk would average distinct demonstrated behaviours rather than choose one, as ACT’s ablation without the CVAE suggests.

Variants

• Behaviour cloning with history. BC-RNN, a recurrent policy with a Gaussian-mixture output, was a strong baseline on robomimic’s human data (Mandlekar et al., 2021); Diffusion Policy reproduces it as LSTM-GMM.
• Interactive imitation. DAgger and its relatives query an expert on the learner’s own states (Ross et al., 2011).
• Guided policy search. A full-state trajectory optimizer teaches an image-based policy (Levine et al., 2016).
• Reinforcement learning from pixels, online (QT-Opt; RAD and DrQ in simulation) or offline from logged data, as in robomimic’s BCQ and CQL comparisons.
• Generative action models. Mixture models, discretized actions with offsets (BeT), energy-based implicit policies (IBC), conditional VAEs over chunks (ACT) and diffusion over action sequences (Diffusion Policy).
• Large pretrained policies. Vision-language-action models fine-tune vision–language models to output actions for many tasks and robots; RT-1 and RT-2 are summarized in the embodied AI article.
• Transfer from simulation. Policies trained in simulation with randomized rendering or dynamics and then run on real robots, the subject of sim-to-real transfer.
• Driving. End-to-end autonomous driving applies behaviour cloning from sensors to a vehicle, with its own history from ALVINN and its own closed-loop benchmarks.

Best Practices

• Evaluate in closed loop, and select checkpoints the same way. In robomimic, choosing a policy by lowest validation loss or by the last checkpoint gave policies 10% to 100% worse than the best checkpoint found by rollouts (Mandlekar et al., 2021). Diffusion Policy reports the average of its last 10 checkpoints, adding the best checkpoint on robomimic and Push-T to match those benchmarks’ original protocols (Chi et al., 2023).
• Model temporal structure. History-dependent models (BC-RNN, HBC, IRIS) were particularly effective on human data, and BC-RNN’s gap over plain BC grew with task length and with several demonstrators (Mandlekar et al., 2021). Action sequences address the same issue from the output side (Zhao et al., 2023; Chi et al., 2023).
• Use a policy that can represent several modes when demonstrations come from people: removing the GMM output hurt BC-RNN on multi-human data (Mandlekar et al., 2021), and removing the CVAE nearly erased ACT’s success on human data (Zhao et al., 2023).
• Shift-augment images and add a wrist camera. Removing pixel-shift randomization cut robomimic’s image policies by 47% and 35% relative on Square and Transport, removing the wrist camera by 9% and 43%; on the real Can task, removing either lowered success from 73.3% to 26.7% and 43.3% respectively (Mandlekar et al., 2021). Random crop and translation also dominated RAD’s comparison (Laskin et al., 2020).
• Keep the observation lean. Adding end-effector velocities or joint states hurt low-dimensional robomimic agents by 49–88% relative (Mandlekar et al., 2021).
• Tune learning rate and capacity for image policies. Raising the learning rate from 1e-4 to 1e-3 cut image agents by 35–63%, and a shallow convolutional encoder instead of a ResNet by 25–62% (Mandlekar et al., 2021).
• Prefer position actions with chunked policies. Diffusion Policy did better with position control and was robust to latency of up to 4 steps with it, while velocity control suffered more from latency (Chi et al., 2023).
• Remove spurious cues from the training scene. Levine et al. (2016) covered the arm that held the target so its appearance could not stand in for the target’s position.

Pitfalls

• Compounding error. A policy accurate on the expert’s states can drift into states it never saw, where its errors grow (Ross et al., 2011); the example shows this on a toy task.
• Averaging multimodal demonstrations. A regression policy averages distinct valid actions; Diffusion Policy shows LSTM-GMM and IBC biased to one side of the Push-T block and BeT switching between sides within a rollout, while its own policy committed to one (Chi et al., 2023).
• Idle actions. Pauses in teleoperated demonstrations can make single-step policies stall; BC-RNN and IBC often got stuck in Diffusion Policy’s real-world experiments unless idle actions were removed (Chi et al., 2023).
• Open-loop chunks. Executing long chunks without re-observing makes the policy slow to react; ACT’s success dipped at the longest chunks and Diffusion Policy found an intermediate horizon best (Zhao et al., 2023; Chi et al., 2023).
• Unstable training of implicit policies, which forces evaluating many checkpoints (Chi et al., 2023).
• Offline reinforcement learning on human data. BCQ (44.7%) and CQL (6.0%) fell below plain BC (64.0%) even on a diagnostic dataset with one successful and one failed demonstration per start (Mandlekar et al., 2021).
• Distractors and backgrounds. Levine et al. (2016) report moderate tolerance to distractors separated from the target and poor performance under drastic background changes, as the manipulation article notes.
• Comparing numbers across set-ups. Success rates depend on the checkpoint rule, the number of rollouts, the action space and the dataset; the benchmarks below each fix these differently.

Evidence and Impact

Benchmarks and their protocols

Benchmark	Content	Demonstrations	Protocol in the source
robomimic (Mandlekar et al., 2021)	5 simulated tasks (Lift, Can, Square, Transport, Tool Hang) and 3 real ones	PH: 200 by one experienced operator; MH: 300 by 6 operators of three proficiency levels; MG: rollouts of RL agents (Lift, Can)	Every policy evaluated with 50 rollouts at fixed epoch intervals; best success over training, mean over 3 seeds
Diffusion Policy suite (Chi et al., 2023)	robomimic tasks, Push-T, Block Pushing, Franka Kitchen; 4 real tasks	As in each source benchmark	Mean of last 10 checkpoints (also the best checkpoint on robomimic and Push-T), 3 seeds × 50 initial conditions; Push-T scored by target coverage
ACT tasks (Zhao et al., 2023)	2 simulated bimanual tasks (Transfer Cube, Bimanual Insertion), 6 real ALOHA tasks	50 per task (100 for Thread Velcro), scripted or human in simulation	Simulation: 3 seeds × 50 evaluations; real: 1 seed × 25 trials; success per subtask
IBC tasks (Florence et al., 2021)	D4RL human-expert tasks, particle, pushing and sweeping tasks, 4 real pushing tasks	D4RL’s own datasets; 50 to 2,000 per task in the other simulated tasks; 95–502 for the real tasks	3 seeds; real tasks 20 rollouts per seed
LIBERO (Liu et al., 2023)	130 procedurally generated tasks in four suites: Spatial, Object and Goal (10 tasks each) and LIBERO-100 (split into 90 short and 10 long tasks)	50 human-teleoperated demonstrations per task	Lifelong learning, scored from success rates by forward transfer, negative backward transfer and area under the success curve

robomimic’s image observations replace the ground-truth object poses of its “low-dim” observations with camera images, keeping proprioception. With images, BC-RNN reached 100% on Lift (PH), 98.0% on Can (PH), 82.0% on Square (PH), 72.0% on Transport (PH) and 67.3% on Tool Hang (PH), against 62.0%, 55.3% and 20.0% for plain BC on the last three (their Table 3); on low-dimensional observations its Transport (PH) score was 71.3%, and the authors note that image policies kept nearly the same performance as low-dimensional ones except on Transport (MH), 42.0% against 65.3%. Training on 50% or 20% of the data hurt Square and Transport substantially, while Lift and Can still reached 75–100% with 20%. On the real robot, with no hyperparameter tuning, BC-RNN reached 96.7% on Lift, 73.3% on Can and 3.3% on Tool Hang over 30 rollouts of the final checkpoint.

Diffusion Policy’s image-based results on robomimic, in the format best checkpoint / mean of last 10, include 0.98/0.92 (CNN version) on Square (PH) against 0.82/0.59 for LSTM-GMM and 0.03/0.00 for IBC, and 0.89/0.69 on Transport (MH) against 0.44/0.24 for LSTM-GMM (their Table 2). The authors report an average improvement of 46.9% over the prior state of the art; the abstract attaches it to all 12 tasks from four benchmarks, simulated and real, and the evaluation section to the simulation tables. LSTM-GMM’s numbers there come from the authors’ reproduction of robomimic’s BC-RNN, which for the state-based table they report as slightly better than the original paper’s; in our reading they should therefore not be mixed with robomimic’s own table (Transport (PH) with images is 0.88 in the reproduction against 72.0% in robomimic). The paper also gives two different sizes for the Franka Kitchen demonstration set: 566 demonstrations in the text and 656 in its task table (Table III).

ACT, on its two simulated tasks, succeeded in 86% and 50% of Transfer Cube trials with scripted and human data, and 32% and 20% of Bimanual Insertion trials, where the best of four baselines (a CNN-MLP, BeT, RT-1 and VINN) reached 27%, 1%, 3% and 0% (their Table II). On the real tasks it reached 88% on Slide Ziploc and 96% on Slot Battery, where the baselines made no progress past the first subtask, and 84% (Open Cup), 20% (Thread Velcro), 64% (Prep Tape) and 92% (Put On Shoe) where BeT reached 0%, from 50 demonstrations per task (100 for Thread Velcro), 10–20 minutes of data.

Impact

Levine et al. (2016) showed that a deep network trained from raw images to torques could outperform a modular pose-then-act pipeline on tasks with millimetre tolerances, and their spatial soft-argmax encoder reappears in Diffusion Policy (2023). QT-Opt showed that off-policy reinforcement learning could scale to hundreds of thousands of real grasps and produce closed-loop behaviours a pick-pose predictor does not have. RAD showed that image augmentation alone could make reinforcement learning from pixels much more data-efficient, and robomimic found pixel-shift augmentation similarly important for imitation. In our reading of the sources, from 2021 the emphasis moved to imitation from human demonstrations with expressive policies: implicit models, then diffusion over action sequences and action chunking, which address the multimodality and temporal structure of human data that plain behaviour cloning handles poorly. Generative heads that output action chunks reappear in later vision-language-action models such as π0, which generates chunks of 50 actions by flow matching, as the embodied AI article describes.

These results are the authors’ reports from their own set-ups, each with its own robots, objects, demonstrations, action spaces and checkpoint rules, and with tens of real trials per condition. They show that the methods work under those conditions; they do not establish how the methods compare on a common real-world benchmark, which none of these sources provides.