Sim-to-Real Transfer

Sim-to-real transfer is the practice of training a model in simulation, a perception network on rendered images or a control policy on simulated physics, and then running it on a real robot or on real camera images. It is also written sim2real, and the closely related use of rendered images to train detectors and pose estimators is usually called training on synthetic data. Simulation gives unlimited labeled data, free resets and safe failure; the cost is that the model was trained on a world that differs from the one it will meet. Methods for sim-to-real transfer either make the simulator more faithful, make the model indifferent to the differences, or use some real data to bridge them.

Embodied AI introduces the subject in a short section, with Tobin et al.’s domain randomization, Peng et al.’s dynamics randomization and the measurements of the gap in RoboTHOR and Habitat, and shows a small classification example on rendered shapes; this article assumes that section and goes further into the mechanisms, the alternatives and the evidence. Robot perception is the hub for how robots sense, and data augmentation describes the transformations that randomization extends from one domain to the gap between two.

Purpose

The problem is a distribution shift with a particular cause. A simulator is a model, and every model leaves things out. Tobin et al. (2017) list what physics simulators miss: non-rigidity, gear backlash, wear and tear and fluid dynamics, which even strong system identification cannot capture, and add that image renderers are often unable to reproduce the richness and noise of real cameras. These differences, collectively, are what they call the reality gap. Tan et al. (2018) name unmodeled dynamics, wrong simulation parameters and numerical errors as causes, and argue that the gap is amplified in locomotion, where frequent changes of contact can turn a small discrepancy into a qualitatively different outcome.

The phrase is older than deep learning: it is in the title of Jakobi, Husbands and Harvey’s 1995 paper on simulation in evolutionary robotics, which Tan et al. cite for the idea of injecting noise in simulation to make a controller more robust. That paper could not be read for this article.

Two things make the gap worse for learned systems than for hand-designed ones. A network uses whatever features predict its training labels, including features that exist only in the renderer, such as a perfectly uniform background or a light that never moves. And a policy trained by reinforcement learning optimizes against the simulator itself: Peng et al. (2018) note that such policies are prone to exploiting the simulator’s idiosyncrasies to realize behaviours that are infeasible in the real world. The purpose of sim-to-real methods is to keep what simulation makes cheap, which is labels, interaction and resets, while removing what the model can learn only from the simulator’s peculiarities.

When to Use It

Simulation is worth its gap when real data is the bottleneck:

• Labels are expensive or impossible to collect by hand. Dense depth, 3D positions and 6-DoF poses are hard to annotate; Tremblay, To et al. (2018) note that labeled 3D detection data is almost impossible to produce manually and that they knew of no real training data for pose estimation that included extreme lighting.
• Learning needs many trials. Reinforcement learning may need millions of samples, and random exploration can be dangerous on hardware (Tobin et al., 2017); Tan et al. (2018) add that every fall during learning can damage a legged robot, and that resets are hard to automate.
• Rare or dangerous situations must be covered. CAD2RL trained a collision-avoidance policy for indoor flight, where trial-and-error learning is impractical (Sadeghi & Levine, 2017).

It helps less, or not at all, when real data is plentiful and cheap for the task, when the factor that decides success is one the simulator cannot model (contact with deformable objects, reflective materials, the behaviour of people), or when the evaluation that matters cannot be done in simulation either. A model that transfers on one robot, in one room, is not shown to transfer to another; most published results come from a single laboratory setting, as the Evidence section shows.

How It Works

The objective: one simulator or many

Training in one simulator with fixed parameters $\mu$ optimizes performance in that simulator. Peng et al. (2018) write the randomized alternative as an expectation over a distribution $\rho_\mu$ of simulator parameters:

$$\max_\pi \; \mathbb{E}_{\mu \sim \rho_\mu} \left[ \mathbb{E}_{\tau \sim p(\tau \mid \pi, \mu)} \left[ \sum_{t=0}^{T-1} r(s_t, a_t) \right] \right],$$

where $\tau$ is a trajectory of states $s_t$ and actions $a_t$ produced by policy $\pi$ in the simulator with parameters $\mu$, and $r$ is the reward. For a perception model the same idea replaces the return with a supervised loss averaged over rendering parameters: textures, lights, camera pose, noise. The hope, in Tobin et al.’s words, is that with enough variability the real world appears to the model as just another variation. Whether it does depends on whether the real conditions fall inside, or near, the range that $\rho_\mu$ covers, which is a property of the chosen ranges rather than of the method.

The other families change what is trained on rather than how widely:

• System identification narrows $\rho_\mu$ toward the true parameters, by measuring or fitting them, so that one accurate simulator suffices.
• Domain adaptation uses data from the target domain, usually unlabeled real images, to align the two domains, either in the network’s features or in pixel space.
• Real-to-sim translation maps real observations into the simulator’s appearance at test time, so that the model never has to leave the domain it was trained on.
• Fine-tuning continues training on a small amount of labeled real data after pretraining in simulation, as in transfer learning.

These are complementary, and the stronger published systems combine several.

Visual domain randomization

Visual randomization varies the rendering. Tobin et al. (2017) randomized, for every training image, the number and shape of distractor objects, the position and texture of every object, the textures of the table, floor, skybox and robot, the camera’s position, orientation and field of view, the number of lights and their position, orientation and specular properties, and the type and amount of image noise. Their textures were deliberately non-realistic: a random colour, a gradient between two random colours, or a checkerboard of two. The camera was placed randomly within a 10 × 5 × 10 cm box around a hand-placed starting point that roughly matched the real camera, pointed at a fixed point on the table and then offset by up to 0.1 rad, with the field of view scaled by up to 5%. Because the camera is randomized, the method needs no precise calibration of the real one.

Sadeghi and Levine’s CAD2RL (2017), which Tobin et al. describe as the work most similar to theirs, built hallway models in Blender, nine of them for training, and textured the walls with randomly chosen materials (wood, metal, textile, carpet, stone, glass), lit by randomly placed lights, and rendered from a camera at random heights and orientations. Their reasoning is the core of the method: the wider the parameters vary in simulation, the more likely it is that properties of the real world are captured somewhere in the set of scenes considered.

Tremblay, Prakash et al. (2018) extended randomization to detecting real objects in front of real backgrounds. They placed car models from a set of 36 in a 3D scene at random positions and orientations, added a random number of geometric shapes they call flying distractors, applied random textures to cars and distractors, inserted 1 to 12 point lights, rendered from a random camera, and composited the result over a random photograph. Both textures and backgrounds came from the Flickr 8K dataset.

Dynamics randomization

Dynamics randomization varies the physics: masses, friction, damping, controller gains, latency, sensor noise. The parameters are sampled at the start of each training episode. Peng et al. (2018) held mass and damping fixed during an episode but resampled the time between actions and the observation noise at every step, modelling controller latency and sensor noise. Tan et al. (2018) randomized ten quantities for a quadruped (their Table I), including mass (80–120% of the measured value), inertia (50–150%), motor strength (80–120%), control step (3–20 ms), latency (0–40 ms), battery voltage, contact friction (0.5–1.25) and IMU bias and noise, choosing narrow ranges for what they had measured and wider ones for what they had only estimated, such as inertia.

A policy that sees many dynamics during training still has to act well under the one it meets. Peng et al. contrast two ways to give it the means. One is explicit online system identification, a module that predicts the parameters $\hat\mu$ from a history of states and actions and feeds them to a policy $\pi(a_t \mid s_t, \hat\mu)$, an approach they attribute to Yu et al. The other, which they use, embeds identification implicitly in a recurrent policy $\pi(a_t \mid s_t, z_t, g)$, whose internal memory $z_t$ summarizes past states and actions, for goal $g$; its representation is learned end to end without choosing which parameters to infer.

Randomization has a price that Tan et al. measured. Controllers trained with randomization had a lower mean and a lower standard deviation of return across test environments in simulation: lower peak performance, more consistent behaviour. In their words randomization “is not a free meal”, and they recommend using it only when necessary.

System identification and simulator fidelity

System identification tunes the simulator’s parameters to match the physical system. Tobin et al. (2017) call it time-consuming and error-prone, which is one reason for randomizing instead; Tan et al. (2018) show what it can achieve when done carefully. They disassembled a Minitaur quadruped, measured link dimensions and masses and centres of mass, estimated inertia from shape and mass assuming uniform density, and measured motor friction. They then added two models the simulator lacked:

• An actuator model. The simulator’s position control satisfied a constraint at the end of each time step, which allowed gains that were stable in simulation but oscillated on the robot. They replaced it with a model of an ideal DC motor driven by pulse-width modulation, $\tau = K_t I$ with $I = (V_{\text{pwm}} - V_{\text{emf}})/R$ and $V_{\text{emf}} = K_t \dot q$, and then a piecewise-linear torque–current curve, because real torque saturates as current grows.
• Latency. The simulator applied commands and reported sensor readings instantaneously, which makes the stable region of a feedback controller much larger than on hardware. They measured the latencies on the robot (3 ms for the microcontroller’s servo loop, typically 15–19 ms for the policy on the onboard computer) and reproduced them in simulation by interpolating delayed observations.

They report that without either the actuator model or the latency model the learned controllers did not work on the real robot. Their third measure was a compact observation space, discussed under Evidence.

Domain adaptation

Domain adaptation, as James et al. (2019) summarize it, lets a model trained on a source domain generalize to a target domain by using existing but mostly unlabeled target data; in sim-to-real transfer the source is the simulator and the target is the real world. They divide methods into feature-level adaptation, which learns features that do not distinguish the domains, and pixel-level adaptation, which re-stylizes source images to look like target images.

Feature level: domain-adversarial training. Ganin et al. (2016) start from the theory that effective transfer needs predictions based on features that cannot discriminate between source and target. Their domain-adversarial neural network (DANN) has a feature extractor $G_f$ with parameters $\theta_f$, a label predictor $G_y$ with $\theta_y$, and a domain classifier $G_d$ with $\theta_d$. With $n$ labeled source examples, $n'$ unlabeled target examples, label loss $L_y$ and domain loss $L_d$, training seeks a saddle point of

$$E(\theta_f, \theta_y, \theta_d) = \frac{1}{n}\sum_{i=1}^{n} L_y^i(\theta_f, \theta_y) \;-\; \lambda \left( \frac{1}{n}\sum_{i=1}^{n} L_d^i(\theta_f, \theta_d) + \frac{1}{n'}\sum_{i=n+1}^{N} L_d^i(\theta_f, \theta_d) \right),$$

minimized over $\theta_f, \theta_y$ and maximized over $\theta_d$. The domain classifier learns to tell the domains apart, while the feature extractor learns to predict labels and to defeat the classifier. Ganin et al. implement this with ordinary backpropagation by inserting a gradient reversal layer between $G_f$ and $G_d$: the identity in the forward pass, multiplication of the gradient by $-1$ (scaled by $\lambda$) in the backward pass. In their image experiments $\lambda$ rose from 0 to 1 during training as

$$\lambda_p = \frac{2}{1 + \exp(-\gamma p)} - 1, \qquad \gamma = 10,$$

with $p$ the training progress from 0 to 1, a schedule they say was not tuned and that makes the domain classifier less sensitive to noise early in training. Half of each batch was labeled source data and half unlabeled target data.

Two of their experiments are synthetic-to-real. Trained on about 500,000 synthetic digit images rendered from fonts (Syn Numbers) and tested on the Street View House Numbers set, a source-only network scored 0.8674 accuracy, DANN 0.9109 and a network trained on labeled target data 0.9220, so DANN covered 79.7% of the gap; on 100,000 synthetic traffic signs tested on GTSRB, the figures were 0.7900, 0.8865 and 0.9980 (46.4% of the gap; Ganin et al., 2016, Table 2). The text reports a failure: adapting from MNIST to SVHN did not improve on the non-adapted model, which reached about 0.25 accuracy.

Pixel level: GraspGAN. Bousmalis et al. (2018) adapted domain adaptation, previously evaluated mainly on offline image classification, to learning robotic grasping primarily in simulation. Their abstract reports that synthetic data with domain adaptation reduced the real-world samples needed for a given level of performance by up to 50 times, and good grasping with no real labeled data; only the abstract could be read for this article. James et al. (2019) describe the method, GraspGAN, as combining pixel-level with feature-level adaptation, and note that it needed significant amounts of unlabeled real data collected by pre-existing grasping networks; Tan et al. (2018) describe it as training a generator network that converts simulated images to realistic ones. Pixel-level adaptation is a form of image-to-image translation without paired examples, typically trained with a generative adversarial network objective.

Real-to-sim translation: randomized-to-canonical

James et al. (2019) turned the direction around and removed the need for real data. Their Randomized-to-Canonical Adaptation Networks (RCAN) learn a conditional GAN generator that maps a heavily randomized rendering to the same scene rendered in a fixed, canonical style: uniform colours for the background, tray and each robot link, a fixed light, and the objects’ own textures kept. Training pairs come free from the simulator, which can freeze a scene, render it with a new random appearance (textures drawn from over 5,000 images, random light position, direction and colour), and then render it canonically. The generator also predicts a segmentation mask and a depth image as auxiliary outputs, and its loss combines a pixel equality term, the auxiliary terms and an adversarial term for sharpness.

The grasping policy (QT-Opt, a Q-learning method) is trained in simulation on the generator’s canonical output, concatenated channel-wise with the camera image itself. At test time a real camera image goes through the generator, and if the real world looks to the generator like one more randomization, it comes out canonical. The authors list three advantages over adaptation: no real data to learn the translation, an interpretable intermediate image, and a supervised preprocessing step that spares reinforcement learning from learning directly on randomized inputs, which earlier work found can destabilize algorithms such as DDPG and A3C (QT-Opt itself trained stably under heavy randomization in their experiments). The idea of adapting real images toward simulation rather than the reverse they credit, for a simple driving task, to Zhang et al., who used unlabeled real data. Visual navigation describes a related component, the “Goggles” network of the Gibson environment, which maps real images toward the simulator’s rendering.

Photorealistic and mixed synthetic data

Randomization deliberately gives up realism. The alternative is to make the rendering as close to the target as possible, and the two can be mixed. Tremblay, To et al. (2018) trained their pose estimator DOPE, which predicts the image positions of the eight corners of a known object’s 3D bounding box and recovers the pose with Perspective-n-Point, on about 60,000 randomized images per object together with about 60,000 photorealistic images in which YCB household objects fall under gravity into Unreal Engine scenes (a kitchen, a temple, a forest) and are photographed from random viewpoints. Their stated reason is that the two kinds of data complement one another.

How to Apply It

In practice, transfer is designed around the task’s dominant source of gap. The following order reflects how the sources above proceed, and our reading of where each method’s evidence is strongest:

1. Decide what must transfer. For perception, appearance dominates; for control, dynamics, latency and sensing do. Tan et al. (2018) note that the domain-adaptation methods developed for grasping were hard to apply to locomotion, which needs the gap in dynamics rather than in perception narrowed and cannot easily collect real data.
2. Fix what can be measured. Masses, camera intrinsics, latencies and motor characteristics are cheaper to measure than to randomize away, and Tan et al.’s ablation shows that randomization did not compensate for a simulator that was too wrong.
3. Randomize what cannot. Set ranges that cover the real values with a margin, as Tan et al. did from measurements plus “small safety margins”, and include clutter: distractors were the largest single factor in Tobin et al.’s ablation (below).
4. Shrink what the model sees. Inputs that the simulator renders poorly invite overfitting; Tan et al.’s smaller observation space narrowed the gap.
5. Measure on real data as early as possible. No validation on real data happens during training when no real data is available, as Tobin et al. note; a held-out set of real images or trials is the only check.
6. Add real data when it is affordable, by unlabeled adaptation or by fine-tuning; in the results below, a little real data after simulated pretraining gave large gains.

Example: randomization against adaptation for localization

Embodied AI’s example classifies shapes under fixed and randomized rendering styles. This example asks a different question of the same idea: whether a network can localize an object among distractors after training in simulation, which is what Tobin et al.’s detector did, and how domain randomization compares with feature-level adaptation by DANN, which may see unlabeled target images.

The script renders 32 × 32 grayscale images of a ring, the object to localize, with zero to three square or triangular distractors drawn beneath it, and trains a small convolutional network in PyTorch to regress the ring’s centre. Four training conditions are compared:

• fixed: one clean style, a bright ring and bright distractors on a dark background;
• randomized, no distractors: random textures of Tobin et al.’s three types for background and ring, random illumination gain and gradient, random blur and noise, but no distractors;
• randomized: the same, with distractors, each with its own random texture;
• fixed + DANN: the fixed style with labels, plus unlabeled images of the target style, with a domain classifier behind a gradient reversal layer and Ganin et al.’s schedule for $\lambda$.

The target style stands in for the real world and is never used with labels: a smoothly textured background, a ring and distractors darker than their surroundings, vertical illumination, vignetting, blur and noise. Each model is scored by its mean centre error in pixels on 1,000 test images of its own training style (the fixed style for DANN) and of the target style, and by the share of target images located within 2 pixels. The last line is the error of guessing the mean centre of the target test images for every image, the chance level. Data, initialization and batch order come from the seed; the script loops over seeds 0, 1 and 2 and prints the mean and range. It runs with PyTorch 2.14.1, NumPy 2.5.4 and SciPy 1.18.1 on 4 CPU threads, in about four to six minutes on an Apple M5 laptop.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from scipy.ndimage import gaussian_filter

torch.set_num_threads(4)
N = 32                                   # image size (pixels)
yy, xx = np.mgrid[0:N, 0:N].astype(float)

def ring_mask(cy, cx, r):                # the object to localize: a ring 2 px thick
    d = np.hypot(yy - cy, xx - cx)
    return (d <= r) & (d >= r - 2)

def distractor_mask(rng):                # a filled square or upright triangle
    cy, cx = rng.uniform(5, 27, 2)
    h = rng.uniform(2.5, 4.5)
    if rng.random() < 0.5:
        return (abs(yy - cy) <= h) & (abs(xx - cx) <= h)
    return (yy >= cy - h) & (yy <= cy + h) & (abs(xx - cx) <= (yy - (cy - h)) / 2)

def texture(rng):
    """Tobin et al.'s three random texture types: flat, gradient, checker."""
    a, b = rng.uniform(0, 1, 2)
    kind = rng.integers(3)
    if kind == 0:
        return np.full((N, N), a)
    if kind == 1:
        th = rng.uniform(0, 2 * np.pi)
        t = np.cos(th) * yy + np.sin(th) * xx
        t = (t - t.min()) / (t.max() - t.min())
        return a + (b - a) * t
    k = rng.integers(2, 7)
    return np.where(((yy // k) + (xx // k)) % 2 == 0, a, b)

def render(style, rng):
    cy, cx = rng.uniform(7, 25, 2)
    ring = ring_mask(cy, cx, rng.uniform(4, 6))
    nd = 0 if style == "randomized-nodistract" else rng.integers(0, 4)
    dms = [distractor_mask(rng) for _ in range(nd)]
    if style == "fixed":                 # one clean rendering style
        img = np.full((N, N), 0.2)
        for m in dms:
            img[m] = 0.9
        img[ring] = 0.9
        sigma_n = 0.02
    elif style.startswith("randomized"): # random textures, illumination, blur, noise
        img = texture(rng)
        for m in dms:
            img = np.where(m, texture(rng), img)
        img = np.where(ring, texture(rng), img)
        th = rng.uniform(0, 2 * np.pi)
        grad = (np.cos(th) * (yy - N / 2) + np.sin(th) * (xx - N / 2)) / N
        img = img * rng.uniform(0.5, 1.5) * (1 + rng.uniform(-0.5, 0.5) * grad)
        img = gaussian_filter(img, rng.uniform(0, 1.0))
        sigma_n = rng.uniform(0, 0.08)
    else:                                # "target": a real-like style never rendered in training
        img = 0.6 + 0.6 * gaussian_filter(rng.normal(0, 1, (N, N)), 2.0)
        fine = 0.3 * gaussian_filter(rng.normal(0, 1, (N, N)), 1.0)
        for m in dms:
            img = np.where(m, rng.uniform(0.2, 0.45) + fine, img)
        img = np.where(ring, 0.3 + fine, img)          # the ring is darker than its surroundings
        vign = 1 - 0.3 * ((yy - N / 2) ** 2 + (xx - N / 2) ** 2) / (N / 2) ** 2
        img = img * (0.8 + 0.4 * yy / N) * vign        # uneven illumination and vignetting
        img = gaussian_filter(img, 0.8)                # lens blur
        sigma_n = 0.04
    img = np.clip(img + rng.normal(0, sigma_n, (N, N)), 0, 1)
    return img, np.array([cx, cy])

def dataset(style, n, rng):
    imgs, xy = zip(*(render(style, rng) for _ in range(n)))
    X = torch.tensor(np.array(imgs), dtype=torch.float32).unsqueeze(1)
    Y = torch.tensor((np.array(xy) - N / 2) / (N / 2), dtype=torch.float32)  # centre scaled to [-1, 1]
    return X, Y

class GradReverse(torch.autograd.Function):   # identity forward, gradient times -lambda backward
    @staticmethod
    def forward(ctx, x, lam):
        ctx.lam = lam
        return x.view_as(x)
    @staticmethod
    def backward(ctx, g):
        return -ctx.lam * g, None

class Localizer(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2), nn.Flatten())
        self.head = nn.Sequential(nn.Linear(1024, 128), nn.ReLU(), nn.Linear(128, 2))
        self.domain = nn.Sequential(nn.Linear(1024, 64), nn.ReLU(), nn.Linear(64, 1))
    def forward(self, x):
        return self.head(self.features(x))

def train(X, Y, seed, Xu=None, epochs=10, bs=64):
    """Supervised training on (X, Y); with unlabeled Xu, adds DANN's adversarial domain loss."""
    torch.manual_seed(seed)
    net = Localizer()
    opt = torch.optim.Adam(net.parameters(), lr=1e-3)
    steps, step = epochs * (len(X) // bs), 0
    for _ in range(epochs):
        perm = torch.randperm(len(X))
        for i in range(0, len(X) - bs + 1, bs):
            idx = perm[i:i + bs]
            f = net.features(X[idx])
            loss = F.mse_loss(net.head(f), Y[idx])
            if Xu is not None:           # Ganin et al.'s schedule: lambda rises from 0 to 1
                lam = 2 / (1 + np.exp(-10 * step / steps)) - 1
                fu = net.features(Xu[torch.randint(len(Xu), (bs,))])
                d = net.domain(GradReverse.apply(torch.cat([f, fu]), lam)).squeeze(1)
                dom = torch.cat([torch.zeros(bs), torch.ones(bs)])
                loss = loss + F.binary_cross_entropy_with_logits(d, dom)
            opt.zero_grad()
            loss.backward()
            opt.step()
            step += 1
    return net

def pixel_error(net, X, Y):
    with torch.no_grad():
        return ((net(X) - Y) * N / 2).norm(dim=1).numpy()

conditions = [("fixed", "fixed", False), ("randomized, no distractors", "randomized-nodistract", False),
              ("randomized", "randomized", False), ("fixed + DANN", "fixed", True)]
res = {name: [] for name, _, _ in conditions}
base = []
for seed in range(3):
    rng = np.random.default_rng(seed)
    Xt, Yt = dataset("target", 1000, rng)            # held-out "real-like" test images
    Xu, _ = dataset("target", 5000, rng)             # unlabeled target images, used only by DANN
    base.append(((Yt - Yt.mean(0)) * N / 2).norm(dim=1).mean().item())
    for name, style, dann in conditions:
        X, Y = dataset(style, 5000, rng)
        Xs, Ys = dataset(style, 1000, rng)           # test images in the training style
        net = train(X, Y, seed, Xu if dann else None)
        es, et = pixel_error(net, Xs, Ys), pixel_error(net, Xt, Yt)
        res[name].append((es.mean(), et.mean(), 100 * (et <= 2).mean()))

print("centre error in pixels, mean over seeds 0-2 (min-max)")
print(f"{'trained on':>27} {'own style':>17} {'target style':>17} {'target <= 2 px (%)':>20}")
for name, r in res.items():
    a = np.array(r)
    cells = [f"{m:5.2f} ({lo:4.2f}-{hi:5.2f})" for m, lo, hi in zip(a.mean(0), a.min(0), a.max(0))]
    cells[2] = f"{a[:, 2].mean():5.1f} ({a[:, 2].min():4.1f}-{a[:, 2].max():5.1f})"
    print(f"{name:>27} {cells[0]:>17} {cells[1]:>17} {cells[2]:>20}")
print(f"{'guess the mean centre':>27} {'':>17} {np.mean(base):5.2f}")

Output (two runs gave identical output):

centre error in pixels, mean over seeds 0-2 (min-max)
                 trained on         own style      target style   target <= 2 px (%)
                      fixed  0.56 (0.53- 0.59)  6.97 (6.88- 7.07)     4.1 ( 3.8-  4.5)
 randomized, no distractors  2.22 (2.20- 2.23)  4.46 (4.26- 4.72)    19.3 (17.3- 20.9)
                 randomized  3.33 (3.27- 3.39)  2.93 (2.77- 3.00)    41.1 (40.1- 43.1)
               fixed + DANN  5.09 (4.21- 5.72)  9.67 (7.89-12.69)     2.9 ( 2.5-  3.3)
      guess the mean centre                    6.90

The fixed-style model is nearly exact in its own style and at chance in the target style: its error is about that of guessing the mean centre. Reading the construction rather than the output, nothing in the fixed style tells the network to look for a ring rather than for the bright region shaped like one, and in the target style the ring is the dark region. Both randomized models move well away from chance, and the one trained with distractors is clearly best on the target, with about two in five target images located within 2 pixels; every seed of it beats every seed of the model trained without distractors. That ordering agrees in direction with Tobin et al.’s ablation, where removing distractors from training raised real-world error from 1.8 to 7.2 cm on images with distractors, although a toy cannot show the size of a real effect.

The randomized models’ errors on their own styles are several times the fixed model’s on its style, but these are different test sets, and the script does not test the randomized models on the fixed style, so it does not measure the trade-off between robustness and peak performance that Tan et al. measured for dynamics. The randomized model even does slightly worse on its own style than on the target. By construction, its ring and background textures are drawn independently, so some training-style images have a ring of nearly the same brightness as its background, while the target’s ring is built darker than its surroundings; the randomized style is the harder test.

DANN failed in this set-up, against our expectation recorded before the run. It is worse than the fixed model on the target, with the widest spread over seeds, and adding the adversarial term raised the error on the labeled fixed style by almost a factor of ten. The printed numbers do not say why; in our reading, aligning the overall distributions of the two domains’ features does not require keeping the features that encode the ring’s position, and Ganin et al. designed and evaluated DANN for classification, not coordinate regression. We used their published schedule for $\lambda$ unchanged rather than tuning it, though with Adam in place of their annealed stochastic gradient descent with momentum, so the example shows that adaptation is not a drop-in addition, not that DANN cannot work for localization. It also shows a practical asymmetry: DANN needed 5,000 images from the target domain, which randomization did not.

What the example cannot show: real images differ from any simulator in ways a hand-written “target style” does not, and here the randomization ranges were chosen by us with the target’s polarity flip inside them. The result depends on that choice in the same way that real transfer depends on whether real conditions fall inside the randomized range.

Variants

Uniform randomization with non-realistic textures. Tobin et al. (2017) and CAD2RL (2017), described above. Tobin et al. contrast their textures, created by a simple random process and allowing hundreds of thousands of unique texturings, with CAD2RL’s selection from about 200 pre-generated materials, most of them realistic.

Randomization for detection on real data. Tremblay, Prakash et al. (2018) with flying distractors and photographic backgrounds; DOPE (Tremblay, To et al., 2018) for 6-DoF pose, mixing randomized and photorealistic images.

Dynamics randomization with memory. Peng et al. (2018), with a recurrent policy as implicit system identification.

System identification plus randomization. Tan et al. (2018): measured parameters, new actuator and latency models, then randomization and perturbation forces on top.

Learning the randomization distribution. James et al. (2019) point to work by Chebotar et al. that learns, from a few real trajectories, the best distribution of simulation parameters, in place of setting the ranges by hand.

Dexterous manipulation. OpenAI et al.’s Learning Dexterous In-Hand Manipulation (2020) could not be read: the journal’s site is not used for this site’s research and OpenAI’s own copy refused automated access, so nothing is reported from it here.

Feature-level adaptation. DANN (Ganin et al., 2016) and related methods that align feature distributions, using unlabeled real images.

Pixel-level adaptation. GraspGAN (Bousmalis et al., 2018) and other sim-to-real image translators, also using unlabeled real images.

Real-to-sim translation. RCAN (James et al., 2019), with no real data; Gibson’s Goggles, described in Visual Navigation.

Simulated evaluation of real policies. The reverse use of simulation, building simulated copies of real set-ups to evaluate policies trained on real data, is described in Embodied AI through the SIMPLER study.

Best Practices

• Randomize the nuisance, not the task. In every source above, what is randomized is what should not matter (textures, lights, camera pose, masses, latency) while the quantity being predicted stays tied to the scene’s geometry. Tobin et al.’s detector never saw the target object’s colour, only its size and shape, and could therefore pick it out next to objects of the same colour.
• Include clutter and occlusion in simulation. Distractors in Tobin et al.’s and Tremblay, Prakash et al.’s ablations, and in the example above.
• Model the delays and the actuators. Peng et al.’s two largest ablation drops came from removing action-timestep randomization and observation noise; Tan et al.’s controllers failed without latency or actuator models.
• Keep the observation compact where the simulator renders some inputs poorly (Tan et al., 2018).
• Compare against a fixed-style baseline and against real-only training, so that the gain from simulation and its size are visible; RCAN’s table includes both, and Tremblay, Prakash et al. compare with photorealistic synthetic data and with real-only training.
• Plan for fine-tuning. Every source that tried it found that a small amount of real data after simulated pretraining improved results considerably (Evidence below).

Pitfalls

• Choosing the method on the test set. Without real data in training, there is a temptation to tune the simulator against the real test images until transfer appears. Tremblay, Prakash et al. (2018) state that they stopped training when performance on the test set saturated and reported only the best results, which makes their numbers optimistic estimates.
• Ranges that miss the real value. Randomization helps only if reality lies within or near the randomized range; a range that is too narrow transfers nothing, and one that is too wide costs performance, as Tan et al.’s trade-off shows.
• Treating randomization as a substitute for a correct simulator. In Tan et al.’s experiments, controllers trained with random perturbations in the baseline simulator still performed poorly on the robot.
• Unmodelled materials. DOPE missed many instances of a potted meat can in heavily occluded frames, where only its top showed, because the synthetic data did not model the highly reflective metal of that surface (Tremblay, To et al., 2018).
• Expecting adaptation or randomization to work unchanged. DANN failed on MNIST to SVHN in its own paper and in the regression example above. Randomization is not a drop-in addition either: RCAN’s authors cite work in which heavy randomization made DDPG and A3C training fail, although QT-Opt trained stably.
• Reading success in simulation as success. RCAN’s canonical-simulation policy succeeded in 99% of simulated grasps and 21% of real ones.
• Generalizing from one setting. Most results here come from one robot, one set of objects and one room; they show that transfer was achieved, not how reliable it is elsewhere.

Evidence and Impact

Results in this field come from different robots, tasks and metrics, so they can be compared only within each study. The figures below are the authors’ reports, each with its set-up.

Localization from randomized renders (Tobin et al., 2017). On 480 webcam images of eight geometric objects on a table, 70 to 105 cm from the camera, 60 per object (20 alone, 20 with distractors, 20 partly occluded), detectors trained only on randomized MuJoCo renders located objects to about 1.5 cm on average, against 0.3 to 0.5 cm on simulated data, which the authors read as overfitting to simulation. In the ablation of their Table II (models trained on 20,000 examples), the full method had errors of 1.3, 1.8 and 2.4 cm on object-only, distractor and occlusion images; without camera randomization 2.0, 2.4 and 2.9 cm; without distractors in training 1.5, 7.2 and 7.4 cm; without noise, almost no change. A Fetch robot picked the detected object in 38 of 40 trials in increasingly cluttered scenes, and a detector trained for a can of Spam picked it among unseen food items in 9 of 10.

Randomized against realistic rendering for flight (Sadeghi & Levine, 2017). On a proxy task, free-space prediction on 620 real RGB-D images from five hallways in two campus buildings, with labels from the depth channel, a network trained on CAD2RL’s randomized hallways had higher precision–recall than networks trained on fixed textures and lighting (three or nine hallways) and than one trained on a realistically textured reconstruction of one of the very buildings the test images came from. The policy also flew real Parrot Bebop quadrotors in indoor scenarios with fewer collisions than the strongest baseline in their hallway-following tests.

Dynamics randomization for pushing (Peng et al., 2018). The headline figures and two ablations are in Embodied AI. Their Table II also compares architectures on the real robot: a feedforward policy trained with randomization had a success rate of 0.67 over 12 trials, one given a history of eight past states and actions 0.70 over 20, the recurrent policy 0.89 over 28. Holding the link masses or the puck’s friction fixed in training gave 0.64 and 0.48 (Table III). With a bag of chips taped under the puck to change its friction, the recurrent policy succeeded in 0.91 of trials.

Locomotion (Tan et al., 2018). Measuring the gap as the difference in expected return between simulation and robot, with 100 controllers trained per condition and the top three run three times each on the robot, controllers trained in the baseline simulator, with or without random perturbations, performed well in simulation and poorly on the robot, while those trained in the improved simulator performed comparably in both. Randomization narrowed the gap for both observation spaces, and with the four-dimensional observation (base roll, pitch and their rates) all nine runs trotted more than three metres and balanced for the whole episode. A twelve-dimensional observation that added the motor angles scored higher in simulation and lower on the robot. The learned galloping gait reached about 1.34 m/s in simulation and 1.18 m/s on the robot.

Synthetic-to-real adaptation of digits and signs (Ganin et al., 2016). Given above under Domain adaptation.

Grasping with randomized-to-canonical translation (James et al., 2019). Each robot attempted 102 grasps of 5 to 6 objects not seen in training, with at most 20 steps per attempt (Table 1):

QT-Opt trained on	Real grasps in training	Success in simulation	Success on robot, no real fine-tuning	After 5,000 real on-policy grasps
Real data	580,000	–	87%	85%
Canonical simulation	0	99%	21%	30%
Mild randomization	0	98%	37%	85%
Medium randomization	0	98%	35%	77%
Heavy randomization	0	98%	33%	85%
RCAN	0	99%	70%	91%

With 28,000 real on-policy grasps, the figures were 96% for the real-data system, 92% with heavy randomization and 94% for RCAN. The abstract says RCAN’s zero-shot result “almost doubles” learning on randomization alone, while the results text says “over double”; the table’s 70% against 33–37% is between 1.9 and 2.1 times. The abstract also calls RCAN’s 91% comparable to the 580,000-grasp system, which the text describes as outperforming it (87% in the table).

Detection of cars on KITTI (Tremblay, Prakash et al., 2018). On 500 images taken at random from KITTI, scored by average precision at an IoU of 0.5 on cars more than 40 pixels high and less than 0.15 truncated (KITTI’s “easy” criteria), detectors trained on 100,000 randomized images scored, against the same architectures trained on the 2,500 photorealistic images of Virtual KITTI, which was rendered to recreate KITTI: 78.1 against 79.7 for Faster R-CNN, 71.5 against 64.6 for R-FCN (both with an ImageNet-pretrained Inception-ResNet V2) and 46.3 against 36.1 for SSD (with ResNet-101) (Table 1; the legend of Figure 5 gives the randomized Faster R-CNN 76.8). Fine-tuned on all 6,000 real training images, the randomized Faster R-CNN reached 98.5, against 98.0 from Virtual KITTI and 96.5 from real images only (Figure 6); the text gives the margins as 1.6% and 2.1%, which do not match the figure’s labelled values. In the ablation (Faster R-CNN with ResNet, 50,000 images, full method 73.7), fixed lighting gave 67.6, no texture on the cars 69.0, half the textures 71.5, no ordinary augmentation 72.0, and no flying distractors a decrease of 1.1%. Starting from COCO weights, the detector scored 56.1 with no further training, 79.7 after training on Virtual KITTI and 83.7 after training on randomized data (Table 2).

Pose estimation (Tremblay, To et al., 2018). On the 2,949 test frames of YCB-Video, scored by accuracy–threshold curves of the ADD metric (average 3D distance of the model’s points between estimated and true pose), DOPE trained only on synthetic data had a higher area under the curve than PoseCNN, trained on synthetic and real images from other videos of the same dataset, for four of five objects, and better accuracy below 2 cm, about the grasping tolerance of their robot’s gripper. On one object, trained on randomized data alone, the area under the curve peaked at 66.64 with 300,000 images and on photorealistic data alone at 62.94 with 600,000, against 77.00 for the 120,000-image mixture. Performance was comparable for any mix that included at least 40% of either dataset.

Across these studies the pattern is consistent within each: training in a single, fixed simulator transferred poorly; randomization, simulator fidelity or adaptation recovered much of the gap; and a small amount of real data after simulated pretraining recovered more. How much each method contributes is specific to the task and set-up. James et al. (2019) describe domain randomization as particularly popular in robotics, and the studies above apply it to grasping, manipulation, flight, locomotion, detection and pose estimation.