Vision Systems & Engineering

Synthetic Data for Computer Vision

Training and testing vision models on images produced by a renderer, a compositor or a generative model, with labels that come from the generator itself, and the conventions, gaps and evidence that decide whether such data helps.

intermediate

Synthetic data for computer vision means images that a program makes rather than a camera records, used to train or test vision models. The program can be a game engine that renders a 3D world, a procedural generator that assembles a scene from rules, a compositor that pastes real object cut-outs onto real backgrounds, or a generative model that produces images from text. What the different kinds share is that the program knows what it made. A renderer that placed a car knows the car’s class, its 3D box, which pixels it covers, how far away each pixel is and how it moves between frames. Labels that a person would draw slowly, inconsistently or not at all come out of the generator at no extra cost. The price is that the images are not the ones the model will meet.

Image annotation describes the human route to labels and notes that some answers can be measured or synthesized instead of judged. Sim-to-real transfer covers the reality gap, domain randomization and adaptation for robots. This article covers the data itself: the families of generators, how a generator turns its scene into labels and which conventions remain open, how synthetic data is mixed with real data, and what the published comparisons show, each within its own set-up.

Purpose

Real labelled data is limited in three ways that synthetic data addresses.

  • Cost and scale of dense labels. Ros et al. (2016) point out that semantic segmentation needs a silhouette drawn around every object. Among the urban datasets they list, CamVid has 701 labelled images and KITTI 430 labelled for segmentation, while Cityscapes has 5,000 finely annotated images.
  • Coverage of conditions. Gaidon et al. (2016) note that the KITTI tracking benchmark has only 29 test sequences, recorded in similar good conditions from a single source. A renderer can change the weather, the time of day or the camera angle and keep everything else fixed. They call this “ceteris paribus” analysis, and they also generate rare or hard-to-observe conditions on demand (“what-if” analysis).
  • Consistency of labels. Gaidon et al. describe KITTI’s 2D boxes as drawn by annotators, usually slightly larger than the cars, with margins that vary from car to car and from annotator to annotator. KITTI’s occlusion states (“fully visible”, “partly occluded”, “largely occluded”) are subjective, and many edge cases carry inconsistent labels. A generator applies one rule to every object.

The last point needs care. A generator’s labels are consistent, but they are exact only once a pixel convention and a definition are fixed: which box, which pixels, what counts as occluded. The example below measures how much these choices change the labels.

When to Use It

Synthetic data is worth considering when:

  • The label cannot be drawn. Optical flow, scene flow and dense depth are examples. FlyingChairs and MPI Sintel render flow that is known exactly; the FlyingChairs article explains why real correspondences are hard to measure and describes Mayer et al.’s FlyingThings3D.
  • The label is expensive per image and the domain is narrow. Instance detection of known products in a kitchen or warehouse is an example: Dwibedi et al. (2017) argue that curating and annotating real images of particular instances does not scale as new instances are added.
  • Conditions must be controlled. Gaidon et al.’s weather and camera variations, described under Evidence and Impact, change one factor at a time on the same scene. Test sets built this way measure robustness, not only training value.
  • Real data is unavailable or restricted. Image generation discusses generated images as a substitute for data that cannot be shared.

It helps less when real labelled data is plentiful and cheap for the task, and when what decides success is something the generator does not model. Training on synthetic data alone is usually not enough in the studies below. The size of the gap depends on the generator, the task and the target data, so it is measured, not assumed.

How It Works

Families of generators

Rendered 3D worlds. A game engine or renderer draws a modelled world through a virtual camera. Gaidon et al. (2016) built Virtual KITTI in the Unity engine by “cloning” five real KITTI tracking sequences. Cars were placed from KITTI’s annotated positions and orientations relative to the recorded camera path, roads were placed along the camera’s position, and secondary roads, trees and buildings were placed by hand. A directional light and an ambient light simulate the sun, set by comparing brightness and shadows with the real frames. The authors report about one person-day per world. At publication the dataset had 35 videos (5 clones, each with 7 variations) and about 17,000 frames of around 1242 × 375 pixels. The variations were the camera turned 15° right or left, morning and sunset light, overcast, fog and rain.

Ros et al. (2016) built SYNTHIA, a virtual city also made in Unity, with pixel-level annotations for 13 classes and a depth map for every frame. They describe it as more than 213,400 synthetic images. SYNTHIA-Rand has 13,400 frames from cameras moved randomly through the city at heights of 1.5 to 2 m, with dynamic objects, illumination and road textures changed at each pose. SYNTHIA-Seqs has four sequences of about 50,000 frames each, one per season, from a virtual car carrying two four-camera rigs. As the Cityscapes article reports, Richter et al. (2016, not read for this article) instead used 24,966 labelled frames from the game GTA V. Driving simulators such as CARLA and general tools such as Blender and Isaac Sim are further sources of rendered worlds (named here only; their documentation was not read for this article), and embodied AI describes the interactive simulators.

Procedural generation and scene grammars. Instead of modelling one world, a program samples many. Kar et al. (2019) describe the typical approach as a probabilistic scene grammar. For a traffic scene the grammar might lay out a road’s centre line, add lanes and place cars on each lane, with each element’s attributes (position, pose, colour, asset) drawn from hand-set distributions that, they note, need careful tuning. Their Meta-Sim keeps the grammar’s structure but learns to change the attributes. A graph neural network (the “distribution transformer”) rewrites each node’s mutable attributes so that the rendered images match a target set of real images under maximum mean discrepancy, computed on Inception features. Optionally it also maximizes a task network’s score on a small labelled real validation set. Gradients pass through the non-differentiable renderer by finite differences. Kar et al. separate the appearance gap, which image translation addresses, from a content gap: the layout and kinds of objects that a hand-built world reproduces. Virtual KITTI, for example, matches Karlsruhe and not, they note, a city with heavier traffic.

Compositing real cut-outs. Dwibedi et al. (2017) do not render a world. They take real images of each object instance (BigBIRD, 600 images per object from five cameras), predict a foreground mask with a fully convolutional network trained on depth-sensor masks, and paste the cut-outs onto real indoor backgrounds (UW Scenes, 1,548 images). They argue that local, patch-level realism matters more for detection than global scene consistency. A detector, they say, will key on the subtle boundary artefacts of naive pasting, so they render the same scene several times with different blending (none, Gaussian blur, Poisson) to make the detector ignore them. They also add 2D and 3D rotations, truncation (at least 25% of the object box inside the image), occlusion (pasted objects may overlap up to an IoU of 0.75) and distractor objects. Image blending covers the blending methods themselves, and mixup and CutMix covers pasting between training images with mixed labels. Scene text recognition describes MJSynth and SynthText (Gupta, Vedaldi & Zisserman, 2016, not read for this article), synthetic word and scene images on which, as it reports, almost all recognizers have been trained; scene text detection covers SynthText’s use for detectors.

Refining simulated images. Shrivastava et al. (2017) proposed Simulated+Unsupervised (S+U) learning: improve the realism of a simulator’s output with unlabelled real images while keeping the simulator’s labels. Their SimGAN trains a refiner network RθR_\theta against a discriminator DϕD_\phi that outputs, for each local patch, the probability that the patch comes from a refined image. With the identity as the feature map ψ\psi in their experiments, the refiner minimizes their Eq. (4):

LR(θ)=−∑ilog⁡(1−Dϕ(Rθ(xi)))+λ∥ψ(Rθ(xi))−ψ(xi)∥1,\mathcal{L}_R(\theta) = -\sum_i \log\bigl(1 - D_\phi(R_\theta(\mathbf{x}_i))\bigr) + \lambda \bigl\lVert \psi(R_\theta(\mathbf{x}_i)) - \psi(\mathbf{x}_i) \bigr\rVert_1 ,

where xi\mathbf{x}_i are synthetic images. The first term pushes refined images toward realism. The second, which they call self-regularization, penalizes changes that could move the label, for example a change of gaze direction. Two further changes stabilize training: the adversarial loss is computed over local patches rather than the whole image, and the discriminator is updated with a buffer of earlier refined images as well as current ones. The refiner is fully convolutional without striding or pooling, so it changes pixels locally. Their data are synthetic and real eye and hand images (gaze estimation on MPIIGaze, hand pose on NYU depth images). Pixel-level adaptation of this kind is a form of image-to-image translation trained with a GAN objective; domain adaptation and Sim-to-Real Transfer cover the feature-level alternatives.

Generative models. Sariyildiz et al. (2023) asked how far a text-to-image model could replace ImageNet’s real training images. Using only the class names, they generated “ImageNet clones” with Stable Diffusion v1.4, with the same number of images per class as the real set, and trained a ResNet-50 from scratch. Prompts made only of the class name produced semantic errors (polysemous class names drawn in the wrong sense), images that looked alike, and shifts toward other visual domains such as cartoons. Adding the WordNet hypernym or definition, placing the object “inside” a random scene class from Places365, reduced the semantic and domain problems, and lowering the guidance scale from 7.5 to 2 gave more diverse image sets and the best results on ImageNet-100. Here the labels are the prompts, so a label is only as correct as the generator’s rendering of the prompt. There is no geometry from which boxes or masks could be derived.

Domain randomization deliberately gives up realism and samples textures, lights, cameras and distractors widely. Embodied AI describes Tobin et al.‘s (2017) proposal; Sim-to-Real Transfer compares randomized with photorealistic and mixed data, including Tremblay and colleagues’ two 2018 studies of detection and pose.

Labels from the generator

Gaidon et al. (2016) describe how Virtual KITTI’s ground truth is produced, and the description serves as a template for rendered data in general. Each moment of the scene is rendered four times:

  1. the photorealistic image;
  2. a depth map, read from the renderer’s depth buffer;
  3. category and instance segmentation, rendered with unlit shaders, so that each object of interest has a unique colour unaffected by lighting and shading;
  4. dense optical flow, computed in a vertex shader from each object’s model, view and projection matrices for the previous and current frames.

They report that the whole pipeline runs at 5 to 8 frames per second on one desktop. Boxes and metadata come from geometry:

  • 2D box: the perspective projection of the object’s 3D bounding box onto the image plane, clipped to the image boundaries for truncated objects, and associated with the object’s identity for tracking. This is an amodal box: it includes parts hidden by other objects. Bounding box contrasts PASCAL VOC’s visible-extent boxes with MOTChallenge’s amodal ones.
  • Truncation rate: in the paper’s words, approximated “by dividing the volume of an object’s 3D bounding box by the volume of the 3D bounding box of the visible part”, the visible part being the box intersected with the camera frustum. As printed, this is the full volume over the visible volume, a ratio of at least 1, not a share. The thresholds the authors apply to it are not given.
  • 2D occupancy rate: the number of pixels in the object’s segmentation mask divided by the area of the projected 2D box, which, the authors stress, includes the occluder. The paper does not say whether the box is clipped before its area is taken.

To keep evaluation comparable with real KITTI, Gaidon et al. ignored cars smaller than 25 pixels and heavily truncated or occluded cars, and set per-sequence thresholds on occupancy and truncation chosen to be as close as possible to KITTI’s annotations. A generator thus replaces subjective judgements (“partly occluded”) with thresholds on computed rates. The thresholds are still choices, and so are the definitions of the rates.

How to Apply It

The sources use synthetic data in four ways, which can be combined:

  1. Train on synthetic data only, then test on real data. In the studies below this was the weakest configuration, except in some of SYNTHIA’s comparisons.
  2. Mix synthetic and real data in each batch. Ros et al. (2016) used Balanced Gradient Contribution: every batch had 6 real and 4 synthetic images, with real images dominating and the synthetic ones acting, in their words, as a sophisticated regularization term.
  3. Pretrain on synthetic data, then fine-tune on real. Gaidon et al. (2016) call this virtual pre-training. They found it critical to validate an early-stopping point for the fine-tuning stage, to avoid overfitting the small real set.
  4. Add synthetic data to a small real set. Dwibedi et al. (2017) measured this with 10%, 40% and 70% of their real training images.

A workable order, in our reading of these sources:

  • Decide which labels the task needs and whether the generator can produce them exactly. Geometry yields boxes, masks, depth and flow; a text prompt yields only a class.
  • Fix the label conventions before generating: amodal or visible boxes, how truncation and occlusion are computed, the minimum object size, and how pixels on boundaries are assigned.
  • Generate with variation along the factors the real data varies in. Hold some real labelled data back for validation, because only real data shows whether training transfers.
  • Mix or fine-tune with whatever real labels exist, and measure each configuration on the same real test split.

Example: labels from a z-buffer, and the conventions that remain

Rendering images with known ground truth is routine in this site’s examples. This example looks instead at how the labels are derived and what the derivation’s open choices do to a labelled set. Instance segmentation’s example derives the visible mask of one occluded disk, and label assignment’s takes boxes from masks; here a generator labels thousands of objects, and the measures are the label definitions and filters.

The scene. A 160 × 120 pinhole camera with focal length 100 pixels and principal point (80, 60) looks along +z+z, with xx to the right and yy down, so u=100X/Z+80u = 100X/Z + 80 and v=100Y/Z+60v = 100Y/Z + 60. Pixel (r,c)(r, c) covers [c,c+1)×[r,r+1)[c, c+1) \times [r, r+1), its centre is (c+0.5,r+0.5)(c + 0.5, r + 0.5), and pixel boxes are written [cmin⁡,rmin⁡,cmax⁡+1,rmax⁡+1][c_{\min}, r_{\min}, c_{\max} + 1, r_{\max} + 1]. Each of 500 scenes holds 3 to 8 axis-aligned cuboids or ellipsoids, each with probability 0.5 and half-extents of 0.4 to 1.5 m. The objects are separated in depth, nearest from 3 to 6 m and each next one 0.2 to 3 m behind the previous one, and their projected centres fall up to 10% beyond the image edges, so many cross the border. Rays through the pixel centres give a z-buffer: instance-ID, depth and class buffers and a shaded image. Rays through a 4 × 4 grid of sub-samples in each pixel give coverage.

The labels. Three boxes are computed for each object:

  • A1, Gaidon et al.’s amodal box: the projection of the object’s 3D bounding box, clipped to the image. For an ellipsoid this is the box enclosing the ellipsoid.
  • A2, a second reading of “projection of the whole object”: the tight pixel box of the object’s full silhouette, rendered as if nothing were in front of it.
  • V, the visible box: the tight pixel box of the pixels the object owns in the instance buffer.

Truncation TT is our 2D analogue of the paper’s rate: the share of A1’s unclipped area that falls outside the image. Occupancy is visible pixels over a box area, computed three ways because the paper leaves clipping open: O1 over clipped A1 (our reading of Gaidon et al.), O1u over unclipped A1, and O2 over A2. An object is occluded if any pixel of its silhouette inside the image is not visible, and labelled if it has at least one visible pixel.

The measures.

  • C1, a check. The z-buffer’s silhouettes and visible masks must equal regions computed in closed form: the convex hull of the eight projected corners for a cuboid, the conic C=(PQ∗P⊤)−1C = (P Q^* P^\top)^{-1} for an ellipsoid (the projection of its dual quadric Q∗Q^* by the camera matrix PP), and visibility from depth order.
  • D1–D4, data checks. Each scene has 3 to 8 objects; objects are separated in depth with the nearest face at least 3 m away; the data include both truncated and occluded objects; and at least 100 objects qualify for H1.
  • H1. For untruncated, unoccluded objects, A2 and V are the same box (IoU exactly 1).
  • H1b. For those objects A1 and V never coincide, and they agree less for ellipsoids than for cuboids.
  • H2. Mean IoU between amodal and visible boxes rises with occupancy.
  • H3. For each occupancy definition, a threshold of 0.25 keeps more objects than 0.5, and at 0.5 the three definitions keep three different counts.
  • H4. Supersampled masks (an object owns a pixel if it covers at least 8 of the 16 sub-samples) differ from centre-sampled masks only on boundary pixels, defined as pixels whose 3 × 3 neighbourhood in the centre-sampled instance buffer holds more than one label.
  • H5. Filtering at O1 ≥ 0.5 lowers the share of ellipsoids among kept objects, and filtering at O2 ≥ 0.5 lowers it less.

The script also counts objects whose visible box, or whose O1 ≥ 0.5 decision, changes under supersampling. IoU is the intersection over union of two boxes.

How the example was run. The script takes the seed and the number of scenes as arguments; the output below is python synth_labels.py 0 500. It uses NumPy 2.3.5 and SciPy 1.16.3 on one thread and runs in about 15 to 20 seconds on a shared machine. Two runs printed identical output.

The order of work matters for reading the results:

  • We wrote the hypotheses and their bars before any code ran.
  • A planning script, a copy of this one with a timing line, then ran on the first 50 scenes of the pilot’s seed. It printed a value for every check before the plan was frozen, so every outcome was partly seen in advance. The example is therefore an illustration of the derivations above rather than a blind test.
  • One bar was set after that script: the data check that at least 100 untruncated, unoccluded objects exist for H1.
  • After that script, the only change to the code was a print format: the H2 means went from four to six decimals.
  • A pilot then ran on seed 1 and the run shown on seed 0, with the code unchanged between them.

Against the batch plan’s outline for this example, we chose to print both amodal boxes and all three occupancy denominators because the paper leaves those choices open. We also fixed the minimum height at 25 pixels, measured the effect of filters as the kept share and the share of ellipsoids, and added H1b, H5 and the two supersampling counts. Further choices were the depth-separated objects, the closed forms used for C1, and the sizes, seeds and boundary definition above.

import os
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["OMP_NUM_THREADS"] = "1"
import sys
import numpy as np
from scipy.spatial import ConvexHull
from scipy.ndimage import maximum_filter, minimum_filter

SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
N_SCENES = int(sys.argv[2]) if len(sys.argv) > 2 else 500
W, H, F, CX, CY = 160, 120, 100.0, 80.0, 60.0   # image size, focal length (px), principal point
SS = 4                                            # 4 x 4 supersampling for the coverage labels
SIGNS = np.array([[i, j, k] for i in (-1, 1) for j in (-1, 1) for k in (-1, 1)], float)

def ray_grid(offsets):
    """Ray directions (x/z, y/z) through sample points; pixel (r, c) covers [c, c+1) x [r, r+1)."""
    r, c = np.mgrid[0:H, 0:W].astype(float)
    return [((c + ox - CX) / F, (r + oy - CY) / F) for oy in offsets for ox in offsets]

CENTRE = ray_grid([0.5])[0]
SUB = ray_grid((np.arange(SS) + 0.5) / SS)

def make_scene(rng):
    """3-8 axis-aligned boxes (class 1) or ellipsoids (class 2), separated in depth, nearest first."""
    objs, z = [], rng.uniform(3.0, 6.0)               # z: depth of the next object's near face
    for _ in range(rng.integers(3, 9)):
        s = rng.uniform(0.4, 1.5, 3)                  # half-extents / semi-axes (m)
        Z = z + s[2]
        u0, v0 = rng.uniform(-16, 176), rng.uniform(-12, 132)   # projected centre, may be off-image
        objs.append({"cls": 1 + int(rng.random() < 0.5), "s": s,
                     "C": np.array([(u0 - CX) * Z / F, (v0 - CY) * Z / F, Z])})
        z = Z + s[2] + rng.uniform(0.2, 3.0)
    return objs

def hit_depth(o, dx, dy):
    """Depth Z of the first hit of the ray (dx, dy, 1) with the object, inf on a miss."""
    C, s = o["C"], o["s"]
    if o["cls"] == 1:                                 # slab test for the cuboid
        tx = np.sort([(C[0] - s[0]) / dx, (C[0] + s[0]) / dx], axis=0)
        ty = np.sort([(C[1] - s[1]) / dy, (C[1] + s[1]) / dy], axis=0)
        tn = np.maximum(np.maximum(tx[0], ty[0]), C[2] - s[2])
        tf = np.minimum(np.minimum(tx[1], ty[1]), C[2] + s[2])
        return np.where(tn <= tf, tn, np.inf)
    a, b, c = s                                       # ellipsoid: quadratic in t (= Z)
    A = (dx / a) ** 2 + (dy / b) ** 2 + 1 / c ** 2
    B = -2 * (dx * C[0] / a ** 2 + dy * C[1] / b ** 2 + C[2] / c ** 2)
    D = B * B - 4 * A * ((C[0] / a) ** 2 + (C[1] / b) ** 2 + (C[2] / c) ** 2 - 1)
    return np.where(D >= 0, (-B - np.sqrt(np.maximum(D, 0))) / (2 * A), np.inf)

def render(objs, grid):
    """z-buffer: instance id (0 = background), depth and class buffers, plus each object's own hit depth."""
    d = np.array([hit_depth(o, *grid) for o in objs])
    inst = np.where(np.isfinite(d.min(0)), d.argmin(0) + 1, 0)
    cls = np.array([0] + [o["cls"] for o in objs])[inst]
    return inst, d.min(0), cls, d

def project(P):
    return np.stack([F * P[:, 0] / P[:, 2] + CX, F * P[:, 1] / P[:, 2] + CY], 1)

def analytic_silhouette(o):
    """Pixel centres inside the projected outline, computed in closed form (not by ray casting)."""
    r, c = np.mgrid[0:H, 0:W]
    p = np.stack([c.ravel() + 0.5, r.ravel() + 0.5, np.ones(H * W)], 1)
    if o["cls"] == 1:                                 # convex hull of the 8 projected corners
        eq = ConvexHull(project(o["C"] + o["s"] * SIGNS)).equations
        return (p[:, :2] @ eq[:, :2].T + eq[:, 2] <= 0).all(1).reshape(H, W)
    T = np.eye(4); T[:3, 3] = o["C"]                  # dual quadric projects to the dual conic
    Qd = T @ np.diag(np.r_[o["s"] ** 2, -1.0]) @ T.T
    P = np.array([[F, 0, CX, 0], [0, F, CY, 0], [0, 0, 1, 0]])
    Cn = np.linalg.inv(P @ Qd @ P.T)
    q = np.einsum("ni,ij,nj->n", p, Cn, p)
    pc = np.r_[project(o["C"][None])[0], 1.0]
    return (np.sign(q) == np.sign(pc @ Cn @ pc)).reshape(H, W)

def pixbox(m):
    r, c = np.nonzero(m)
    return np.array([c.min(), r.min(), c.max() + 1, r.max() + 1], float)

def area(b):
    return max(b[2] - b[0], 0) * max(b[3] - b[1], 0)

def iou(a, b):
    inter = area([max(a[0], b[0]), max(a[1], b[1]), min(a[2], b[2]), min(a[3], b[3])])
    return inter / (area(a) + area(b) - inter)

rng = np.random.default_rng(SEED)
rows, n_obj, k_count, zmin, gaps = [], 0, [], np.inf, []
mis_sil = mis_vis = n_pix = 0
diff_bnd = diff_int = mask_pix = box_changed = flip = 0
for _ in range(N_SCENES):
    objs = make_scene(rng)
    k_count.append(len(objs)); n_obj += len(objs)
    zmin = min(zmin, objs[0]["C"][2] - objs[0]["s"][2])
    gaps += [(b["C"][2] - b["s"][2]) - (a["C"][2] + a["s"][2]) for a, b in zip(objs, objs[1:])]
    inst, depth, cls, d = render(objs, CENTRE)
    image = np.where(inst > 0, (0.3 + 0.3 * cls) * 5 / np.where(inst > 0, depth, 1), 0.1)  # shaded image
    sub = np.array([render(objs, g)[0] for g in SUB])          # instance ids at the 16 sub-samples
    boundary = maximum_filter(inst, 3, mode="nearest") != minimum_filter(inst, 3, mode="nearest")
    nearer = np.zeros((H, W), bool)
    for k, o in enumerate(objs):
        sil, vis = np.isfinite(d[k]), inst == k + 1
        sil_an = analytic_silhouette(o)
        mis_sil += int((sil != sil_an).sum())
        mis_vis += int((vis != (sil_an & ~nearer)).sum())
        nearer |= sil_an; n_pix += H * W
        ss = (sub == k + 1).sum(0) >= SS * SS / 2               # coverage >= 0.5
        diff = ss != vis
        diff_bnd += int((diff & boundary).sum()); diff_int += int((diff & ~boundary).sum())
        mask_pix += int(vis.sum())
        uv = project(o["C"] + o["s"] * SIGNS)                   # A1: projected 3D box, then clipped
        a1u = np.r_[uv.min(0), uv.max(0)]
        a1 = np.r_[np.clip(a1u[[0, 2]], 0, W), np.clip(a1u[[1, 3]], 0, H)][[0, 2, 1, 3]]
        nvis, nsil = int(vis.sum()), int(sil.sum())
        if nvis == 0:
            rows.append(dict(cls=o["cls"], nvis=0)); continue
        V, A2 = pixbox(vis), pixbox(sil)
        O1 = nvis / area(a1)
        if ss.any():
            box_changed += int(not np.array_equal(pixbox(ss), V))
            flip += int((O1 >= 0.5) != (ss.sum() / area(a1) >= 0.5))
        else:
            box_changed += 1; flip += int(O1 >= 0.5)
        rows.append(dict(cls=o["cls"], nvis=nvis, occluded=nsil - nvis, T=1 - area(a1) / area(a1u),
                         O1=O1, O1u=nvis / area(a1u), O2=nvis / area(A2),
                         iou1=iou(a1, V), iou2=iou(A2, V), h1=a1[3] - a1[1], hv=V[3] - V[1]))

lab = [r for r in rows if r["nvis"] > 0]
col = lambda key, rs=lab: np.array([r[key] for r in rs])
cl, T, occ = col("cls"), col("T"), col("occluded")
print(f"seed {SEED}: {N_SCENES} scenes, {n_obj} objects ({(col('cls', rows) == 1).sum()} boxes, "
      f"{(col('cls', rows) == 2).sum()} ellipsoids), {min(k_count)}-{max(k_count)} per scene")
print(f"nearest face {zmin:.3f} m, smallest depth gap {min(gaps):.3f} m, "
      f"objects with no visible pixel {len(rows) - len(lab)}, labelled {len(lab)}")
print(f"labelled objects truncated (T > 0) {(T > 0).sum()} ({100 * (T > 0).mean():.1f}%), "
      f"occluded (>= 1 hidden pixel) {(occ > 0).sum()} ({100 * (occ > 0).mean():.1f}%)")
print(f"C1 z-buffer vs closed form: silhouette mismatches {mis_sil}, visible-mask mismatches {mis_vis} "
      f"(of {n_pix} object-pixels)")

clean = (T == 0) & (occ == 0)
i1, i2 = col("iou1"), col("iou2")
print(f"\nH1 untruncated, unoccluded objects: {clean.sum()}; IoU(A2, V) == 1 for {(i2[clean] == 1).sum()}")
for c, name in ((1, "boxes"), (2, "ellipsoids")):
    m = clean & (cl == c)
    print(f"   IoU(A1, V) {name:10s} n={m.sum():4d}  mean {i1[m].mean():.4f}  "
          f"min {i1[m].min():.4f}  max {i1[m].max():.4f}  (< 1: {(i1[m] < 1).sum()})")

O1 = col("O1")
print("\nH2 occupancy O1 bin    n   mean IoU(A1,V)  mean IoU(A2,V)  min IoU(A2,V)")
for lo, hi in ((0, .25), (.25, .5), (.5, .75), (.75, np.inf)):
    m = (O1 >= lo) & (O1 < hi)
    print(f"   [{lo:.2f}, {hi:.2f})  {m.sum():5d}     {i1[m].mean():.6f}        "
          f"{i2[m].mean():.6f}       {i2[m].min():.4f}")

print("\nH3 filter                   kept   % of labelled  % ellipsoids")
print(f"   (none)                  {len(lab):5d}       100.0          {100 * (cl == 2).mean():.2f}")
filters = [(f"{k} >= {t}", col(k) >= t) for k in ("O1", "O1u", "O2") for t in (0.25, 0.5)]
filters += [("T <= 0.5", T <= 0.5), ("A1 height >= 25 px", col("h1") >= 25),
            ("V height >= 25 px", col("hv") >= 25)]
for name, m in filters:
    print(f"   {name:22s}  {m.sum():5d}       {100 * m.mean():5.1f}          {100 * (cl[m] == 2).mean():.2f}")

print(f"\nH4 supersampled (>= 8 of 16) vs centre-sampled masks: {mask_pix} visible pixels, "
      f"{diff_bnd + diff_int} differ: {diff_bnd} on boundary pixels, {diff_int} interior")
print(f"   objects whose visible box changes {box_changed}, whose O1 >= 0.5 decision flips {flip}")
seed 0: 500 scenes, 2807 objects (1446 boxes, 1361 ellipsoids), 3-8 per scene
nearest face 3.012 m, smallest depth gap 0.201 m, objects with no visible pixel 341, labelled 2466
labelled objects truncated (T > 0) 1273 (51.6%), occluded (>= 1 hidden pixel) 238 (9.7%)
C1 z-buffer vs closed form: silhouette mismatches 0, visible-mask mismatches 0 (of 53894400 object-pixels)

H1 untruncated, unoccluded objects: 1051; IoU(A2, V) == 1 for 1051
   IoU(A1, V) boxes      n= 525  mean 0.9323  min 0.7406  max 0.9913  (< 1: 525)
   IoU(A1, V) ellipsoids n= 526  mean 0.6713  min 0.4512  max 0.9587  (< 1: 526)

H2 occupancy O1 bin    n   mean IoU(A1,V)  mean IoU(A2,V)  min IoU(A2,V)
   [0.00, 0.25)    152     0.206973        0.636953       0.0106
   [0.25, 0.50)    485     0.546746        0.966928       0.3714
   [0.50, 0.75)    712     0.734361        0.990194       0.5000
   [0.75, inf)   1117     0.937444        0.999242       0.8235

H3 filter                   kept   % of labelled  % ellipsoids
   (none)                   2466       100.0          47.40
   O1 >= 0.25               2314        93.8          45.33
   O1 >= 0.5                1829        74.2          32.15
   O1u >= 0.25              1866        75.7          44.53
   O1u >= 0.5               1209        49.0          32.18
   O2 >= 0.25               2404        97.5          47.38
   O2 >= 0.5                2358        95.6          47.07
   T <= 0.5                 1757        71.2          48.61
   A1 height >= 25 px        657        26.6          46.27
   V height >= 25 px         582        23.6          39.18

H4 supersampled (>= 8 of 16) vs centre-sampled masks: 969245 visible pixels, 10560 differ: 10534 on boundary pixels, 26 interior
   objects whose visible box changes 771, whose O1 >= 0.5 decision flips 24

Checks C1 and D1–D4 held. The z-buffer’s silhouettes and visible masks agree with the closed forms at every one of the 53.9 million object-pixels, so the buffers the labels come from are exact at pixel centres. The data checks held as well: 3 to 8 objects per scene, a smallest depth gap of 0.201 m, a nearest face at 3.012 m, 51.6% of labelled objects truncated and 9.7% occluded, and 1,051 objects for H1. About one object in eight (341 of 2,807) has no visible pixel, either hidden behind nearer objects or wholly outside the image, so it gets no label at all.

H1 held, and H1b held.

  • A2 against V (H1). For all 1,051 untruncated, unoccluded objects, the tight box of the full silhouette and the visible box are the same box. This follows from the construction: when nothing is hidden or clipped, the two masks are the same pixels.
  • A1 against V (H1b). Gaidon et al.’s projected 3D box never matched the visible box, with mean IoU 0.9323 for cuboids and 0.6713 for ellipsoids.
    • For a cuboid, the projection of its 3D box is its own outline’s bounding box, so A1 and V differ only because A1 has continuous edges and V is rounded to pixel edges.
    • For an ellipsoid, the corners of the enclosing box project outside the ellipsoid’s outline, so A1 is larger than any box drawn around the visible object.

Which of the two readings of “amodal box” a dataset uses therefore changes even the boxes of fully visible objects.

H2 held for both amodal boxes. Mean IoU with the visible box rises across the four occupancy bins: from 0.207 to 0.937 for A1 and from 0.637 to 0.999 for A2. On the 50-scene planning subset of the pilot seed, H2 for A2 had failed its last step (1.0000, then 0.9995 at four decimals). It held in the pilot (seed 1, 500 scenes) and in the run shown. Low O1 does not by itself mean occlusion: an ellipsoid’s outline covers at most π/4 of the area of its own bounding box, and A1 is larger than that box. In our reading, this is why A2’s mean IoU is already close to 1 in the [0.5, 0.75) bin.

H3 held. Each definition keeps more objects at 0.25 than at 0.5, and the three counts at 0.5 differ:

  • O1 ≥ 0.5 keeps 74.2% of labelled objects.
  • O1u ≥ 0.5, the same threshold over the unclipped box, keeps 49.0%. For a truncated object the unclipped area includes the part outside the image, so truncation also lowers its occupancy.
  • O2 ≥ 0.5 keeps 95.6%.
  • Minimum height of 25 pixels keeps about a quarter of the objects whether it is applied to A1 (26.6%) or to V (23.6%).

H5 held. At O1 ≥ 0.5, ellipsoids fall from 47.40% of labelled objects to 32.15% of kept ones. At O2 ≥ 0.5 they are 47.07%. The height filter shows the same shape effect: ellipsoids are 46.27% of objects kept by A1 height but 39.18% of those kept by V height, because an ellipsoid’s A1 is taller than its outline. A filter written for one shape of object therefore changes the class balance of a test set in which shapes differ.

H4 failed. Of 969,245 visible pixels, 10,560 change owner between centre sampling and 8-of-16 coverage. Of those, 10,534 are on boundary pixels and 26 are interior pixels, so the planned bar of zero interior differences was not met. In the pilot H4 also failed (12 interior pixels of 10,109 differing). No other outcome differed between the pilot and the run shown.

Beyond H4, the pixel convention changed the visible box of 771 of the 2,466 labelled objects and flipped the O1 ≥ 0.5 keep-or-drop decision for 24. Generated labels are exact only relative to a sampling rule, and a dataset that does not state its rule leaves boundary pixels, and with them some boxes and a few filter decisions, open.

Variants

  • By what is synthetic. The whole image may be synthetic (Virtual KITTI, SYNTHIA, Meta-Sim). Only the composition may be (Dwibedi et al.’s real cut-outs on real backgrounds, SynthText’s rendered text on background images). The image may be synthetic with its appearance changed afterwards (SimGAN’s refined images, or the image translation Meta-Sim adds). Or the image may come from a generative model whose label is its prompt (Sariyildiz et al.).
  • By how the scene is chosen. A world can be modelled after real recordings (Virtual KITTI’s cloning), sampled from hand-set distributions (scene grammars, domain randomization) or fitted to real data (Meta-Sim’s distribution matching and task loss).
  • By purpose. Most studies use synthetic data for training. Gaidon et al. stress the reverse use: synthetic test data, whose labels must be accurate, for measuring how a model trained on real data behaves under controlled changes.
  • By label type. Classes, boxes, instance and semantic masks, depth, flow, 3D pose and keypoints all come from the same scene description. Robotic grasping and robotic manipulation describe synthetic depth and pose data for robots.

Best Practices

  • State the label definitions. Write down whether boxes are amodal or visible, what truncation and occlusion mean and how they are computed, the minimum size, and the pixel sampling rule. The example shows each of these changing labels or the kept set.
  • Match filters to the evaluation they imitate, as Gaidon et al. did with per-sequence occupancy and truncation thresholds.
  • Keep real data in the loop. In the comparisons below, mixing or fine-tuning with real data beat synthetic-only training on the real test set in every case but one: SYNTHIA’s FCN on CBCL, with 53.5% per-class accuracy mixed against 56.9% synthetic-only.
  • Vary what the real data varies, including layout. Kar et al. show detection accuracy changing when only the scene’s content changes (object attributes, camera, context), with the same renderer.
  • Remove artefacts the model could learn. Dwibedi et al. render each composite with several blendings so that boundary artefacts stop predicting the label.
  • Check the generator’s licence and provenance. Game assets, background photographs and the training data of generative models carry their own terms and biases. Sariyildiz et al. note that Stable Diffusion was trained on LAION-2B, scraped from the internet and filtered automatically, which has been shown to contain problematic content.

Pitfalls

  • Calling generated labels perfect. Meta-Sim’s authors write that modified synthetic scenes “will still render perfect ground-truth” (Kar et al., 2019, §3). The masks are exact at the sample points, but boundary pixels change owner under another sampling rule (771 visible boxes changed in the example), and the box definition itself is a choice.
  • Treating occupancy as occlusion. Under Gaidon et al.’s definition a fully visible ellipsoid has occupancy well below 1, so an occupancy threshold removes round objects before square ones. In the example an O1 ≥ 0.5 filter cut the share of ellipsoids from 47.40% to 32.15%.
  • Comparing virtual and real metrics without matched ground truth. Gaidon et al. trace the lower MOTP and mostly-tracked ratios on their virtual clones mainly to the inconsistent manual labels of heavily truncated and occluded cars in real KITTI.
  • Expecting synthetic-only training to transfer. In the studies below, synthetic-only training trailed real training on the same real test data by 7.6 and 14.4 MOTA points (Virtual KITTI), 10.1 and 5.4 mAP points (Dwibedi et al.) and 33.2 top-1 points (ImageNet-1K); only SYNTHIA’s came close to, or above, small real training splits.
  • Learning the generator’s signature. Shrivastava et al. warn that networks trained on synthetic images can learn details present only in those images. Dwibedi et al. found that unblended pasting gave a poorer detector than Gaussian blurring or a mix of blendings (65.9 against 68.9 and 73.7 mAP), although Poisson blending alone scored lower still (58.4).
  • Bias inherited from the generator. Sariyildiz et al. write that synthetic clones have “no reason to be immune to bias”, and that the generator can reinforce stereotypes through a lack of diversity.

Evidence and Impact

Each result below holds for its own data, model and metric. Numbers from different studies are not comparable.

Virtual KITTI (Gaidon et al., 2016) compared trackers on five real KITTI training sequences and their virtual clones, using CLEAR MOT metrics and a Fast R-CNN car detector. Average MOTA was 81.7% real against 82.2% virtual for the DP-MCF tracker, and 85.9% against 86.0% for MDP (Table 1). MOTP was lower on the clones (85.7% against 78.9% for DP-MCF). For training, the authors scored seven held-out real KITTI training sequences (Table 2):

Tracker Trained on virtual Trained on real Virtual, then real
DP-MCF (MOTA) 64.3% 71.9% 76.7%
MDP (MOTA) 63.7% 78.1% 78.7%

Under the controlled variations (Table 3), fog lowered MOTA on the clones by 45.2 points for DP-MCF and 57.4 for MDP. Turning the camera, changing the time of day, overcast and rain lowered it by 2.0 to 8.1 points for DP-MCF and 4.0 to 12.0 for MDP.

SYNTHIA (Ros et al., 2016) trained T-Net and FCN segmentation networks on images resized to 180 × 120 pixels and scored 11 classes on the validation splits of four real datasets.

  • Synthetic only. Training on SYNTHIA-Rand alone gave per-class accuracies close to, and sometimes above, training on each dataset’s small real training split. For T-Net on CamVid it was 48.9% against 46.3% (Tables 2 and 3).
  • Mixed with real data. Adding SYNTHIA-Rand to the real training split raised per-class accuracy in every case: from 46.3% to 56.5% for T-Net on CamVid, and from 52.8% to 72.1% for FCN on CamVid (Table 3).
  • Global accuracy. This rose for T-Net but fell for FCN on KITTI, U-LabelMe and CBCL, from 79.7% to 75.2% on CBCL, for example. The authors suggest the drop may relate to FCN’s upsampling and to the batch mixing, and leave it to further investigation.

Cut, Paste and Learn (Dwibedi et al., 2017) trained Faster R-CNN with a VGG-16 pretrained on COCO and scored mAP at IoU 0.5 for 11 instances on the GMU Kitchen test splits, averaged over three splits.

  • Blending (Table 1): no blending gave 65.9 mAP, Gaussian blurring 68.9, Poisson blending 58.4, all three blendings 72.4, and all three on the same scene 73.7. Adding distractors raised it to 76.2. Removing occlusion lowered it to 63.1.
  • Against real data (Table 2): the synthetic data scored 76.2, real GMU training images 86.3, and the two together 88.8.
  • Across datasets (Table 3: 6 instances on the Active Vision dataset, training on GMU images): real training images scored 41.9, synthetic 36.5 and both 51.1. Ten percent of the real images scored 15.8, and the same 10% plus synthetic images 43.2.

SimGAN (Shrivastava et al., 2017) estimated gaze on real MPIIGaze eye images with a CNN trained on synthetic UnityEyes images. The mean error was 11.2° when training on synthetic images and 7.8° on refined images, against 9.9° for the best earlier method, also trained on UnityEyes (Table 3). Ten subjects told refined from real images correctly 517 times in 1,000 trials, against 162 of 200 for unrefined images. On 100 synthetic images and their refined versions, hand-fitted pupil centres moved by 1.1 ± 0.8 pixels on 55-pixel-wide eyes. This is their check that refinement keeps the label.

Meta-Sim (Kar et al., 2019) trained Mask R-CNN (ResNet-50-FPN, ImageNet initialization) on rendered driving scenes from a simplified road grammar, rendered in Unreal Engine 4, and scored car detection AP at IoU 0.5 on KITTI’s validation set (easy, moderate, hard; Table 3):

Training data Easy Moderate Hard
Scenes from the hand-set grammar 63.7 63.7 62.2
Meta-Sim with the task loss 66.7 66.3 66.2

With MUNIT image translation added to both (Table 4), the grammar scored 71.1, 75.5 and 65.3 and Meta-Sim 77.5, 75.1 and 68.2. Meta-Sim was higher on easy and hard and lower on moderate. The authors read the remaining differences as support for their content-gap hypothesis.

Synthetic ImageNet clones (Sariyildiz et al., 2023) trained ResNet-50 from scratch on Stable Diffusion images of the ImageNet classes and tested on real images.

  • ImageNet-1K validation set (Table 1): the best clone scored 42.9% top-1 and 70.3% top-5, against 76.1% and 92.9% for the ResNet-50 trained on the real images that the paper lists as PyTorch’s.
  • ImageNet-100: a clone 50 times larger than the real set reached 73.3% top-1, against 87.4% for real training.
  • Transfer (Table 2): used as frozen feature extractors with linear classifiers on ten transfer datasets, the ImageNet-1K clone’s encoder averaged 68.4% top-1, against 65.7% and 66.5% for the two encoders trained on real ImageNet-1K.

The authors write that this suggests synthetic images can pretrain strong general-purpose encoders. Classification on ImageNet’s own classes still lags.

Overall. In these studies, synthetic data is most clearly useful added to real data or as pretraining before real fine-tuning. In every comparison above, that configuration scored higher than real-only training on the main metric, though SYNTHIA’s global accuracy fell for FCN on three datasets. Synthetic data also serves as a controlled test bed. Where the gap comes from varies by study: appearance (SimGAN, the blending ablation), content and layout (Meta-Sim), label conventions (Virtual KITTI’s MOTP), or the generator’s semantics (the ImageNet clones).

Related

  • Domain Adaptation

    Adapting a model trained on labelled images from one domain to a shifted target domain, usually with only unlabelled target images; the settings, the theory of what alignment can and cannot guarantee, discrepancy, adversarial, pixel-level, self-training and test-time methods, domain generalization, and the benchmarks.

  • Embodied AI

    The study of agents that perceive their surroundings through their own sensors and act in them, covering where the idea comes from, simulators used as research platforms, embodied tasks from navigation and question answering to instruction following and rearrangement, learning by imitation, reinforcement and large pretrained models including vision-language-action models, transfer from simulation to real robots, and how embodied agents are evaluated.

  • Image Blending

    Joining two images, or parts of them, so that the transition is invisible; hard cuts, feathering, multiband (Laplacian-pyramid) blending and Poisson (gradient-domain) blending, and the trade-off each makes between a visible seam, overlapping textures and a shift in brightness.

  • Flying Chairs

    A synthetic optical flow dataset of rendered chairs moving over Flickr photographs, created to train FlowNet and still used to pretrain flow networks.

  • CARLA

    An open-source driving simulator built on Unreal Engine, with a server–client architecture, synchronous stepping, camera, depth, segmentation and LiDAR sensors, a Traffic Manager and a recorder, described from its 0.9.16 documentation and LibCarla source, with an example that reproduces its depth encoding and camera coordinates.

  • Instance Segmentation

    Detecting every object of a fixed set of "thing" classes and giving each one its own pixel mask, class label and confidence score, so that two touching cars come out as two cars rather than one car-colored region.

References

  1. Gaidon, A., Wang, Q., Cabon, Y. & Vig, E. (2016). Virtual Worlds as Proxy for Multi-Object Tracking Analysis. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  2. Ros, G., Sellart, L., Materzynska, J., Vazquez, D. & Lopez, A. M. (2016). The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3234–3243.
  3. Richter, S. R., Vineet, V., Roth, S. & Koltun, V. (2016). Playing for Data: Ground Truth from Computer Games. European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, 102–118 (not read).
  4. Mayer, N., Ilg, E., Häusser, P., Fischer, P., Cremers, D., Dosovitskiy, A. & Brox, T. (2016). A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4040–4048 (not read).
  5. Gupta, A., Vedaldi, A. & Zisserman, A. (2016). Synthetic Data for Text Localisation in Natural Images. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2315–2324 (not read).
  6. Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W. & Webb, R. (2017). Learning from Simulated and Unsupervised Images through Adversarial Training. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  7. Dwibedi, D., Misra, I. & Hebert, M. (2017). Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection. IEEE International Conference on Computer Vision (ICCV).
  8. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W. & Abbeel, P. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 23–30 (not read).
  9. Kar, A., Prakash, A., Liu, M.-Y., Cameracci, E., Yuan, J., Rusiniak, M., Acuna, D., Torralba, A. & Fidler, S. (2019). Meta-Sim: Learning to Generate Synthetic Datasets. IEEE/CVF International Conference on Computer Vision (ICCV).
  10. Sariyildiz, M. B., Alahari, K., Larlus, D. & Kalantidis, Y. (2023). Fake It Till You Make It: Learning Transferable Representations from Synthetic ImageNet Clones. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).