Vision-Based Robotic Manipulation

Vision-based robotic manipulation uses cameras and depth sensors to decide how a robot should change the state of objects: pick one out of a bin and place it, insert it into a fixture, fold it, push it or hang it up. Robotic grasping answers where a gripper can take hold of an object; manipulation needs whatever else the task requires. Placing a part in a fixture needs its full position and orientation, its 6D pose; hanging a mug needs its handle; folding a cloth needs a state that no rigid pose describes. A modular pipeline estimates an explicit state, usually object poses, and passes it to a planner and a controller. A learned policy maps images more or less directly to actions, trained from demonstrations or trials. Between them sit keypoints and pixel-wise affordance maps, which keep some structure without a full object model.

Problem Definition

In the modular formulation, the central perception problem is object pose estimation. The pose of a rigid object is the transform $P = [R \mid \mathbf{t}]$, a rotation $R \in SO(3)$ and a translation $\mathbf{t} \in \mathbb{R}^3$, that maps a point $\mathbf{x}_m$ of the object’s 3D model into the camera frame, $\mathbf{x}_c = R\mathbf{x}_m + \mathbf{t}$ (Hodaň et al., 2018). It is the camera pose problem seen from the object’s side. With the camera-to-robot transform from hand–eye calibration, grasps and placements defined on the model become robot motions.

The BOP benchmark formulates the task for a single image (Hodaň et al., 2018). At training time a method receives, for each object, a 3D mesh model or images of the object in known poses. At test time it receives an image and an object identifier and must estimate the pose of one visible instance of that object. If several instances are present, any one may be reported. The authors chose this variant because it reflects industry-relevant bin picking, where a robot must grasp any single instance of the required part, such as a bolt or a nut.

Learned visuomotor policies define the problem differently: given observations $o_t$ (images, and usually the robot’s own joint state), output actions $a_t$ that complete the task, judged by whether the task succeeds rather than by the accuracy of any intermediate state. Transporter Networks, for example, frame pick-and-place as $f(o_t) \to a_t = (\mathcal{T}_{\text{pick}}, \mathcal{T}_{\text{place}})$, two end-effector poses in $SE(2)$ or $SE(3)$ (Zeng et al., 2021).

Input and Output

The usual input is an RGB-D image or a point cloud from a camera fixed above the workspace or mounted on the robot’s wrist, with known intrinsic parameters. Some methods use color alone; FoundationPose, for instance, needs RGB-D (Wen et al., 2024). Hand–eye calibration and robot guidance in factories are covered in the Industrial Machine Vision article.

The output depends on the formulation:

• Object poses in $SE(3)$, one per detected object instance, often with a confidence, and the object identity from detection or instance segmentation.
• Pixel-wise action maps, such as the probability of picking success for each pixel and gripper orientation (Zeng et al., 2022), or pick and place poses read off such maps (Zeng et al., 2021).
• Keypoints on the object, such as the bottom center and handle of a mug, from which a target configuration is planned (Manuelli et al., 2022).
• Actions directly: joint torques (Levine et al., 2016) or a short sequence of end-effector positions (Chi et al., 2023).

Variants

• Model-based or model-free. With a 3D model of the object, such as a CAD model, pose estimation can register the model to the image; without one, a method works from reference images, from category knowledge, or with no object representation at all.
• Instance, category or novel objects. Wen et al. (2024) distinguish instance-level methods, which work only on the objects seen in training, category-level methods, which generalize within a predefined category, and methods for arbitrary novel objects given a CAD model (their model-based setting) or a few reference images (model-free) at test time.
• Tasks. Pick-and-place (kitting, palletizing, sorting, packing; bin picking when many objects share a container), assembly and insertion with tolerances of millimeters or less, and manipulation of deformable objects such as cables, cloth and bags.
• Open-loop or closed-loop. A pose estimated once, followed by a planned motion, against perception repeated during the motion, either through pose tracking, through visual servoing, which controls motion directly from image features, or through a learned policy that re-plans every few steps.

Challenges

• Occlusion and clutter. In the BOP evaluation of 15 methods, every method scored at least 30% better on LM than on LM-O, which contains the same objects partly occluded, and scores dropped quickly at low levels of occlusion (Hodaň et al., 2018).
• Symmetry and pose ambiguity. A can rotated about its axis looks the same, and a part seen from one side may match several poses. Training a network to regress one rotation for such objects gives inconsistent targets (Xiang et al., 2018), and evaluation must not penalize poses that are indistinguishable (Hodaň et al., 2018).
• Texture-less, shiny and similar objects. Feature-point methods need texture (Xiang et al., 2018). In the first BOP evaluation, methods based on 3D local features and learning-based methods scored very low on T-LESS, which the authors attribute to its objects’ symmetries and similarities (Hodaň et al., 2018). Depth sensors may miss glossy surfaces: BOP 2020 changed its error computation so that pixels without depth measurements count as visible, to evaluate the shiny parts of the ITODD dataset (Hodaň et al., 2020).
• Lighting. On BOP’s TUD-L dataset, built to vary lighting, methods trained on synthetic color images rendered with fixed lighting struggled, while methods using only depth were noticeably more robust (Hodaň et al., 2018).
• Precision. A pose accurate to a centimeter may not be enough. In Levine et al.’s (2016) experiments, a vision network trained for pose prediction was accurate to about 1 cm, but tasks such as screwing on a bottle cap had tolerances of a few millimeters; the authors also cite a camera-to-end-effector accuracy of about 2 cm during open-loop motion on the robot they used. Calibration errors and arm positioning add to perception errors, as they do in grasping.
• Training data. Annotating real images with 6D poses takes significant human effort, so most learned methods train largely on synthetic images rendered from 3D models (Hodaň et al., 2020). Demonstrations and robot trials for learned policies are expensive in robot time.
• Many valid actions. A block can be picked at many points, and a demonstrator may approach from either side; a policy that averages such modes matches none of them (Zeng et al., 2021; Chi et al., 2023).

Approaches

Pose estimation of known objects

BOP’s first evaluation compared the classical families with early learned methods. Feature matching finds 2D–3D correspondences between image keypoints and the model and solves a Perspective-n-Point problem with RANSAC. Template matching compares the image with views of the model rendered from many viewpoints, and occlusion degrades it significantly (Xiang et al., 2018). Point-pair features (PPF) match pairs of oriented 3D points between the scene’s point cloud and the model, each pair described by its distance and relative orientation, and vote for a pose; because they use only the model, they need no training images (Hodaň et al., 2020). In the first BOP evaluation, PPF methods performed best: the top method, by Vidal et al., reached an average recall of 74.6% at the default thresholds, ahead of template matching, learning-based methods and methods based on 3D local features, with the best learning-based method at 55.4% (Hodaň et al., 2018).

Many methods end with a refinement step in which the coarse pose initializes iterative closest point against the depth data; ICP converges only locally, so the coarse estimate decides where it ends.

PoseCNN (Xiang et al., 2018) decouples the pose into parts a convolutional neural network can learn. It labels each pixel with an object class, predicts at each pixel a unit vector pointing toward the object’s center, and lets pixels of each class vote for the center in the image, which stays possible when the center itself is occluded. A predicted distance of the center from the camera then gives the 3D translation, and the rotation is regressed as a quaternion from features inside the object’s box. For symmetric objects the authors introduced ShapeMatch-Loss, which measures the distance from each model point under the estimated rotation to the closest model point under the true rotation, so that rotations equivalent under the object’s symmetry are not penalized. On their YCB-Video dataset, with color alone, PoseCNN reached an area under the accuracy–threshold curve of 53.7 for ADD and 75.9 for ADD-S (metrics below); refining with depth-based ICP raised these to 79.3 and 93.0. Because ICP is not robust to local minima, they refined several perturbed copies of the network’s estimate.

DenseFusion (Wang et al., 2019) uses depth inside the network rather than only in post-processing. After semantic segmentation, a convolutional network embeds the color of each pixel in the object’s crop, and a PointNet-style network embeds each 3D point of the masked depth; the two are fused per pixel, each fused feature predicts a pose with a confidence, and a small network refines the pose iteratively. On YCB-Video, with PoseCNN’s segmentation masks for all methods, they report a mean ADD-S AUC of 93.1 and 96.8% of poses with ADD-S below 2 cm, against 93.0 and 93.2% for PoseCNN with ICP, at about 0.06 s per frame against 10.6 s, most of which went to ICP (their Table 1 and runtime table). On a robot, without explicit domain adaptation and with a different RGB-D sensor and background than in the training data, grasps that aligned the fingers with each object’s narrower dimension from the estimated pose succeeded in 73% of 60 attempts on five YCB objects placed at random positions and orientations on a table, with the banana hardest (7 of 12).

In the BOP Challenge 2020, methods based on deep networks caught up with PPF methods, which had won the 2017 and 2019 challenges. Almost all of them applied networks to the color channels only, and many used depth for ICP refinement; the top method, a variant of CosyPose, scored an $\text{AR}_{\text{Core}}$ of 69.8%, of which ICP refinement contributed 6.1 points, and the third-placed method used no depth at all. The organizers supplied 350,000 physically based rendered training images, and they report that the photorealism of these images improved results despite strong data augmentation (Hodaň et al., 2020).

Novel objects. FoundationPose (Wen et al., 2024) is one model for unseen objects in both settings: given a CAD model or a few reference images (16 in its experiments), from which it builds a neural object field on a signed distance function representation for rendering new views. It detects the object with an off-the-shelf detector such as Mask R-CNN, initializes the translation from the median depth inside the box and the rotation from viewpoints sampled on a sphere, refines each hypothesis by comparing a rendering with the image crop using a transformer-based network, and ranks the refined hypotheses. Trained only on synthetic data, it reports an average recall of 78.8, 83.0 and 88.0 on the BOP datasets LM-O, T-LESS and YCB-V in the model-based setting, all methods using RGB-D and Mask R-CNN detections, a mean of 83.3 against 79.7 for an instance-level method with ICP. The authors report about 1.3 s for pose estimation of one object and about 32 Hz for tracking on their desktop GPU.

Example: pose of a known part from a depth image

The example renders a depth image of a table seen from 0.6 m above, with two known objects standing on it: an L-shaped bracket and a can. It segments the objects as pixels more than 5 mm above the table, back-projects each segment to a point cloud, and registers the object’s model to it with point-to-point ICP, whose inner step is the closed-form rigid fit of the ICP article. Because the object stands upright on a known table, the coarse pose has one unknown rotation, the yaw, and twelve starts 30° apart are refined; the pose with the lowest residual is kept. The scene points are the source and the full model the target, so every observed point has a counterpart. Each result is compared with the true pose by rotation and translation error and by the ADD, ADD-S and MSSD errors defined under Evaluation Metrics, counted correct below a tenth of the object’s diameter $d$: the threshold Xiang et al. (2018) use for ADD and ADD-S on OccludedLINEMOD; a single threshold for MSSD is this example’s choice, where BOP averages over several.

import sys
import numpy as np
from scipy import ndimage
from scipy.spatial import ConvexHull, cKDTree
from scipy.spatial.distance import pdist
from scipy.spatial.transform import Rotation

seed = int(sys.argv[1]) if len(sys.argv) > 1 else 0
rng = np.random.default_rng(seed)

# --- Object models (metres), origin at the centre of the base, z up ---------
def box_surface(n, lo, hi):
    """n points uniformly on the surface of an axis-aligned box."""
    lo, hi = np.asarray(lo, float), np.asarray(hi, float)
    s = hi - lo
    areas = np.array([s[1] * s[2], s[1] * s[2], s[0] * s[2], s[0] * s[2], s[0] * s[1], s[0] * s[1]])
    face = rng.choice(6, size=n, p=areas / areas.sum())
    p = lo + rng.uniform(size=(n, 3)) * s
    ax = face // 2
    p[np.arange(n), ax] = np.where(face % 2 == 0, lo[ax], hi[ax])
    return p

def bracket(n):
    """L-shaped bracket: a 90 x 30 x 15 mm foot and a 30 x 30 x 60 mm upright."""
    a = box_surface(n * 2 // 3, [-0.045, -0.015, 0.0], [0.045, 0.015, 0.015])
    b = box_surface(n - n * 2 // 3, [0.015, -0.015, 0.015], [0.045, 0.015, 0.075])
    return np.vstack([a, b])          # the face where the boxes meet is kept

def can(n, r=0.035, h=0.10):
    """Closed cylinder of radius r and height h."""
    side = 2 * np.pi * r * h
    cap = np.pi * r ** 2
    k = rng.choice(3, size=n, p=np.array([side, cap, cap]) / (side + 2 * cap))
    th = rng.uniform(0, 2 * np.pi, n)
    rad = np.where(k == 0, r, r * np.sqrt(rng.uniform(size=n)))
    z = np.where(k == 0, rng.uniform(0, h, n), np.where(k == 1, 0.0, h))
    return np.column_stack([rad * np.cos(th), rad * np.sin(th), z])

# --- Camera 0.6 m above the table, looking straight down ---------------------
W, H, f, cx, cy, D_TABLE = 320, 240, 500.0, 160.0, 120.0, 0.6
R_CW = np.diag([1.0, -1.0, -1.0])            # world (z up) -> camera (z forward)
T_CW = np.array([0.0, 0.0, D_TABLE])

def pose(x, y, yaw_deg):
    """Object-to-camera pose of an object standing upright at table point (x, y)."""
    Rw = Rotation.from_euler("z", yaw_deg, degrees=True).as_matrix()
    return R_CW @ Rw, R_CW @ np.array([x, y, 0.0]) + T_CW

def render(objects):
    """Depth image by z-buffering dense surface samples; table plane behind."""
    u, v = np.meshgrid(np.arange(W), np.arange(H))
    depth = np.full((H, W), D_TABLE)
    for pts, (R, t) in objects:
        pc = pts @ R.T + t
        pu = np.round(f * pc[:, 0] / pc[:, 2] + cx).astype(int)
        pv = np.round(f * pc[:, 1] / pc[:, 2] + cy).astype(int)
        ok = (pu >= 0) & (pu < W) & (pv >= 0) & (pv < H)
        np.minimum.at(depth, (pv[ok], pu[ok]), pc[ok, 2])
    return depth + rng.normal(scale=0.001, size=depth.shape)   # 1 mm noise

def backproject(depth, mask):
    v, u = np.nonzero(mask)
    z = depth[v, u]
    return np.column_stack([(u - cx) * z / f, (v - cy) * z / f, z])

# --- ICP: scene points (source) onto the full model (target) -----------------
def kabsch(p, q):
    p0, q0 = p.mean(0), q.mean(0)
    U, _, Vt = np.linalg.svd((p - p0).T @ (q - q0))
    R = Vt.T @ np.diag([1, 1, np.sign(np.linalg.det(Vt.T @ U.T))]) @ U.T
    return R, q0 - R @ p0

def icp(scene, tree, model, R, t, iters=60):
    """Refine the object-to-camera pose (R, t); returns pose and RMS residual."""
    Ri, ti = R.T, -R.T @ t                   # camera -> object, the inverse
    for _ in range(iters):
        p = scene @ Ri.T + ti
        d, j = tree.query(p)
        keep = d < 0.02                      # reject pairs farther than 2 cm
        dR, dt = kabsch(p[keep], model[j[keep]])
        Ri, ti = dR @ Ri, dR @ ti + dt
    d, _ = tree.query(scene @ Ri.T + ti)
    return Ri.T, -Ri.T @ ti, np.sqrt(np.mean(d ** 2))

# --- Error measures -----------------------------------------------------------
def errors(model, R, t, Rg, tg, symmetries):
    est, gt = model @ R.T + t, model @ Rg.T + tg
    rot = np.degrees(Rotation.from_matrix(R.T @ Rg).magnitude())
    add = np.linalg.norm(est - gt, axis=1).mean()                  # ADD
    adds = cKDTree(est).query(gt)[0].mean()                        # ADD-S
    mssd = min(np.linalg.norm(est - (model @ S.T) @ Rg.T - tg, axis=1).max()
               for S in symmetries)                                # MSSD
    return rot, 1000 * np.linalg.norm(t - tg), 1000 * add, 1000 * adds, 1000 * mssd

objects = {
    "bracket": (bracket, pose(-0.06, 0.02, 35.0), [np.eye(3)]),
    "can": (can, pose(0.08, -0.03, 0.0),
            [Rotation.from_euler("z", a, degrees=True).as_matrix() for a in range(0, 360, 1)]),
}
depth = render([(fn(200_000), P) for fn, P, _ in objects.values()])

# Segment: pixels more than 5 mm above the table, split into connected parts.
# Labels follow raster order (bracket first); a detector would name the parts.
labels, n = ndimage.label(depth < D_TABLE - 0.005)
print(f"depth image {W}x{H}, segments found: {n}")

for k, (name, (fn, (Rg, tg), syms)) in enumerate(objects.items()):
    model = fn(4000)                                 # sparse model for ICP
    diam = pdist(model[ConvexHull(model).vertices]).max()   # object diameter d
    tree = cKDTree(model)
    scene = backproject(depth, labels == k + 1)
    c = scene.mean(0)
    t0 = np.array([c[0], c[1], D_TABLE])             # base on the table plane
    results = []
    for yaw in range(0, 360, 30):                    # 12 rotation hypotheses
        R0 = R_CW @ Rotation.from_euler("z", yaw, degrees=True).as_matrix()
        R, t, rms = icp(scene, tree, model, R0, t0)
        results.append((rms, yaw, errors(model, R, t, Rg, tg, syms)))
    passed = [sum(e[i] < 0.1 * 1000 * diam for _, _, e in results) for i in (2, 3, 4)]
    print(f"\n{name}: {len(scene)} scene points, diameter {1000 * diam:.0f} mm; "
          f"starts within 0.1d by ADD / ADD-S / MSSD: {passed[0]} / {passed[1]} / {passed[2]} of 12")
    print(" start  rms(mm)  rot(deg)  trans(mm)  ADD(mm)  ADD-S(mm)  MSSD(mm)")
    for rms, yaw, (rot, tr, add, adds, mssd) in sorted(results):
        print(f"  {yaw:3d}   {1000 * rms:6.2f}   {rot:7.1f}   {tr:7.1f}  {add:7.1f}"
              f"   {adds:7.1f}   {mssd:7.1f}")

Output:

depth image 320x240, segments found: 2

bracket: 2355 scene points, diameter 120 mm; starts within 0.1d by ADD / ADD-S / MSSD: 5 / 11 / 5 of 12
 start  rms(mm)  rot(deg)  trans(mm)  ADD(mm)  ADD-S(mm)  MSSD(mm)
   90     1.49       0.3       0.5      0.5       0.5       0.7
    0     1.49       0.9       0.5      0.7       0.6       1.2
   30     1.49       0.7       0.5      0.6       0.6       1.0
  330     1.49       0.8       0.5      0.6       0.6       1.1
   60     1.49       0.2       0.6      0.6       0.6       0.7
  120     3.28      14.1      17.2     13.3       4.7      24.4
  300     3.29      14.2      17.4     13.4       4.8      24.7
  270     6.63     173.4       8.4     49.0       8.9     107.1
  180     6.67     179.4       8.4     49.1       8.9     106.7
  150     7.05     105.4      33.7     45.9      14.6     100.2
  210    29.76     178.8       2.8     58.3       9.4      98.5
  240    29.86     179.3       2.7     58.4       9.5      98.6

can: 4374 scene points, diameter 122 mm; starts within 0.1d by ADD / ADD-S / MSSD: 1 / 12 / 12 of 12
 start  rms(mm)  rot(deg)  trans(mm)  ADD(mm)  ADD-S(mm)  MSSD(mm)
  330     1.78      30.7       0.8     17.0       1.4       1.0
  270     1.78      89.6       0.8     45.2       1.5       1.1
  240     1.78     120.0       0.8     55.6       1.4       0.8
  300     1.78      59.2       0.9     31.7       1.5       1.0
   60     1.78      59.0       0.8     31.6       1.5       0.8
  210     1.79     150.5       0.7     62.1       1.4       1.0
    0     1.79       0.9       0.8      0.8       0.7       0.9
   90     1.79      90.7       0.7     45.6       1.5       0.9
   30     1.79      30.1       0.8     16.7       1.4       0.9
  180     1.79     179.7       0.8     64.2       1.4       0.9
  150     1.79     150.7       0.7     62.1       1.4       0.9
  120     1.79     119.6       0.6     55.5       1.4       0.8

For the bracket, five of the twelve starts, those within about 65° of the true yaw, converge to the correct pose with errors under 1° and 1 mm, and they also have the lowest residual, so the residual picks a correct pose. The other starts converge to wrong poses, several of them turned by about 180°. In our reading, the residual separates them only because the bracket is asymmetric and well observed; a near-symmetric part would leave wrong poses with small residuals too.

The error measures disagree in two instructive ways. On the bracket, ADD-S accepts 11 of the 12 results, including the four that are turned by 173° to 179°: with closest-point matching, each point of the turned model finds some nearby point of the true one, so the average stays below 12 mm although the upright stands at the wrong end. MSSD and ADD reject them. On the can, every start converges to a pose that fits equally well, with yaw errors spread over the whole circle, because the depth data cannot determine the rotation about the axis. ADD rejects eleven of these correct fits; ADD-S and MSSD, which allow for the symmetry, accept all twelve. For grasping the can, the rotation about its axis is irrelevant, which is the case for symmetry-aware measures; for the bracket, the closest-point measure accepts a pose that would send the gripper to the wrong end. With seeds 0 to 9 (python pose.py <seed>, which changes the model samples and the sensor noise), the counts were the same in every run, and the bracket pose with the lowest residual had a rotation error between 0.3° and 1.0° and a translation error between 0.3 and 0.7 mm.

Model-free pick-and-place

Classical pick-and-place recognizes the object, estimates its pose and plans a grasp on the model, which Zeng et al. (2022) describe as practical for tightly constrained scenarios but hard to scale to novel objects that have no 3D model or training data. Their system for the Amazon Robotics Challenge reversed the order: it grasps first and recognizes afterwards. Fully convolutional networks map RGB-D views of the bin (for grasps, re-projected into a top-down heightmap) to dense maps of picking success for four primitives (suction from above, suction from the side, a parallel-jaw grasp from above, and a flush grasp for objects against the bin wall), without segmenting or recognizing objects; the robot executes the best-scoring action. The grasped object is then held up to other cameras and recognized by matching its images to product images with a two-stream network, so that objects never seen in training need only their product images. They report a top-1 recognition accuracy of 88.6% on a mix of known and novel objects with their two-stage matching network.

Transporter Networks (Zeng et al., 2021) extend pixel-wise action maps to placing, without any object model, pose or keypoints. The RGB-D views are re-projected into a top-down orthographic image, so an object looks the same wherever it is. A fully convolutional network scores every pixel as a pick location. For the place, features of a crop around the chosen pick are rotated and cross-correlated with features of the whole scene, a learned form of template matching:

$$Q_{\text{place}}(\tau \mid o_t, \mathcal{T}_{\text{pick}}) = \psi\big(o_t[\mathcal{T}_{\text{pick}}]\big) * \phi(o_t)[\tau], \qquad \mathcal{T}_{\text{place}} = \arg\max_{\tau}\, Q_{\text{place}}(\tau \mid o_t, \mathcal{T}_{\text{pick}}),$$

where $\psi$ and $\phi$ are two feature networks. Translating or rotating the scene translates or rotates the predicted actions with it, a spatial equivariance that the authors link to the method’s sample efficiency. On ten simulated tasks learned from demonstrations, it reached more than 90% success on most tasks with 100 demonstrations; on block insertion it succeeded in all test runs from a single demonstration, where an image-to-action baseline reached 8% with 1,000, and on building a six-block pyramid it reached 78.2% with 1,000 demonstrations (task success averaged over 100 unseen test configurations). Baselines given the true object poses as input were also less sample-efficient, in the authors’ comparison.

Keypoint and affordance representations

A pose fixes an object to one rigid template, which cannot describe a category whose members differ in shape. kPAM (Manuelli et al., 2022) represents objects of a category by a few semantic 3D keypoints, such as the bottom center, top center and handle of a mug, and states the task as costs and constraints on those keypoints: to hang a mug, its handle keypoint must reach the rack’s peg. The pipeline segments the object, detects the keypoints in 3D, solves an optimization for the target transform and grasps with a geometric method. Because the target is stated on the keypoints, a mug of any shape can be placed without knowing its pose relative to a canonical mug. On real hardware with objects not seen in training, they report placing shoes on a rack in 98 of 100 trials over 20 shoes, and hanging mugs on a rack by the handle in all 100 trials with 25 regular-sized mugs but in 50% of 20 trials with five small mugs, whose tolerances were around 1 to 1.5 cm.

Keypoints can also be learned without labels. The visuomotor policies of Levine et al. (2016) end their convolutional layers with a spatial softmax: each feature map becomes a probability distribution over image positions, and its expected position is a 2D feature point, a differentiable soft-argmax. The policy acts on these points together with the robot’s joint state, and the points it learns lie on the target object and the gripper.

The pixel-wise maps of Zeng et al. (2022) and Transporter Networks are affordance representations: for every location they predict the outcome of an action there, rather than describing the object.

Assembly and insertion

Insertion tolerances are smaller than perception errors, so vision usually brings the part close and contact feedback finishes the job. Lee et al. (2019) inserted 3D-printed pegs of five cross-sections, with clearances of about 2 mm, using a policy that combines RGB images, readings of a wrist force–torque sensor and the robot’s own state through a representation learned with self-supervision. In their simulated ablation, the full model completed insertion in nearly 80% of evaluation episodes, while models without vision, without force, or without both stayed below 5%; they hypothesize that vision locates the box and the hole for reaching and alignment, whereas force is uninformative until contact. Levine et al.’s (2016) end-to-end policies inserted a block into a shape-sorting cube from images and joint state alone, with the success rates given below.

Deformable objects

Cables, cloth and bags have complex dynamics and high-dimensional configuration spaces, and their goals are hard even to specify, since no compact pose describes them (Seita et al., 2021). Seita et al. built DeformableRavens, a simulated benchmark of 12 tasks with cables, fabrics and bags, and added goal conditioning to Transporter Networks, so that a target configuration can be given as an image of the goal rather than as a pose. Transporter Networks had earlier been applied to a rope task (Zeng et al., 2021).

Learned visuomotor policies

End-to-end visuomotor policies (Levine, Finn, Darrell and Abbeel, 2016; a different paper from the same year’s large-scale grasping study by Levine et al.) map monocular RGB images and the robot’s joint state directly to motor torques with a convolutional network of about 92,000 parameters running at 20 Hz. They are trained with guided policy search, which turns policy search into supervised learning: trajectory optimization under full state information, available only at training time, supervises the network, which at test time sees only images and joint state. On four tasks (hanging a coat hanger, inserting a block into a shape-sorting cube, fitting a toy hammer’s claw under a nail, screwing on a bottle cap), the authors compared end-to-end training with a modular baseline whose vision network was first trained to predict the target’s pose. On the shape-sorting cube, the end-to-end policy succeeded in 96.3% of 27 trials at training positions, 91.7% of 36 trials at new positions and 87.5% of 40 trials with visual distractors; the pose-prediction baseline succeeded in none of its trials at training or new positions. The authors write that this baseline succeeded only on the coat hanger, the task needing the least accuracy, although their results table also gives it 8.9% and 18.3% on the toy hammer. The authors also note that their policies performed poorly under drastic changes of background or with distractors next to or occluding the objects.

Diffusion Policy (Chi et al., 2023) represents the policy as a conditional denoising diffusion model over actions: starting from noise, a network conditioned on the recent images refines a sequence of future actions in $K$ steps. Predicting a sequence and executing only its first part before re-planning, receding-horizon control, keeps actions consistent over time while remaining closed-loop, and sampling from a learned distribution lets the policy commit to one of several valid ways of doing a task instead of averaging them. Learning from demonstrations on 12 tasks from four benchmarks, the authors report an average improvement in success rate of 46.9% over the best prior methods. On a real pushing task, pushing a T-shaped block into a target region and then moving away, it succeeded in 95% of 20 trials, where success meant a final intersection over union with the target above the lowest in the human demonstrations, against 0% and 20% for the best variants of the two baselines; humans averaged an IoU of 0.84 against 0.80 for the policy.

Datasets and Benchmarks

Benchmark	Year	Content	What is scored
LINEMOD (LM) and Occluded LINEMOD (LM-O)	2012; occlusion annotations later	Texture-poor household objects (15 in BOP’s count, 13 in DenseFusion’s); LM-O annotates 8 objects in one cluttered video	ADD / ADD-S within $0.1d$; later BOP recall
YCB-Video	2018	21 YCB objects, 92 videos, 133,827 frames	AUC of ADD and ADD-S up to 10 cm
BOP	2018, challenges from 2017	8 datasets in 2018, 11 by 2020, in one format	Recall under VSD (2018); AR over VSD, MSSD, MSPD (2020)
YCB object and model set	2015	Physical objects with scans and models, and task protocols	Protocol-specific physical trials
Ravens; DeformableRavens	2020; 2021	10 simulated tabletop tasks; 12 tasks with cables, fabrics and bags	Task success, 0–100, over unseen test runs
Diffusion Policy’s suite	2023	Robomimic tasks, Push-T, Block Pushing, Franka Kitchen	Success rate; target coverage for Push-T

LINEMOD, created by Hinterstoisser et al. in 2012, was for years the most commonly used pose dataset, though it was not intended as a general benchmark: its lighting is constant and its objects are easy to distinguish, unoccluded and near the image center (Hodaň et al., 2018). DenseFusion’s authors describe it as 13 low-textured objects in 13 videos (Wang et al., 2019). Occluded LINEMOD annotates eight objects in one LINEMOD video of 1,214 frames with significant occlusion (Xiang et al., 2018); BOP credits the occlusion annotations to Brachmann et al., while PoseCNN cites Krull et al. for the dataset.

YCB-Video (Xiang et al., 2018) uses 21 objects from the YCB object set; the standard split trains on 80 of its 92 videos and tests on 2,949 key frames of the other 12, and 80,000 synthetic images were released for training. The authors list errors in its annotations from the RGB sensor’s rolling shutter, inaccurate object models, slight asynchrony between color and depth, and uncertain camera parameters.

BOP (Hodaň et al., 2018) brought eight datasets into one format: 3D models of 89 objects, 277,000 training images and 62,000 test images, including two new datasets with varying lighting, and an online evaluation system that stays open for submissions. By 2020 it held eleven datasets, seven of them core datasets on which challenge entries must be evaluated, with physically based rendered training images for the core datasets (Hodaň et al., 2020).

The YCB object and model set (Calli et al., 2015) addresses physical evaluation: it distributes a set of everyday objects with high-resolution RGB-D scans and models, together with a template for task protocols, so that different laboratories can test on the same objects. Ravens (Zeng et al., 2021), built on a physics simulator, places a robot with a suction gripper over a tabletop and supplies a scripted oracle that generates demonstrations, and DeformableRavens (Seita et al., 2021) extends it to deformables.

Evaluation Metrics

Pose errors. For the model points $\mathcal{M}$, ground-truth pose $(R, \mathbf{t})$ and estimate $(\hat{R}, \hat{\mathbf{t}})$, the average distance ADD, which PoseCNN adopts from Hinterstoisser et al.’s 2012 paper, is

$$\text{ADD} = \frac{1}{m} \sum_{\mathbf{x} \in \mathcal{M}} \big\lVert (R\mathbf{x} + \mathbf{t}) - (\hat{R}\mathbf{x} + \hat{\mathbf{t}}) \big\rVert ,$$

and for symmetric objects the closest-point version is

$$\text{ADD-S} = \frac{1}{m} \sum_{\mathbf{x}_1 \in \mathcal{M}} \min_{\mathbf{x}_2 \in \mathcal{M}} \big\lVert (R\mathbf{x}_1 + \mathbf{t}) - (\hat{R}\mathbf{x}_2 + \hat{\mathbf{t}}) \big\rVert$$

(Xiang et al., 2018). The outer average runs over the model in the true pose, as in BOP’s description of the closest-point error; DenseFusion’s text describes the opposite direction, from the model in the estimated pose to its closest neighbor in the true pose (Wang et al., 2019). By our reasoning the two directions can give different values for a wrong pose; the example follows PoseCNN’s equation. A pose is counted correct when the error is below 10% of the object’s diameter $d$ (Xiang et al., 2018), the largest distance between two model vertices (Hodaň et al., 2018). Xiang et al. also vary the threshold and report the area under the accuracy–threshold curve, with a maximum threshold of 10 cm on YCB-Video; DenseFusion adds the fraction of poses with ADD-S below 2 cm, which its authors take as the tolerance of most robot grippers (Wang et al., 2019). BOP calls the closest-point error ADI and notes that its many-to-one matching can make it unintuitively low for wrong poses (Hodaň et al., 2018), which the example above reproduces.

Visible Surface Discrepancy. BOP’s 2018 error renders the model in the estimated and true poses, compares both renderings with the test image’s depth to find where the object is visible, and counts the fraction of those pixels where the two renderings disagree by more than a tolerance $\tau$:

$$e_{\text{VSD}} = \operatorname*{avg}_{p \in \hat{V} \cup \bar{V}} \begin{cases} 0 & \text{if } p \in \hat{V} \cap \bar{V} \,\wedge\, |\hat{S}(p) - \bar{S}(p)| < \tau \\ 1 & \text{otherwise,} \end{cases}$$

where $\hat{S}, \bar{S}$ are the rendered distance maps and $\hat{V}, \bar{V}$ the visibility masks (Hodaň et al., 2018). Because only the visible surface is compared, poses that cannot be told apart in the image count as equivalent. A pose is correct when $e_{\text{VSD}} < \theta$; the authors argue that manipulation needs both $\tau$ and $\theta$ low and used $\tau = 20$ mm and $\theta = 0.3$ by default, while augmented reality can relax $\tau$. Recall is the fraction of test targets with a correct pose, averaged per dataset so that large datasets do not dominate.

MSSD and MSPD. From 2019, BOP added two symmetry-aware maximum distances (Hodaň et al., 2020):

$$e_{\text{MSSD}} = \min_{S \in \mathcal{S}_M} \max_{\mathbf{x} \in V_M} \big\lVert \hat{P}\mathbf{x} - \bar{P}S\mathbf{x} \big\rVert_2, \qquad e_{\text{MSPD}} = \min_{S \in \mathcal{S}_M} \max_{\mathbf{x} \in V_M} \big\lVert \operatorname{proj}(\hat{P}\mathbf{x}) - \operatorname{proj}(\bar{P}S\mathbf{x}) \big\rVert_2 ,$$

where $\mathcal{S}_M$ holds the object’s global symmetry transformations and $V_M$ its vertices. The organizers argue that the maximum surface deviation strongly indicates the chance of a successful grasp, which makes MSSD relevant for manipulation, and that the maximum depends less than the average on the model’s geometry and sampling; MSPD, measured in pixels, ignores errors along the optical axis and suits augmented reality and color-only methods. The symmetries are found from the model, as transformations that move its vertices by less than $\max(15 \text{ mm}, 0.1d)$ in Hausdorff distance, minus those that texture resolves. Each error gives an average recall over a range of thresholds (for MSSD, 5% to 50% of $d$), and a dataset’s score is $\text{AR} = (\text{AR}_{\text{VSD}} + \text{AR}_{\text{MSSD}} + \text{AR}_{\text{MSPD}})/3$.

Task success. The definition of success is part of the result. Diffusion Policy’s simulated results average the last ten checkpoints over three training seeds and 50 initial conditions, and also report the best checkpoint to match earlier papers’ practice (Chi et al., 2023); Transporter Networks report the checkpoint with the best validation performance, averaged over 100 unseen test runs, with partial credit for multi-step tasks (Zeng et al., 2021). As in grasping, a physical result is interpretable only with its protocol: trials, objects, positions, distractors and success criterion. Levine et al.’s (2016) separation into training positions, new positions and visual distractors shows how much the condition matters: the same bottle-cap policy succeeded in 88.9%, 83.3% and 62.5% of trials. With 20 trials, a single failure moves the rate by five points, so small differences between methods may not be meaningful; the statistical significance article discusses how to judge them.

Applications

• Industrial bin picking and machine tending. Known parts with CAD models suit model-based pose estimation, the setting of BOP’s task (Hodaň et al., 2018). Transporter Networks’ authors describe classic pick-and-place with pose estimation of known objects and scripted motions as robust in structured settings such as manufacturing but hard to deploy in unstructured ones (Zeng et al., 2021).
• Warehouse order fulfillment, where products are many and change often, as in Zeng et al.’s (2022) model-free system.
• Kitting, packing and palletizing, which the Ravens authors mark as tasks common in industry (Zeng et al., 2021).
• Household and service robots: category-level placing of shoes and mugs (Manuelli et al., 2022), and cloth and bags (Seita et al., 2021).

Open Problems

• Occlusion, varying lighting, and object symmetries and similarities, which the first BOP evaluation identified as open problems (Hodaň et al., 2018).
• From pose accuracy to task success. Pose benchmarks score poses, not grasps or insertions; MSSD is motivated by grasping (Hodaň et al., 2020), but how a pose error becomes a failure depends on the gripper and the task.
• Generalization of policies beyond the backgrounds, objects and robots of their demonstrations (Levine et al., 2016).
• Comparable physical evaluation. As in grasping, real-robot results come from different objects, robots and protocols; shared object sets such as YCB (Calli et al., 2015) fix only the objects.
• Deformable and contact-rich tasks, which lack compact state representations and depend on sensing beyond vision (Seita et al., 2021; Lee et al., 2019).