Robotic Grasping

Robotic grasping, as a perception task, finds where and how a robot’s gripper should take hold of an object so that the object can be lifted and moved. Given a view of a scene, usually a depth map or a point cloud, the output is one or more gripper poses with a score for how likely each is to hold. The problem has two traditions. Analytic grasping asks whether contact forces can resist any disturbance, given the object’s shape and friction. Data-driven grasping learns to rank candidate grasps from human labels, physical trials or simulated outcomes; since the mid-2010s its methods have mostly been networks that read grasps from depth images and point clouds. Reported success rates range widely, and a rate means little without the objects, clutter and trial protocol behind it.

Problem Definition

Bohg et al. (2014) define grasp synthesis as finding, for a given object, a grasp configuration that satisfies the criteria relevant for the task. In the data-driven literature they survey, a grasp is commonly parameterized by the point on the object with which the tool center point is aligned, the approach vector, the wrist orientation and an initial finger configuration. For a parallel-jaw gripper the finger configuration reduces to the opening width, and the grasp is a rigid pose in $SE(3)$ plus a width.

Bohg et al. organize methods by what the robot knows about the object beforehand:

• Known objects have a stored model with precomputed grasps. Perception reduces to recognizing the object and estimating its pose, for example by registering the model to the point cloud with iterative closest point, and transforming the stored grasps into the scene.
• Familiar objects are similar to objects seen before, so grasps are transferred by similarity of shape or category.
• Unknown objects have no model. Grasps are inferred directly from local features of the observed surface.

Most of what follows concerns unknown objects. Ten Pas et al. (2017) describe grasp detection for them as treating grasp perception like object detection: the input is a noisy, partially occluded RGB-D image or point cloud, and the output is the pose of viable grasps, without a CAD model of the object.

Input and Output

The input is usually a single depth image from an RGB-D camera above or beside the workspace, sometimes fused from several viewpoints into a point cloud; color is optional and several methods use depth alone. The camera’s intrinsic parameters lift pixels to 3D, and the camera-to-robot transform from camera calibration carries the grasp into the robot’s frame. The Industrial Machine Vision article covers that robot-guidance geometry and hand–eye calibration.

The output depends on the gripper, which is chosen by principle:

• Parallel-jaw grippers close two plates or fingers along one axis, so a grasp is a pose and a width. The detection methods below all target them.
• Suction grippers hold with a single contact by forming a vacuum seal. Mahler et al. (2018) note that vacuum end effectors are widely used in industry and in warehouse order fulfillment, partly because they can reach into narrow spaces; a suction grasp is a contact point and an approach axis, and its quality depends on whether a seal forms against the local surface.
• Multi-fingered hands can hold objects in many more ways, and the analytic tradition was developed largely for them (Bohg et al., 2014). A grasp then includes the finger joint configuration as well as the hand pose.

For parallel jaws, the grasp representations in use are:

• Oriented rectangles in the image. A grasp is $g = (x, y, \theta, h, w)$: the rectangle’s center, its orientation relative to the image’s horizontal axis, and its height and width, with two edges standing for the gripper plates (Redmon & Angelova, 2015). Lenz et al. (2015) attribute the rectangle representation to Jiang et al. (2011); Redmon and Angelova describe the five-parameter form as a simplification of Jiang et al.’s seven-dimensional one. The approach direction and gripper depth are not part of the rectangle; Lenz et al. derive the approach vector from the surface normal at the rectangle’s center.
• Planar grasps with depth. Dex-Net 2.0 specifies a grasp as the planar position, angle and depth of the gripper relative to the camera, $u = (i, j, \varphi, z)$ (Mahler et al., 2017). GG-CNN uses a grasp center pixel, an angle and a width, with a quality score, for grasps executed perpendicular to the table (Morrison et al., 2018).
• 6-DoF grasp poses. A full position and orientation of the gripper in 3D, plus its width, so that the robot can grasp from the side or at an angle (ten Pas et al., 2017; Sundermeyer et al., 2021). AnyGrasp calls the pose with width a 7-DoF grasp (Fang et al., 2023).

Variants

• Singulated objects or clutter. One object on a table, or many objects touching and occluding each other, where a grasp must also avoid collisions with neighbors.
• Planar (top-down) or full 6-DoF grasping. Planar grasps suit a camera and gripper looking down on a table; 6-DoF grasps are needed for objects reachable only from the side or in bins.
• Open-loop or closed-loop. A grasp is planned once from one image and executed blindly, or re-estimated continuously while the gripper moves. Morrison et al. (2018) designed GG-CNN for closed-loop control at up to 50 Hz, and Levine, Pastor, Krizhevsky and Quillen (2016) learned to servo the gripper continuously from monocular images, a learned relative of visual servoing.
• Dynamic grasping. Grasping objects that move; AnyGrasp tracks grasps across frames and demonstrates catching robot fish swimming in a tank (Fang et al., 2023).
• Task-oriented grasping. Choosing a grasp that suits what will be done with the object afterwards, such as holding a tool by its handle.

Challenges

• Partial and noisy views. The back of an object is unseen, thin parts may be missing from the depth image, and depth is noisy at edges. Missing sensor data on thin parts and collisions misjudged as safe were the most common failures of Dex-Net 2.0’s planner (Mahler et al., 2017).
• Unknown physics. Friction coefficients, mass and its distribution are usually unknown (Bohg et al., 2014).
• Closure is not stability. Bohg et al. note that force closure is often equated with stability, although closure only states that an equilibrium exists; stability concerns how the grasp behaves when perturbed from it.
• Calibration and control errors. A grasp correct in the image fails if the camera-to-robot transform or the arm’s positioning is off. In ten Pas et al.’s clutter experiments, a quarter of the failures (5 of 20) came from point-cloud registration errors or inaccurate kinematic calibration.
• Many valid answers. An object can usually be grasped in many ways, so labels are never exhaustive (Redmon & Angelova, 2015).
• Speed. Sampling and scoring thousands of candidates takes time: Lenz et al. (2015) report 13.5 s per image for their two-stage MATLAB implementation, and ten Pas et al. (2017) report about 0.3 to 6.2 s to classify 1,000 candidates, depending on the representation and cloud size.

Approaches

Analytic grasping

Analytic methods model the contacts between gripper and object and ask what forces they can apply. The usual contact model is Coulomb friction: at a contact with normal force $f_n$, the tangential force $f_t$ satisfies $f_t \le \mu f_n$. The forces a contact can transmit form a friction cone around the inward surface normal with half-angle $\alpha = \tan^{-1}\mu$; Lynch and Park (2017, §12.2) give 0.1 to 1 as the typical range of $\mu$, and approximate the circular cone in 3D by an inscribed pyramid, which underestimates the available friction and so errs on the safe side for grasping.

A contact force $\mathbf{f}_i$ at point $\mathbf{r}_i$ acts on the object as a wrench, the force together with the torque it produces about a reference point:

$$\mathbf{w}_i = \begin{pmatrix} \mathbf{f}_i \\ \mathbf{r}_i \times \mathbf{f}_i \end{pmatrix} \in \mathbb{R}^6 .$$

Each edge of a contact’s friction cone gives a primitive wrench. A grasp is in force closure when nonnegative combinations of the primitive wrenches of all contacts span the whole wrench space, so that any external wrench can be balanced; Ferrari and Canny (1992) state the equivalent geometric condition: the origin of the wrench space lies inside the convex hull of the primitive wrenches. Lynch and Park (2017, §12.2.3) give a linear-programming test that is exact for planar problems. They also state the counts: in the plane, two frictional contacts can suffice, and they are in force closure if the line between them lies inside both friction cones, a result they attribute to Nguyen (1988); in 3D, two point contacts cannot resist a torque about the line joining them, and three non-collinear frictional contacts are the minimum. Lynch and Park add that force closure only says the cones can generate any wrench: whether the object stays put also depends on the internal forces the fingers can actually apply.

For two-fingered grippers, the line-of-sight condition gives antipodal grasps: contact pairs whose outward normals point away from each other along the line joining them, within the friction cone. Ten Pas et al. (2017) use the stricter frictionless antipodal condition, under which the two contact forces balance each other with no friction at all, because such a grasp is in force closure for any nonzero friction, assuming soft contacts, and stays in force closure under small perturbations of the contacts.

Force closure is binary; grasp quality metrics rank grasps that satisfy it. Ferrari and Canny (1992) measure quality by the largest wrench a grasp can resist in its weakest direction, for bounded finger forces. With hard-finger contacts and friction pyramids of $m$ edges, let $\mathbf{w}_{i,1}, \dots, \mathbf{w}_{i,m}$ be the primitive wrenches of contact $i$ for unit normal force. If the sum of the finger forces is bounded by 1, the set of resistible wrenches is

$$\mathcal{W}_{L_1} = \operatorname{ConvexHull}\Big(\bigcup_{i=1}^{n} \{\mathbf{w}_{i,1}, \dots, \mathbf{w}_{i,m}\}\Big),$$

and if each finger force is bounded by 1 independently, it is the Minkowski sum of the contacts’ hulls, $\mathcal{W}_{L_\infty} = \operatorname{ConvexHull}\big(\bigoplus_{i} \{\mathbf{w}_{i,1}, \dots, \mathbf{w}_{i,m}\}\big)$. The quality $Q$ is the distance from the origin to the nearest facet of the hull, the radius of the largest ball around the origin that fits inside it. The $L_1$ criterion limits the total finger force, which Ferrari and Canny relate to the power needed to drive the gripper, and the $L_\infty$ criterion the largest single finger force; $Q_{L_\infty} \ge Q_{L_1}$, but a grasp optimal under one is not necessarily optimal under the other. Torque and force have different units, so the wrench norm needs a scale $\lambda$ between them, whose choice they call somewhat arbitrary. Their paper says one of the two criteria had been proposed before and the other is new, and its examples are planar objects grasped by two- and three-jaw grippers. Bohg et al. (2014) describe this $\epsilon$-metric as widely used for ranking grasps in simulation.

Analytic methods give guarantees, but only under their assumptions: known shape, rigid bodies, Coulomb friction, simplified contacts and exact placement (Bohg et al., 2014). Bohg et al. summarize studies that tested grasps rated stable by classical metrics on real robots and found them to be poor predictors of real-world success; one study found the $\epsilon$-metric particularly poor under object pose error for large objects. Up to about 2000, they write, the field was dominated by analytic approaches.

Data-driven grasping

Data-driven methods sample candidate grasps and rank them with a score learned from experience (Bohg et al., 2014). Bohg et al. date the rise of the approach to the GraspIt! simulator in 2004, where candidates were still ranked by analytic metrics, and point to Saxena et al.’s logistic regressor, trained on synthetic labeled images to predict grasping points in a monocular image, as the work that set off learning from visual features. Low-cost depth sensors from 2010 on then made real 3D data common in robot labs. Methods since differ in where their labels come from:

• Human labels. The Cornell grasping dataset annotates RGB-D images with positive and negative grasp rectangles drawn by people (Lenz et al., 2015). Depierre et al. (2018) note that human labels can be biased toward grasps easy for a human hand.
• Physical trials. Robots label their own attempts: Pinto and Gupta (2016) collected 50,000 attempts over 700 robot hours, and Levine et al. (2016) over 800,000 attempts in two months with 6 to 14 robot arms. Bohg et al. (2014) note that such data has no transfer gap to the real robot but is extremely time-consuming to collect.
• Simulated analytic labels. Grasps on 3D models are scored with an analytic or simulated criterion and rendered into synthetic depth images, as in Dex-Net 2.0 and Jacquard, or transferred to real captured scenes, as in GraspNet-1Billion. Labels are cheap and dense; the cost is the gap between rendered and real sensor data and between the criterion and real outcomes.

Example: antipodal grasp sampling on a rendered depth image

The example renders a top-down depth image of three objects of known shape on a table: a box, an upright cylinder and a triangular prism. It estimates the outward normals of their vertical walls from the depth gradient at the object edges, samples pairs of edge points, keeps the pairs that fall inside both friction cones ($\mu = 0.5$), checks that the closing segment lies on one object and that the fingers have free space, and ranks the survivors by the worse of the two cone angles, with a simple form of non-maximum suppression to drop near-duplicates. This follows the idea of Dex-Net 2.0’s candidate sampler, antipodal pairs found with normals from depth-image gradients (Mahler et al., 2017), in a simplified form: an orthographic camera, 2D contacts on the object outline and point-like fingertips. Every accepted pair is then checked against the true geometry, where the closing line actually meets the shape and what the true normals are there, and the ten best grasps also against Cornell-style ground-truth rectangles drawn by hand.

import sys
import numpy as np
from scipy import ndimage

SEED = int(sys.argv[1]) if len(sys.argv) > 1 else 0
rng = np.random.default_rng(SEED)
PX = 2.0                                   # mm per pixel (orthographic top-down camera)
H, W, CAM = 150, 200, 700.0                # image size; camera 700 mm above the table
MU = 0.5                                   # friction coefficient
ALPHA = np.degrees(np.arctan(MU))          # friction-cone half-angle, 26.6 degrees

def rot(t):
    t = np.radians(t)
    return np.array([[np.cos(t), -np.sin(t)], [np.sin(t), np.cos(t)]])

def polygon(center, verts, angle):
    return np.asarray(center) + np.asarray(verts, float) @ rot(angle).T

tri = [[25 * np.cos(np.radians(a)), 25 * np.sin(np.radians(a))] for a in (90, 210, 330)]
tri = np.array(tri) * (50 / (25 * np.sqrt(3)))           # equilateral, side 50 mm
objects = [  # name, kind, geometry, height (mm)
    ("box", "poly", polygon((100, 100), [[-30, -15], [30, -15], [30, 15], [-30, 15]], 30), 40),
    ("cylinder", "circle", ((180, 95), 20.0), 60),
    ("triangle", "poly", polygon((300, 210), tri, 10), 30),
]

def inside(obj, x, y):
    _, kind, g, _ = obj
    if kind == "circle":
        (cx, cy), r = g
        return (x - cx) ** 2 + (y - cy) ** 2 <= r ** 2
    ok = np.ones_like(x, bool)
    for a, b in zip(g, np.roll(g, -1, axis=0)):          # convex, counter-clockwise
        ok &= (b[0] - a[0]) * (y - a[1]) - (b[1] - a[1]) * (x - a[0]) >= 0
    return ok

# Render the depth image with 4x4 supersampling (mixed pixels at edges), plus noise.
s = (np.arange(4) + 0.5) / 4
yy, xx = np.meshgrid((np.arange(H)[:, None] + s).ravel() * PX,
                     (np.arange(W)[:, None] + s).ravel() * PX, indexing="ij")
height = np.zeros_like(xx)
for obj in objects:
    height = np.where(inside(obj, xx, yy), np.maximum(height, obj[3]), height)
depth = (CAM - height).reshape(H, 4, W, 4).mean(axis=(1, 3))
depth += rng.normal(0, 1.0, depth.shape)                  # 1 mm sensor noise

# Surface normals of the vertical walls from the depth gradient at object edges.
smooth = ndimage.gaussian_filter(depth, 1.5)
gy, gx = ndimage.sobel(smooth, 0), ndimage.sobel(smooth, 1)
mag = np.hypot(gx, gy)
on_obj = depth < CAM - 10
edge = on_obj & ndimage.binary_dilation(~on_obj) & (mag > 20)
ey, ex = np.nonzero(edge)
pts = np.c_[ex + 0.5, ey + 0.5] * PX                      # contact points (mm)
nrm = np.c_[gx[edge], gy[edge]] / mag[edge][:, None]      # outward: depth increases
print(f"edge pixels: {len(pts)}, friction cone half-angle: {ALPHA:.1f} deg")

def depth_at(p):
    c, r = np.floor(np.asarray(p) / PX).astype(int).T
    ok = (r >= 0) & (r < H) & (c >= 0) & (c < W)
    return np.where(ok, depth[np.clip(r, 0, H - 1), np.clip(c, 0, W - 1)], CAM)

# Sample point pairs, keep those inside both friction cones, and check the jaws.
i, j = rng.integers(0, len(pts), (2, 20000))
d = pts[j] - pts[i]
width = np.linalg.norm(d, axis=1)
keep = (width > 10) & (width < 80)                        # gripper stroke 10-80 mm
i, j, d, width = i[keep], j[keep], d[keep] / width[keep, None], width[keep]
ang1 = np.degrees(np.arccos(np.clip(-(nrm[i] * d).sum(1), -1, 1)))
ang2 = np.degrees(np.arccos(np.clip((nrm[j] * d).sum(1), -1, 1)))
cone = np.maximum(ang1, ang2)
t = np.linspace(0.1, 0.9, 9)[:, None, None]
span = pts[i] + t * (pts[j] - pts[i])                     # closing segment on one object
solid = (depth_at(span.reshape(-1, 2)).reshape(9, -1) < CAM - 10).all(0)
free = (depth_at(pts[i] - 8 * d) > CAM - 3) & (depth_at(pts[j] + 8 * d) > CAM - 3)
good = (cone <= ALPHA) & solid & free
print(f"sampled pairs: {len(cone)}, in both cones: {(cone <= ALPHA).sum()}, "
      f"also collision-free on one object: {good.sum()}")

# Rank by the worse of the two cone angles; suppress near-duplicate grasps.
cand = np.nonzero(good)[0][np.argsort(cone[good], kind="stable")]
centers = (pts[i] + pts[j]) / 2
theta = np.degrees(np.arctan2(d[:, 1], d[:, 0])) % 180
chosen = []
for k in cand:
    if all(np.linalg.norm(centers[k] - centers[m]) > 10 or
           min(abs(theta[k] - theta[m]), 180 - abs(theta[k] - theta[m])) > 20
           for m in chosen):
        chosen.append(k)

def true_contacts(c, u):
    """Fingertip contacts where the closing line through c along u meets the true shape."""
    for name, kind, g, _ in objects:
        if not inside((name, kind, g, 0), np.array(c[0]), np.array(c[1])):
            continue
        if kind == "circle":
            (cx, cy), r = g
            w = c - np.array([cx, cy])
            b = w @ u
            s_ = np.sqrt(b * b - (w @ w - r * r))
            p1, p2 = c + (-b - s_) * u, c + (-b + s_) * u
            n1, n2 = (p1 - [cx, cy]) / r, (p2 - [cx, cy]) / r
            return name, p1, p2, n1, n2, False
        hits = []
        for a, b in zip(g, np.roll(g, -1, axis=0)):
            e = b - a
            den = u[0] * e[1] - u[1] * e[0]
            if abs(den) < 1e-12:
                continue
            q = a - c
            tt, ss = (q[0] * e[1] - q[1] * e[0]) / den, (q[0] * u[1] - q[1] * u[0]) / den
            if 0 <= ss <= 1:
                n = np.array([e[1], -e[0]]) / np.linalg.norm(e)
                corner = min(ss, 1 - ss) * np.linalg.norm(e) < 1.0
                hits.append((tt, c + tt * u, n, corner))
        hits.sort(key=lambda h: h[0])
        (_, p1, n1, c1), (_, p2, n2, c2) = hits[0], hits[-1]
        return name, p1, p2, n1, n2, c1 or c2
    return None

# Hand-labelled ground-truth rectangles (Cornell style): center, closing angle, opening, jaw.
c_box, c_tri = np.array([100.0, 100.0]), np.array([300.0, 210.0])
along = np.array([np.cos(np.radians(30)), np.sin(np.radians(30))])
gt = [(c_box + o * along, 120, 50, 20) for o in (-15, 0, 15)] + [(c_box, 30, 80, 20)]
gt += [(np.array([180.0, 95.0]), a, 60, 20) for a in (0, 45, 90, 135)]
for a in (100, 220, 340):                     # vertex-to-face (altitude) grasps
    gt.append((c_tri + 7.2 * np.array([np.cos(np.radians(a)), np.sin(np.radians(a))]),
               a % 180, 63.3, 20))

def rect_mask(c, ang, opening, jaw, gx_, gy_):
    u = np.array([np.cos(np.radians(ang)), np.sin(np.radians(ang))])
    du, dv = (gx_ - c[0]) * u[0] + (gy_ - c[1]) * u[1], -(gx_ - c[0]) * u[1] + (gy_ - c[1]) * u[0]
    return (np.abs(du) <= opening / 2) & (np.abs(dv) <= jaw / 2)

def rectangle_metric(c, ang, opening):
    """Cornell rectangle metric: angle within 30 degrees and IoU above 0.25 for some label."""
    gx_, gy_ = np.meshgrid(np.arange(c[0] - 60, c[0] + 60, 0.5), np.arange(c[1] - 60, c[1] + 60, 0.5))
    best = 0.0
    pred = rect_mask(c, ang, opening, 20, gx_, gy_)
    for gc, ga, go, gj in gt:
        dang = abs(ang - ga) % 180
        if min(dang, 180 - dang) > 30:
            continue
        lab = rect_mask(gc, ga, go, gj, gx_, gy_)
        best = max(best, (pred & lab).sum() / (pred | lab).sum())
    return best

def check(k):
    """Cone angle at the true contacts, and a verdict: yes, NO, or corner contact."""
    name, p1, p2, n1, n2, corner = true_contacts(centers[k], d[k])
    u = (p2 - p1) / np.linalg.norm(p2 - p1)
    tc = max(np.degrees(np.arccos(np.clip(-n1 @ u, -1, 1))),
             np.degrees(np.arccos(np.clip(n2 @ u, -1, 1))))
    return name, tc, "corner" if corner else ("yes" if tc <= ALPHA else "NO")

tally = {o[0]: [0, 0, 0] for o in objects}       # accepted, NO, corner
for k in np.nonzero(good)[0]:
    name, _, verdict = check(k)
    tally[name][0] += 1
    tally[name][1] += verdict == "NO"
    tally[name][2] += verdict == "corner"
for name, (n, no, corner) in tally.items():
    print(f"{name:9s} accepted {n:4d}, true check: NO {no:3d}, corner {corner:3d}")

print(" #  object    center(mm)   angle  width  est.cone  true.cone  closure  rect.IoU")
for rank, k in enumerate(chosen[:10], 1):
    name, tc, verdict = check(k)
    iou = rectangle_metric(centers[k], theta[k], width[k] + 20)
    print(f"{rank:2d}  {name:9s} ({centers[k][0]:5.1f},{centers[k][1]:5.1f}) {theta[k]:6.1f}"
          f" {width[k]:6.1f} {cone[k]:8.1f} {tc:10.1f}  {verdict:7s}  {iou:.2f}"
          f" {'pass' if iou > 0.25 else 'fail'}")

Output (NumPy 2.3.5, SciPy 1.16.3, seed 0):

edge pixels: 197, friction cone half-angle: 26.6 deg
sampled pairs: 7835, in both cones: 1065, also collision-free on one object: 1062
box       accepted  442, true check: NO   3, corner   0
cylinder  accepted  476, true check: NO   2, corner   0
triangle  accepted  144, true check: NO  93, corner  51
 #  object    center(mm)   angle  width  est.cone  true.cone  closure  rect.IoU
 1  cylinder  (180.0, 95.0)   43.0   41.0      0.0        0.0  yes      0.93 pass
 2  cylinder  (180.0, 95.0)    0.0   38.0      0.1        0.0  yes      0.97 pass
 3  triangle  (306.0,208.0)  159.8   40.5      0.1       60.2  corner   0.94 pass
 4  cylinder  (180.0, 95.0)  137.0   41.0      0.2        0.0  yes      0.93 pass
 5  cylinder  (180.0, 95.0)   98.5   40.4      0.2        0.0  yes      0.79 pass
 6  cylinder  (180.0, 95.0)   76.0   41.2      0.3        0.0  yes      0.70 pass
 7  box       (112.0,107.0)  120.3   27.8      1.0        0.3  yes      0.86 pass
 8  box       (102.0, 96.0)   31.0   58.3      1.2        1.0  yes      0.62 pass
 9  box       (100.0,100.0)  118.3   29.5      1.2        1.7  yes      0.96 pass
10  box       ( 84.0, 90.0)  118.3   29.5      1.2        1.7  yes      0.66 pass

About one sampled pair in seven lies inside both estimated cones. The collision checks remove almost none here because the three objects stand well apart; in clutter, those checks would do much more of the work. Checked against the true shapes, the accepted pairs on the box and the cylinder are nearly all sound, with 5 failures among 918. The triangle is different: all 144 of its accepted pairs fail. This follows from its geometry. Any two faces of an equilateral triangle meet at 60°, so fingertips on two faces need $\mu \ge \tan 30° \approx 0.577$, the bound Lynch and Park (2017, §12.2.3, Figure 12.22) state for two fingers on an equilateral triangle; at $\mu = 0.5$ no fingertip grasp of the triangle is in force closure. The sampler accepts such pairs because smoothing the depth image rounds each corner into a fan of normals that the true faces do not have: 51 of the pairs touch within 1 mm of a corner, and the other 93 meet the faces at true cone angles above 26.6° that the smoothed normals underestimated.

Ranking by cone angle favors the cylinder, where every diameter is exactly antipodal, so five of the ten slots go to one object at different angles; a planner that wants alternatives across objects needs a ranking that rewards diversity or robustness, not only the cone angle. The box grasps across its short side (widths near the true 30 mm) and along its length (58.3 mm against 60) are genuine face-to-face grasps.

The third grasp runs along an altitude of the triangle, from a vertex to the opposite face. Smoothing gives the vertex a normal along its bisector, which points straight at the opposite face, so the estimate looks perfectly antipodal; the true closing line meets the outline at the vertex itself, where the faces that meet there have normals 60° from it. With flat jaw plates such a vertex-to-face grasp can hold, but with fingertips it rests on a point the depth image cannot resolve. The hand-drawn rectangles include the triangle’s altitude grasps, as a person might label them, and score this one 0.94; like every grasp in the list, it passes the rectangle metric, which measures agreement with annotators rather than whether the contacts are sound.

Running the script with seeds 1 to 9 as well (python grasp.py <seed>), every pair accepted on the triangle again failed the true check, the box and cylinder had between 3 and 16 failing pairs, every top-ten list passed the rectangle metric, and each top-ten list contained between 0 and 2 grasps that fail the true check, all on the triangle.

Grasp detection in depth images

Lenz et al. (2015) applied deep learning to Cornell-style rectangles. A network scores each candidate rectangle from color, depth and surface-normal patches, searched exhaustively over positions, sizes and 15° steps of orientation; a small first network prunes the candidates and a larger second network re-ranks the top 100. Depth alone gave the best single-modality recognition, and all modalities together the best overall. On real robots, with one of 30 objects at a time placed in a 25 cm square so that at least one viable grasp was visible and reachable (five further objects, which the grippers could not hold even with hand-chosen grasps, were left out), they report success in 84 and 89 of 100 trials on two robot platforms; a martini glass failed in all three trials on both.

Redmon and Angelova (2015) replaced the sliding window with a convolutional neural network that regresses the rectangle in one pass. Pretrained on ImageNet, it takes RGB-D input by replacing the image’s blue channel with depth; their MultiGrasp variant divides the image into a grid and predicts, for each cell, a grasp and the likelihood that it is feasible. They report 88.0% (image-wise split) and 87.1% (object-wise split) rectangle-metric accuracy on Cornell, against 73.9% and 75.6% for Lenz et al. Their Table I lists 76 ms per image; the text explains that 76 ms is the GPU time for a batch of 128 images, which they report as a latency of 13 frames per second.

Dex-Net 2.0 (Mahler et al., 2017) separates proposal from scoring. Antipodal candidates are sampled in the depth image, and a Grasp Quality CNN (GQ-CNN) of about 18 million parameters scores each from a 32 × 32 depth patch centered on the grasp and rotated to its axis, together with the gripper’s depth. It is trained only on synthetic data: 6.7 million such grasp images, cut from depth images rendered of 1,500 object models in their stable poses on a table, each labeled positive if the gripper does not collide and the robust epsilon quality, the expected value of the $\epsilon$-metric under uncertainty in object pose, gripper pose and friction, exceeds a threshold.

GG-CNN (Morrison et al., 2018) drops candidates altogether. A small, fully convolutional network, in the manner of the FCN for segmentation, maps a 300 × 300 depth image to three images of the same size: grasp quality, angle and gripper width at every pixel. Because a parallel grasp is unchanged by a half-turn, the angle is predicted as $\sin 2\Phi$ and $\cos 2\Phi$, which avoids the wrap-around at $\pm\pi/2$. Training targets come from Cornell’s positive rectangles, using the middle third of each rectangle as the region where the gripper’s center may lie, with data augmentation by crops, zooms and rotations. The network has 62,420 parameters, and the authors report 19 ms for the full detection pipeline, fast enough to re-plan during the reach.

Grasp detection in point clouds

Planar grasps cannot reach objects from the side. Grasp pose detection (GPD) (ten Pas et al., 2017) samples 6-DoF hand poses around points of the cloud, keeps those whose open hand does not collide with the cloud and whose closing region contains points, encodes the points inside each closing region as a stack of images projected along three directions, and scores them with a four-layer CNN. Training labels mark a pose positive if it forms a frictionless antipodal grasp on the object’s mesh, with a 1 mm tolerance for mesh noise.

Contact-GraspNet (Sundermeyer et al., 2021) treats each observed point as a possible contact of one finger. For each point it predicts whether a grasp makes contact there, and the remaining grasp rotation and width, so that a full grasp pose follows from the contact $\mathbf{c}$, the approach vector $\mathbf{a}$, the baseline direction $\mathbf{b}$ between the fingers, the width $w$ and the constant distance $d$ from the gripper’s baseline to its base:

$$\mathbf{t}_g = \mathbf{c} + \frac{w}{2}\,\mathbf{b} + d\,\mathbf{a}, \qquad R_g = \begin{bmatrix} \mathbf{b} & \mathbf{a} \times \mathbf{b} & \mathbf{a} \end{bmatrix} .$$

Anchoring grasps to visible points reduces the learning problem to four degrees of freedom per point. The network is trained on 17 million simulated grasps and runs in 0.28 s on a full scene. Because grasps are attached to points, they can be assigned to objects by filtering their contacts with instance segmentation masks rather than by cropping the cloud.

The GraspNet-1Billion paper (Fang et al., 2020) introduced a point-cloud network that learns the approach direction and the remaining grasp parameters in separate stages, together with its benchmark (below). AnyGrasp (Fang et al., 2023) builds on that line with dense supervision from real captured scenes and analytic labels, awareness of the object’s center of mass, and correspondence of grasps across frames for tracking. It is trained on real data of 144 objects, and its authors hope their analysis of training data will encourage the field to prioritize real-world data over large simulated object sets.

Suction grasping

For suction, Dex-Net 3.0 (Mahler et al., 2018) models the cup as a quasi-static system of springs that must deform to meet the surface: a seal is feasible if the energy needed is low enough, and the grasp must then resist the object’s weight. Robust labels averaged over perturbations of pose, material and wrench train the same kind of GQ-CNN on 2.8 million synthetic point clouds. The authors note that suction planners often simply target flat surfaces near the object’s estimated centroid, and their model assumes rigid, non-porous objects.

Datasets and Benchmarks

Dataset	Year	Data	Labels	Evaluation
Cornell grasping dataset	2011; extended by Lenz et al.	Real RGB-D images of single objects	Hand-drawn positive and negative rectangles	Rectangle metric, image-wise and object-wise splits
Dex-Net 2.0	2017	6.7 million synthetic depth patches, 1,500 models	Robust analytic quality, thresholded	Synthetic classification; physical trials
Jacquard	2018	54k synthetic RGB-D images, 11k objects	1.1 million rectangles from simulated grasp trials	Rectangle metric; simulated grasp trial
GraspNet-1Billion	2020	97,280 real RGB-D images, 190 cluttered scenes, 88 objects	Over 1.1 billion 6-DoF grasps from analytic force closure	AP over Precision@k at several $\mu$

The Cornell grasping dataset was extended for Lenz et al. (2015) from the data of Jiang et al. (2011); Lenz et al. describe 1,035 images of 280 objects with roughly equal numbers of graspable and non-graspable rectangles. Later papers give other counts: Jacquard’s text describes 885 images of 240 objects with 8,019 hand-labeled rectangles, while its comparison table lists 1,035 images, and GG-CNN’s paper gives 885 images with 5,110 positive and 2,909 negative grasps, which also sum to 8,019. Results are reported under five-fold cross-validation, split by image or by object; Redmon and Angelova found the two splits give comparable results and suggest that the objects are similar to one another. Fang et al. (2020) write that reported accuracies on Cornell had passed 99% by 2020.

Jacquard (Depierre et al., 2018) was generated from ShapeNet models: objects are dropped on a plane in simulation, rendered with realistic depth from simulated stereo, and grasps are labeled positive when a simulated gripper lifts the object. The authors offer a web interface that executes submitted grasps in simulation, the simulated grasp trial (SGT), so that grasps not in the annotation can still be scored.

GraspNet-1Billion (Fang et al., 2020) captures each of 190 cluttered scenes from 256 viewpoints with two RGB-D cameras, 512 images per scene. Grasps are generated on each object’s mesh, scored by the lowest friction coefficient at which they remain antipodal, and transferred to the scenes with annotated 6D object poses, with collision checks. The 90 test scenes are divided into 30 with seen objects, 30 with unseen but similar objects and 30 with novel objects. To check its labels, the authors executed randomly chosen annotated grasps of three score levels on 10 objects with a real arm, 100 per setting, and report an average success rate of 0.96 for the highest-scored grasps.

Physical benchmarks fix objects and procedures rather than images. Morrison et al. (2018) note that grasp success rates are hard to compare, because object sets, robots, grippers and cameras all differ and many papers report results on their own “household” objects; they propose 20 reproducible items, eight 3D-printed adversarial objects from Dex-Net and twelve from the ACRV Picking Benchmark and YCB object sets. Ten Pas et al. (2017) define a dense-clutter task: ten objects drawn at random from a set of 27 are shaken in a box, poured into a tray, and grasped one at a time until the tray is empty or fixed stopping rules apply.

Evaluation Metrics

The rectangle metric. A predicted rectangle is correct if its orientation is within 30° of a ground-truth rectangle and their intersection over union (the Jaccard index) exceeds 25% (Redmon & Angelova, 2015; Lenz et al., 2015, say “at least 25%”). Only the top-ranked prediction per image is scored, and the accuracy is the fraction of images where it matches some label. The threshold is lower than the 50% common in detection because labels are not exhaustive: a correctly oriented rectangle that overlaps a label by a quarter is often a good grasp (Redmon & Angelova, 2015). Lenz et al. contrast it with the point metric, which accepts a prediction whose center lies near a labeled center; chance scores 35.9% on the point metric against 6.7% on the rectangle metric, so the point metric can overstate performance. Fang et al. (2020) add that the rectangle metric applies only to rectangle representations and that its tolerance may overestimate performance.

Analytic online evaluation. GraspNet-1Billion scores any predicted 6-DoF grasp without matching it to labels: the grasp is assigned to the object whose points lie inside the gripper and labeled by the force-closure test at friction $\mu$. After a pose non-maximum suppression, Precision@k measures the fraction of correct grasps among the top $k$; $\text{AP}_\mu$ averages Precision@k over $k = 1, \dots, 50$, and AP averages $\text{AP}_\mu$ over $\mu = 0.2, 0.4, \dots, 1.0$ (Fang et al., 2020). Despite the name, this is not the average precision of a precision–recall curve.

Physical success rate. The fraction of grasp attempts on a real robot that hold the object. Its definition varies: Dex-Net 2.0 counts a grasp as successful only if the object stays in the gripper after lifting, transport and shaking (Mahler et al., 2017); Contact-GraspNet allows at most two attempts per object and reports both overall success and success at the first attempt (Sundermeyer et al., 2021). Related measures include precision, the success rate of grasps the planner itself rates as robust, which matters when the robot may decline to grasp (Mahler et al., 2017); the fraction of objects cleared from a scene; and picks per hour.

Success rates depend on the conditions at least as much as on the method. In ten Pas et al.’s (2017) dense-clutter task, the same detector succeeded in 93% of 288 attempts with a point cloud built by moving a wrist camera over the pile, 84% of 138 attempts with two fixed body-mounted sensors, 75% of 155 attempts when grasps were chosen at random among the detections instead of by their selection heuristics, and 53% of 142 attempts without the learned classifier. With single objects on a table, Dex-Net 2.0’s GQ-CNN planner (the GQ-L-Adv model in their Table III) succeeded in 93% of 80 trials on its eight known objects, where registering the point cloud to stored models reached 95%, and in 80% of 50 trials on ten novel objects, a rate the authors give with a 95% confidence interval of ±11 points (Mahler et al., 2017). AnyGrasp reports 93.3% of attempts successful when clearing bins of over 300 unseen objects, which its authors describe as on par with two human volunteers (93.9%) using a two-finger jaw open-loop under controlled conditions; the robot’s grasps were restricted to within 25° of vertical, and objects that scattered outside the workspace were pushed back by hand (Fang et al., 2023). A success rate is interpretable only with its object set, its degree of clutter, the number of trials and attempts allowed, the gripper and sensor, and who resets the scene.

Applications

Grasp perception comes before most manipulation: picking parts and products from bins and conveyors, order fulfillment in warehouses, and household and service robots that handle unknown objects. Dex-Net 2.0’s authors demonstrated order fulfillment, separating objects from a pile by pushing and then grasping and transporting the target objects (Mahler et al., 2017). In industry, robot guidance for structured picking often relies on known part models and calibrated geometry; grasping unknown objects in clutter is where learned grasp detection has been aimed. What follows the grasp, such as estimating object pose for placement, pick-and-place, assembly and handling deformable objects, belongs to vision-based robotic manipulation.

Ten Pas et al. (2017) wrote that the success rates of 75% to 95% then reported for novel objects in isolation or light clutter were “too low for practical grasping applications”, and that light clutter does not reflect real-world grasping.

Open Problems

• Comparable evaluation. Physical protocols differ from paper to paper, and offline metrics rest either on incomplete labels or on an analytic criterion that may not match real outcomes.
• From proxies to real success. Learned scores are trained on human rectangles, simulated physics or analytic labels, none of which is the real outcome.
• Difficult geometry. Beyond thin parts lost by the sensor, Contact-GraspNet’s authors report less confident predictions for thick objects near the gripper’s maximum width and for small objects (Sundermeyer et al., 2021).
• Simulation against real data. Dex-Net 2.0 and Jacquard are rendered from 3D models, while AnyGrasp’s authors argue for real-world training data.
• Hands and tasks. Bohg et al. (2014) found that very few of the approaches they surveyed explicitly address the task or the kinematics of the robot’s hand, and the detection methods above are built for parallel jaws.