Affordance Detection

Affordance detection finds the places on an object where an action is possible: the handle that can be grasped, the blade that can cut, the bowl’s inside that can hold liquid. Where semantic segmentation labels each pixel by what it is, affordance segmentation labels it by what an agent could do with it, so a mug’s handle and a hammer’s handle share the label grasp although they belong to different objects, and two parts that look alike can carry different labels. The word comes from the psychologist James J. Gibson, and robotics papers adopted it because a robot that must use an unfamiliar tool needs to know which part to hold and which part does the work, more than the tool’s name. The task is studied in two forms. Supervised affordance segmentation predicts per-pixel labels learned from hand-annotated part masks, on benchmarks such as the UMD part affordance dataset and IIT-AFF. Affordance grounding predicts a heatmap of where an interaction would happen, learned with weak supervision from videos or photos of people using objects, on benchmarks such as OPRA, EPIC-Kitchens annotations and AGD20K. Each form has its own metrics, and their values cannot be compared across forms.

Problem Definition

Gibson introduced affordances in his ecological theory of perception; vision papers cite either his 1979 book, The Ecological Approach to Visual Perception (Nguyen et al., 2017; Nagarajan et al., 2019), or his 1977 “The Theory of Affordances” (Myers et al., 2015; Luo et al., 2022). Neither was read for this article, and the definitions below are the vision papers’ paraphrases. Myers et al. describe affordances as the latent “action possibilities” available to an agent given its capabilities and its environment: for an adult, stairs afford climbing, an apple affords eating and a knife affords cutting another object. Because the last kind matters most for a robot using tools, they speak of effective affordances, those of an object that an agent can use to produce an effect on another object, and they define each surface’s effective affordance by the way it touches the object it affects: the inside of a mug contains because it touches the liquid, its handle can be grasped. Nagarajan et al. summarize the idea as objects having action possibilities, a chair affording sitting and a broom cleaning.

The concept says nothing about pixels, so every benchmark fixes its own operational definition, and these differ in what is labelled and by whom:

• One label per pixel, chosen by annotators from part geometry. The UMD dataset labels each pixel of a tool with its most likely affordance, and additionally ranks all seven affordances at every pixel, with ties allowed, because a part can be used in more than one way (Myers et al., 2015; dataset README). IIT-AFF assumes one affordance per pixel and states the task as mapping each pixel to its most probable label (Nguyen et al., 2017).
• Labels inside detected objects. IIT-AFF and AffordanceNet couple each object’s box and class with the affordance labels of the pixels inside it, so that a robot knows both which object it is looking at and which part to use (Do et al., 2018).
• A heatmap of where people interact. OPRA’s annotators watch a video clip of a person using a product and then mark ten points on a separate image of the product where the interaction happened; the heatmap is a mixture of Gaussians at those points and is treated as a probability distribution over the image (Fang et al., 2018). AGD20K’s annotators place dense points on regions of frequent interaction and sparse points elsewhere, and heatmaps are made from them in the same way (Luo et al., 2022).

Nagarajan et al. (2019) argue that the first kind of label captures annotators’ expectations of what is important but is detached from real interactions, which may only partly overlap a part’s region; they call such labels “ungrounded” and the heatmaps from observed interactions “grounded”. This is a difference in what the ground truth means, not only in its format.

Input and Output

The input is an image of one or more objects: RGB-D in UMD, where Myers et al. found geometric features from depth essential, and RGB in most later work; AffordanceNet uses only the RGB images of UMD (Do et al., 2018). For grounding, the training input also includes videos or third-person (exocentric) photos of people interacting with objects, and the test input is an image of an object at rest, often with the affordance or action to ground: AGD20K’s test procedure takes an egocentric object image and an affordance label and returns the region for that affordance (Luo et al., 2022), and interaction hotspots return one map per action (Nagarajan et al., 2019).

The output is one of:

• a label map $L(x, y) \in \{0, 1, \dots, K\}$, with 0 for background and $K$ affordance classes (7 in UMD, 9 in IIT-AFF);
• one score map $D_k(x, y) \in [0, 1]$ per affordance, which UMD’s methods produce as probabilities (Myers et al., 2015);
• object boxes with class labels, each with a label map over its pixels (IIT-AFF, AffordanceNet);
• a heatmap $Y_a(x, y)$ for an affordance or action $a$, compared with a ground-truth distribution.

Variants

• Part affordance segmentation of tools and household objects, the UMD and IIT-AFF setting, covered in the approaches below.
• Generalization to new categories. UMD is built so that tools from different categories share affordances (cut appears on knives, saws and shears), and its novel_split.txt holds out whole categories for testing (dataset README). Interaction hotspots and AGD20K report results on object classes unseen in training.
• Grasp affordances. The grasp label marks where a hand or gripper can hold an object, but it is a region, not a gripper pose: robotic grasping covers how grasp poses are detected and scored. Several affordance papers turn the grasp region into a grasp, as described under Applications.
• Action-outcome maps for manipulation. In robot learning, an affordance map can also predict, for every location, the outcome of an action taken there, as in the pixel-wise picking maps and Transporter Networks discussed in vision-based robotic manipulation. Those maps are learned from the robot’s own trials or demonstrations rather than from part labels.
• Scene affordances. Demo2Vec’s authors mention work that predicts human-scale labels such as walkable and sittable over whole scenes (Roy and Todorovic, 2016, as described by Fang et al., 2018).
• Language and 3D. SayCan’s title speaks of grounding language in robotic affordances: as embodied AI describes it, a learned value function estimates whether each of a robot’s skills will succeed from the current state, and this estimate grounds a language model’s plan. The AGD20K repository’s README lists, as other relevant works, papers on phrase-based affordance detection and on grounding 3D object affordances from 2D interactions in images.

Challenges

• The same affordance looks different, and different affordances look alike. Do et al. (2018) note that parts with different appearances may share a label, because labels rest on abstract concepts of human action; Myers et al. (2015) name the same difficulty, tools of different categories with different shapes having parts with the same affordance. The example below builds the converse case, a blade and a handle that look identical up close.
• Several affordances per part. A cup’s outside can be wrap-grasped and its inside contains (Myers et al., 2015), and a single part can serve more than one use; UMD’s ranked labels exist for this reason, and AGD20K’s authors describe a many-to-many relation between affordance and object categories in their data (Luo et al., 2022).
• Annotation cost and meaning. Pixel masks are costly and restrict datasets to few objects, often captured on lab tabletops (Nagarajan et al., 2019); heatmaps from points are cheaper but are distributions with no sharp boundary.
• Clutter and viewpoint. Both UMD methods lost much of their accuracy on the cluttered sequences (Myers et al., 2015), and most UMD images were taken on a turntable in a clutter-free setup, which motivated IIT-AFF (Nguyen et al., 2017). In OPRA the product in the video and the product image differ in viewpoint, state (open or closed) and background (Fang et al., 2018).
• Dependence on detection. Object-based methods fail when the detector misses an object or fires on a non-object (Nguyen et al., 2017).
• Active and inactive objects. An object being used looks different from the same object at rest; Nagarajan et al. (2019) cite work that needed separate detectors for open and closed microwaves, and design their model around this gap.

Approaches

Geometric features and classical classifiers

Myers et al. (2015) predicted part affordances from local shape in RGB-D images with two methods. S-HMP segments the image into superpixels with a version of SLIC that also uses depth and surface normals, learns hierarchical sparse codes (hierarchical matching pursuit) from depth, surface normals, principal curvatures, shape index and curvedness, max-pools them within each superpixel and classifies each superpixel with linear SVMs. SRF, a structured random forest of eight trees, maps the features of each $16 \times 16$ patch to stored per-pixel confidence scores for the patch’s affordances, taking about 0.1 s per $640 \times 480$ image. Over the seven affordances on single objects, averaged over random splits, S-HMP reached a mean weighted F-measure $F^w_\beta$ of 0.557 and SRF 0.460; on the cluttered sequences they fell to 0.250 and 0.165. Adding geometric features to smoothed depth raised SRF from 0.323 to 0.460, and SRF on raw RGB-D values reached 0.055. Per affordance, S-HMP was best on contain (0.810) and wrap-grasp (0.767) and worst on grasp (0.367) and cut (0.373). They also applied both methods to the Cornell grasping dataset, where S-HMP reached 95.2% recognition and 92.0% point-wise detection accuracy; robotic grasping explains why the point criterion is lenient.

Segmentation networks

Nguyen et al. (2016) replaced hand-designed features with a convolutional neural network of encoder–decoder form, trained from scratch on UMD with RGB, RGB-D or RGB with HHA-encoded depth as input. Their RGB-D network reached a mean $F^w_\beta$ of 0.770 on UMD’s single objects, against 0.557 for S-HMP and 0.733 for DeepLab; the RGB-only network reached 0.766, and HHA encoding did worse (0.675), which they attribute to the encoding’s gravity estimation failing on UMD’s tabletop scenes. The per-affordance HMP and SRF values in their table do not match Myers et al.’s: the two columns appear in Myers et al.’s row order (grasp, cut, scoop, contain, pound, support, wrap-grasp) but under the labels grasp, w-grasp, cut, contain, support, scoop, pound, so five of the seven per-affordance baseline values carry the wrong name; the means agree. AffordanceNet’s UMD table repeats the same columns. Per-affordance comparisons with the UMD baselines should therefore be read from Myers et al.’s own table.

Object-based affordance detection

UMD shows one tool at a time. For cluttered scenes, Nguyen et al. (2017) narrowed the region of interest with an object detector instead of labelling the whole image: they first detected objects with R-FCN (ResNet-101), then labelled the pixels inside each box with a VGG-16 network converted to fully convolutional form, with atrous convolutions to widen the receptive field and three input scales, and refined the result with a dense CRF. The detector’s score threshold was chosen by the affordance segmentation result on the validation set (0.5 from 0.3 to 0.6), and pixels outside all boxes are background. On their new IIT-AFF dataset, the object-based network BB-CNN reached a mean $F^w_\beta$ of 68.57 (in percent) and 69.62 with the CRF, against 58.38 for DeepLab, 59.86 for DeepLab with CRF and 57.64 for the encoder–decoder on RGB. The CRF added only about one point, which the authors did not expect, although it gave cleaner boundaries.

AffordanceNet (Do et al., 2018) merged the two stages into one network trained end to end, following Mask R-CNN. A VGG-16 backbone and a region proposal network share features; for each region of interest, RoIAlign pools a $7 \times 7$ feature map, an object branch classifies the box and regresses its coordinates, and an affordance branch upsamples the map with three transposed convolutions to $244 \times 244$ and assigns every pixel one of $C + 1$ affordance labels (background included) with a softmax. Three choices separate it from Mask R-CNN, whose mask branch predicts a small binary mask per class ($28 \times 28$ in its FPN version) with a per-pixel sigmoid:

• Multiclass masks. Each box holds several affordances, so the loss is a multinomial cross-entropy over affordance labels, computed only for positive boxes, added to the box classification and regression losses: $L = L_\text{cls} + \mathbb{1}[u \ge 1] L_\text{loc} + \mathbb{1}[u \ge 1] L_\text{aff}$.
• Large masks. On IIT-AFF, mean $F^w_\beta$ rose with the mask size: 57.71 at $14 \times 14$, 66.13 at 28, 71.54 at 56, 72.52 at 112 and 73.35 at 244.
• Resizing labels without mixing them. Interpolating a label mask to $244 \times 244$ blends neighbouring label values, so the labels present are first mapped to $0, \dots, n-1$, the mask is resized, and each pixel keeps value $\hat p$ only if it lies within $\alpha = 0.005$ of it, otherwise it becomes background.

AffordanceNet reached a mean $F^w_\beta$ of 73.35 on IIT-AFF, 3.7 points above BB-CNN-CRF, and 0.799 on UMD’s RGB images, against 0.770 for the RGB-D encoder–decoder, at about 150 ms per image on a Titan X GPU; the authors attribute the smaller gain on UMD to its clutter-free scenes. The official repository’s README notes that the released network differs slightly from the paper’s while reaching the same accuracy, and describes AffordanceNet as a general version of Mask R-CNN for multiple classes inside each instance (as of November 2026, commit 8db1b3f); its configuration keeps the $244 \times 244$ mask and a 10-way affordance output (9 classes and background). The authors’ copy read for this article is the arXiv version, not the ICRA proceedings version.

Example: local appearance against object context

The example tests the claim behind object-based detection, that an affordance label depends on the whole object and not only on how a part looks. It renders single tools at random positions and orientations on noisy backgrounds, in random colours, with part-level labels: knives, hammers and turners share an identical handle (grasp), the knife blade (cut) is deliberately the same width, colour and texture as the handle and differs only in its slanted tip and a small bolster at the junction, hammer heads (pound) and turner heads (support) are plain blocks of different proportions, and cups have thin walls (wrap-grasp) round a darker interior (contain). Two fully convolutional per-pixel classifiers have the same depth: the local network sees a $7 \times 7$ window (three $3 \times 3$ convolutions followed by $1 \times 1$ ones), the context network a $35 \times 35$ window (the $1 \times 1$ layers replaced by $3 \times 3$ layers with dilations 2, 4 and 8, the atrous idea used by IIT-AFF’s network). By construction, the blade’s plain stretch between the bolster and the start of the slanted tip is at most 16 pixels long, so pixels in its middle see only a plain bar through a $7 \times 7$ window, as do pixels in the middle of the handle, while no blade pixel is more than about 8 pixels along the bar from the bolster or the tip, inside the context window.

Scoring follows the UMD code release, ported from MATLAB: WFb.m for $F^w_\beta$ and evaluate_wfb_prediction.m for the protocol, which crops each image to the box around all labelled pixels and scores only the affordances present in its ground truth (see Evaluation Metrics); each affordance’s scores are averaged over the test images that contain it. The UMD code scores continuous probability maps; AffordanceNet assigns each pixel its most probable label and released no evaluation script, so which kind of map its numbers come from is not stated. The example prints both. The first block checks the port on a square “part” and shows how $F^w_\beta$ differs from the plain F1 score.

import numpy as np
from scipy import ndimage

EPS = np.finfo(float).eps                      # MATLAB's eps


def wfb(fg, gt):
    """Weighted F-measure, a port of WFb.m from the UMD part-affordance code release."""
    gt = gt.astype(bool)
    dgt = gt.astype(float)
    E = np.abs(fg - dgt)
    # bwdist: distance to the nearest ground-truth pixel, and that pixel's index
    dst, (iy, ix) = ndimage.distance_transform_edt(~gt, return_indices=True)
    Et = E.copy()
    Et[~gt] = E[iy[~gt], ix[~gt]]              # outside the object, take the error of the nearest object pixel
    g = np.arange(-3, 4)
    K = np.exp(-(g[:, None] ** 2 + g[None, :] ** 2) / (2 * 5.0 ** 2))
    K /= K.sum()                               # fspecial('gaussian', 7, 5)
    EA = ndimage.correlate(Et, K, mode="constant", cval=0.0)   # imfilter: correlation, zero padding
    min_e_ea = E.copy()
    m = gt & (EA < E)
    min_e_ea[m] = EA[m]                        # pixel dependency: only errors on object pixels are smoothed
    B = np.ones_like(E)
    B[~gt] = 2 - np.exp(np.log(0.5) / 5 * dst[~gt])            # pixel importance grows with distance
    Ew = min_e_ea * B
    TPw = dgt.sum() - Ew[gt].sum()
    FPw = Ew[~gt].sum()
    R = 1 - Ew[gt].mean()                      # weighted recall
    P = TPw / (EPS + TPw + FPw)                # weighted precision
    return 2 * R * P / (EPS + R + P)           # beta = 1


def f1(fg, gt):
    tp = (fg & gt).sum()
    return 2 * tp / (fg.sum() + gt.sum())


gt = np.zeros((64, 64), bool)
gt[22:42, 22:42] = True                        # a 20 x 20 object part, 400 pixels
rng = np.random.default_rng(0)

near = gt.copy(); near[21, 22:42] = True; near[42, 22:42] = True          # 40 false positives on the boundary
far = gt.copy(); far[2:6, 2:12] = True                                     # 40 false positives far away
block = gt.copy(); block[22:26, 22:32] = False                             # 40 missed pixels in one block
scatter = gt.copy()
idx = rng.choice(400, 40, replace=False)
scatter[22 + idx // 20, 22 + idx % 20] = False                             # 40 missed pixels scattered

for name, pred in [("perfect", gt), ("empty", np.zeros_like(gt)), ("40 FP on the boundary", near),
                   ("40 FP far away", far), ("40 FN in one block", block), ("40 FN scattered", scatter)]:
    print(f"{name:<24} F1 {f1(pred, gt):.3f}   F^w {wfb(pred.astype(float), gt):.3f}")
print(f"{'soft map, 0.5 on object':<24} F^w {wfb(0.5 * gt, gt):.3f}")
r, c = np.nonzero(gt)
crop = (slice(r.min(), r.max() + 1), slice(c.min(), c.max() + 1))
print(f"{'40 FP far, UMD crop':<24} F^w {wfb(far[crop].astype(float), gt[crop]):.3f}")

Output (NumPy 2.3.5, SciPy 1.16.3; two runs gave identical output):

perfect                  F1 1.000   F^w 1.000
empty                    F1 0.000   F^w 0.000
40 FP on the boundary    F1 0.952   F^w 0.947
40 FP far away           F1 0.952   F^w 0.911
40 FN in one block       F1 0.947   F^w 0.962
40 FN scattered          F1 0.947   F^w 0.994
soft map, 0.5 on object  F^w 0.667
40 FP far, UMD crop      F^w 1.000

F1 cannot tell the pairs apart; $F^w_\beta$ can, in the directions Margolin et al. (2014) intended. False positives far from the part cost more than the same number on its boundary, since the weight $B$ grows from about 1 to 2 with distance. Missed pixels scattered among detected ones cost far less than a missing block, because each scattered error is averaged with its correct neighbours by the Gaussian before the minimum is taken. A map of 0.5 over the whole part earns partial credit with no threshold at all. The last line shows a property of the UMD protocol rather than of the measure: after cropping to the labelled box, false positives outside the object are not scored.

The second block renders the data, trains both networks from three seeds and scores them. It runs in about ten minutes on a laptop CPU, almost all of it training.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from scipy import ndimage

torch.set_num_threads(4)
H = W = 64
NAMES = ["background", "grasp", "cut", "pound", "support", "contain", "wrap-grasp"]
EPOCHS, SEEDS = 20, [0, 1, 2]


def polygon(r, c, shape):
    """Fill a convex polygon: pixels whose centre lies inside or on every edge."""
    yy, xx = np.mgrid[0:shape[0], 0:shape[1]]
    n = len(r)
    area = sum(c[i] * r[(i + 1) % n] - c[(i + 1) % n] * r[i] for i in range(n))
    inside = np.ones(shape, bool)
    for i in range(n):
        x0, y0, x1, y1 = c[i], r[i], c[(i + 1) % n], r[(i + 1) % n]
        inside &= np.sign(area) * ((x1 - x0) * (yy - y0) - (y1 - y0) * (xx - x0)) >= 0
    return np.nonzero(inside)


def rect(x0, x1, y0, y1):
    return [(x0, y0), (x1, y0), (x1, y1), (x0, y1)]


def render(rng):
    """One tool on a noisy background: image (H, W, 3) and per-pixel affordance labels."""
    bg = rng.uniform(0.25, 0.75)
    img = np.full((H, W, 3), bg) + rng.normal(0, 0.03, (H, W, 3))
    while True:
        colour = rng.uniform(0, 1, 3)
        if abs(colour.mean() - bg) > 0.2:
            break
    kind = rng.integers(4)                          # knife, hammer, turner, cup
    Lh, Lb = rng.uniform(16, 20, 2)
    handle = (rect(0, Lh, -2, 2), 1, 1.0)           # the same bar on every handled tool
    if kind == 0:                                   # blade: same width and colour, slanted tip; bolster
        parts = [handle, ([(Lh, -2), (Lh + Lb - 4, -2), (Lh + Lb, 2), (Lh, 2)], 2, 1.0),
                 (rect(Lh - 2, Lh, -3, 3), 1, 1.0)]
    elif kind == 1:
        parts = [handle, (rect(Lh, Lh + 6, -8, 8), 3, 1.0)]        # hammer head: pound
    elif kind == 2:
        parts = [handle, (rect(Lh, Lh + 14, -5, 5), 4, 1.0)]       # turner head: support
    else:                                           # cup: walls (wrap-grasp) round a darker interior (contain)
        parts = [(rect(0, 16, -9, 9), 6, 1.0), (rect(3, 16, -6, 6), 5, 0.55)]
    theta = rng.uniform(0, 2 * np.pi)
    R = np.array([[np.cos(theta), np.sin(theta)], [-np.sin(theta), np.cos(theta)]])
    pts = np.array([p for poly, _, _ in parts for p in poly])
    centre = (pts.min(0) + pts.max(0)) / 2
    rot = (pts - centre) @ R
    cx, cy = rng.uniform(-rot.min(0) + 2, (W - 3) - rot.max(0))
    label = np.zeros((H, W), int)
    for poly, lab, f in parts:
        q = (np.array(poly) - centre) @ R
        rr, cc = polygon(q[:, 1] + cy, q[:, 0] + cx, (H, W))
        label[rr, cc] = lab
        img[rr, cc] = colour * f + rng.normal(0, 0.03, (len(rr), 3))
    return np.clip(img, 0, 1), label


def make_set(n, seed):
    rng = np.random.default_rng(seed)
    imgs, labs = zip(*[render(rng) for _ in range(n)])
    return (torch.tensor(np.stack(imgs).transpose(0, 3, 1, 2), dtype=torch.float32),
            torch.tensor(np.stack(labs)))


def network(context):
    """Same depth for both: three 3x3 convs, then three 1x1 (local) or dilated 3x3 (context) convs."""
    layers, c = [], 3
    for _ in range(3):
        layers += [nn.Conv2d(c, 32, 3, padding=1), nn.ReLU()]
        c = 32
    for d in (2, 4, 8):
        conv = nn.Conv2d(32, 32, 3, padding=d, dilation=d) if context else nn.Conv2d(32, 32, 1)
        layers += [conv, nn.ReLU()]
    layers.append(nn.Conv2d(32, 7, 1))
    return nn.Sequential(*layers)                   # receptive field 7 x 7 (local) or 35 x 35 (context)


def train(context, seed, X, Y):
    torch.manual_seed(seed)
    model = network(context)
    opt = torch.optim.Adam(model.parameters(), lr=2e-3)
    g = torch.Generator().manual_seed(seed)
    for _ in range(EPOCHS):
        losses = []
        for idx in torch.randperm(len(X), generator=g).split(16):
            loss = F.cross_entropy(model(X[idx]), Y[idx])
            opt.zero_grad()
            loss.backward()
            opt.step()
            losses.append(loss.item())
    return model, np.mean(losses)                   # mean loss over the last epoch


EPS = np.finfo(float).eps


def wfb(fg, gt):
    """Weighted F-measure, a port of WFb.m from the UMD part-affordance code release."""
    gt = gt.astype(bool)
    dgt = gt.astype(float)
    E = np.abs(fg - dgt)
    dst, (iy, ix) = ndimage.distance_transform_edt(~gt, return_indices=True)
    Et = E.copy()
    Et[~gt] = E[iy[~gt], ix[~gt]]
    gg = np.arange(-3, 4)
    K = np.exp(-(gg[:, None] ** 2 + gg[None, :] ** 2) / (2 * 5.0 ** 2))
    K /= K.sum()
    EA = ndimage.correlate(Et, K, mode="constant", cval=0.0)
    min_e_ea = E.copy()
    m = gt & (EA < E)
    min_e_ea[m] = EA[m]
    B = np.ones_like(E)
    B[~gt] = 2 - np.exp(np.log(0.5) / 5 * dst[~gt])
    Ew = min_e_ea * B
    TPw = dgt.sum() - Ew[gt].sum()
    FPw = Ew[~gt].sum()
    Rw = 1 - Ew[gt].mean()
    Pw = TPw / (EPS + TPw + FPw)
    return 2 * Rw * Pw / (EPS + Rw + Pw)


def evaluate(maps, lab):
    """evaluate_wfb_prediction.m: crop to the box around all labelled pixels; score affordances present."""
    r, c = np.nonzero(lab)
    crop = (slice(r.min(), r.max() + 1), slice(c.min(), c.max() + 1))
    return {k: wfb(maps[k][crop], lab[crop] == k) for k in np.unique(lab[crop]) if k > 0}


Xtr, Ytr = make_set(400, 0)
Xte, Yte = make_set(100, 1)
results = {}
for context in (False, True):
    for seed in SEEDS:
        model, last_loss = train(context, seed, Xtr, Ytr)
        with torch.no_grad():
            prob = F.softmax(model(Xte), 1).double().numpy()
        hard = np.eye(7)[prob.argmax(1)].transpose(0, 3, 1, 2)       # one-hot argmax maps
        lab = Yte.numpy()
        scores = {"prob": {k: [] for k in range(1, 7)}, "argmax": {k: [] for k in range(1, 7)}}
        for i in range(len(lab)):
            for key, maps in (("prob", prob[i]), ("argmax", hard[i])):
                for k, v in evaluate(maps, lab[i]).items():
                    scores[key][k].append(v)
        cut = lab == 2
        results[context, seed] = dict(
            prob=[np.mean(scores["prob"][k]) for k in range(1, 7)],
            argmax=[np.mean(scores["argmax"][k]) for k in range(1, 7)],
            cut_as_grasp=(prob.argmax(1)[cut] == 1).mean(), loss=last_loss)
        n_par = sum(p.numel() for p in model.parameters())
    print(f"{'context' if context else 'local'} network: {n_par} parameters")

print(f"{'mean F^w over seeds':<20}" + "".join(f"{n:>11}" for n in NAMES[1:]) + f"{'mean':>8}")
for context in (False, True):
    for key in ("prob", "argmax"):
        v = np.mean([results[context, s][key] for s in SEEDS], axis=0)
        print(f"{('context' if context else 'local') + ', ' + key:<20}" + "".join(f"{x:11.3f}" for x in v)
              + f"{v.mean():8.3f}")
for context in (False, True):
    name = "context" if context else "local"
    per_seed = ", ".join(f"{np.mean(results[context, s]['prob']):.3f}/{np.mean(results[context, s]['argmax']):.3f}"
                         for s in SEEDS)
    confusion = ", ".join(f"{results[context, s]['cut_as_grasp']:.3f}" for s in SEEDS)
    losses = ", ".join(f"{results[context, s]['loss']:.3f}" for s in SEEDS)
    print(f"{name:<8} seeds {SEEDS}: mean F^w prob/argmax {per_seed}")
    print(f"{'':<9}cut pixels labelled grasp {confusion}; last-epoch training loss {losses}")

Output (NumPy 2.5.4, SciPy 1.18.1, PyTorch 2.14.1 with 4 threads; two runs gave identical output):

local network: 22791 parameters
context network: 47367 parameters
mean F^w over seeds       grasp        cut      pound    support    contain wrap-grasp    mean
local, prob               0.327      0.251      0.278      0.344      0.244      0.325   0.295
local, argmax             0.500      0.187      0.144      0.139      0.030      0.209   0.202
context, prob             0.477      0.333      0.525      0.681      0.727      0.646   0.565
context, argmax           0.534      0.300      0.686      0.750      0.831      0.769   0.645
local    seeds [0, 1, 2]: mean F^w prob/argmax 0.291/0.202, 0.297/0.205, 0.297/0.198
         cut pixels labelled grasp 0.916, 0.788, 0.900; last-epoch training loss 0.104, 0.104, 0.110
context  seeds [0, 1, 2]: mean F^w prob/argmax 0.698/0.785, 0.353/0.438, 0.643/0.711
         cut pixels labelled grasp 0.478, 0.000, 0.954; last-epoch training loss 0.032, 0.094, 0.043

The context network scores higher than the local one on every affordance, under both kinds of map: its mean $F^w_\beta$ over the six affordances is 0.565 on probability maps and 0.645 on label maps, against 0.295 and 0.202. The gains are largest on contain, support, wrap-grasp and pound. In our reading, these are the parts that a $7 \times 7$ window sees as a patch of colour or a pair of edges, and whose identity depends on proportions and surroundings: by construction the cup’s interior is darker only relative to its walls, and a hammer head and a turner head differ only in shape at a scale of 6 to 14 pixels. The plan expected the thin walls to be easier for the local network than the uniform parts, so that the gap would be smallest on wrap-grasp; the local network scores wrap-grasp above all three uniform parts on label maps but below support on probability maps, and under both kinds of map the gap on wrap-grasp is larger than on pound, so the expectation is not supported.

The blade is where the plan’s main prediction only partly held. Cut is the context network’s weakest affordance (0.333 and 0.300), and the fraction of blade pixels it labels grasp ranges from 0.000 to 0.954 over the three seeds, while the local network labels 79% to 92% of them grasp; on label maps the context network’s grasp score is barely above the local one’s. For the local network this is the expected answer by construction: three of the four tool types carry the same handle and only one carries a blade, so a plain bar is most often a handle. The context network has the bolster and the tip in view, yet only one seed avoided the handle label on the blade entirely, and that seed had the lowest mean $F^w_\beta$ of the three (0.353), so its errors lay elsewhere in ways the printed numbers do not show. Its last-epoch training loss (0.094) is also close to the local network’s, while the other two seeds reached 0.032 and 0.043; in our reading, 20 epochs were not enough for every seed to learn the cues, and the spread over seeds (0.353 to 0.698 on probability maps) is part of the result. The example therefore supports the weaker claim that object context is needed, not that a network given the context will use it.

The two kinds of map change the numbers more than the ordering. Thresholding the local network’s maps to labels lowers most of its scores, contain from 0.244 to 0.030, in our reading because it seldom if ever makes contain its first choice (under the crop, a part that touches the crop’s edge earns some recall even from an empty map; see Evaluation Metrics), but raises grasp from 0.327 to 0.500, its usual first choice for any bar. The context network’s label maps score higher than its probability maps on every affordance but cut. The gap between the networks is 0.27 on probability maps and 0.44 on label maps, so scores computed from different kinds of map are not comparable, and a paper that does not say which it used leaves its numbers open. The toy says nothing about absolute values on UMD or IIT-AFF, whose images, labels and networks are of another scale.

Learning affordances from demonstration videos

Labelled parts say where an annotator thinks an action is possible; videos of people show where it happens. Demo2Vec (Fang et al., 2018) learns from product-review videos: a demonstration encoder (a convolutional LSTM with temporal soft attention over RGB and motion features) embeds a video clip, and an affordance predictor uses the embedding to output, on a separate image of the same product, the interaction heatmap and the action label. The heatmap is trained with a KL-divergence loss against the Gaussian-blurred annotated points. On their OPRA test set, the core model had a KL divergence of 2.34 against 3.45 for a plain LSTM encoder, and classified the action with 40.79% top-1 accuracy; the authors report confusions between similar actions such as rotating and holding. Demo2Vec is supervised: it needs the annotated heatmaps for training and a demonstration video at test time (Nagarajan et al., 2019).

Interaction hotspots (Nagarajan, Feichtenhofer and Grauman, 2019) drop the heatmap annotations. Training uses only action labels for video clips, plus images of the objects at rest:

1. A video action classifier encodes each frame with a ResNet whose last two stages have stride 1 and dilated filters (a $2048 \times 28 \times 28$ feature map), pools each frame with L2 pooling, aggregates the frames with an LSTM and predicts the afforded action, such as pourable or cuttable, across all object categories.
2. An anticipation network maps the embedding of an inactive object image (a catalogue photo in OPRA, a crop from a frame before the action in EPIC-Kitchens) to the embedding of the frame where the LSTM is most confident of the true action, with an L2 feature-matching loss (a triplet loss on EPIC-Kitchens, where the crop shows a different instance), plus an auxiliary classification loss.
3. At test time, the hotspot map for action $a$ weights each channel of the inactive image’s embedding by the gradient of the action score through the anticipation network and sums the positive parts:

$$H_a(x_I) = \sum_k \operatorname{ReLU}\left( \frac{\partial y_a}{\partial x_I^k} \odot x_I^k \right).$$

The L2 pooling makes the gradients vary over space, which they would not after average pooling, and the authors contrast their maps with Grad-CAM, whose spatially averaged gradients give coarser, more diffuse maps. On OPRA, the hotspots reached a KLD of 1.427, SIM of 0.362 and AUC-J of 0.806, against 1.897, 0.296 and 0.720 for the best saliency model, DeepGaze II, on KLD, and 1.473, 0.355 and 0.821 for a supervised image-to-heatmap network; Demo2Vec, scored from its authors’ precomputed predictions, reached 1.197, 0.482 and 0.847. On their EPIC-Kitchens annotations the hotspots reached 1.258, 0.404 and 0.785. On object classes unseen in training, averaged over three class splits, they reached 1.381, 0.374 and 0.826 on OPRA. The Demo2Vec values in the two papers (2.34 and 1.197) come from different set-ups and are not comparable.

Learning from exocentric images

Luo et al. (2022) ground affordances from third-person photos of people using objects, supervised only by the affordance label of each image. In training, $N = 3$ exocentric images and one egocentric image of an object of the same affordance pass through a shared ResNet-50:

• Affordance Invariance Mining removes individual habits from the exocentric features (people hold a racket in different places) by a low-rank decomposition: the features of the $N$ images are factorized by non-negative matrix factorization into a shared dictionary $W$ (rank 64, 6 multiplicative-update iterations) and coefficients, and the reconstruction is added back to the features as a residual.
• Affordance Co-relation Preserving aligns the two branches’ predictions over affordance classes: each branch’s class probabilities $p$ and $q$ form matrices $P = p p^\top$ and $Q = q q^\top$, and a cross-entropy between them makes the egocentric branch keep relations such as “cut with” always accompanying “hold”.
• A knowledge-transfer loss pulls the pooled egocentric features towards the exocentric ones.

At test time only the egocentric branch runs, and class activation mapping (CAM) over its last convolutional layer gives the heatmap for the requested affordance. On AGD20K’s Seen split the method reached a KLD of 1.538, SIM of 0.334 and NSS of 0.927, and on the Unseen split 1.787, 0.285 and 0.829; interaction hotspots, as Luo et al. report them on AGD20K, reached 1.773, 0.278 and 0.615 (Seen) and 1.994, 0.237 and 0.577 (Unseen). In the ablation, removing the three components together left 1.985, 0.238 and 0.302 on Seen. The authors note that the maps can spread over the background between several objects and around slender objects.

Datasets and Benchmarks

Dataset	Year	Images	Labels	Splits	Metrics
UMD part affordance	2015	RGB-D, $640 \times 480$, 105 tools in 17 categories on a turntable; about 30,000 frames, over 10,000 with labels; 3 cluttered sequences	7 affordances; top label and ranking of all 7 per pixel	Category split, novel-category split, 10-fold leave-one-out	$F^w_\beta$, rank-weighted $R^w_\beta$, Kendall’s $\tau$
IIT-AFF	2017	8,835 RGB images, from ImageNet and from two depth cameras in cluttered scenes	10 objects, 9 affordances; 14,642 boxes, 24,677 affordance regions	50% train, 20% validation, 30% test	$F^w_\beta$
OPRA	2018	20,774 pairs of a product-review video clip and a product image	7 actions; heatmap from 10 points	16,976 train, 3,798 test, distinct products	KLD, NLL; later SIM, AUC-J
EPIC-Kitchens hotspot annotations	2019	Crowd-sourced heatmaps for egocentric kitchen video	1.8k annotated instances, 20 actions, 31 objects	Seen and unseen object classes	KLD, SIM, AUC-J
AGD20K	2022	20,061 exocentric and 3,755 egocentric images	36 affordances, 50 objects; image labels for training, heatmaps for testing	Seen; Unseen (25 affordances, disjoint objects)	KLD, SIM, NSS

UMD. Myers et al. (2015) recorded each tool with a Kinect at about 0.8 m, the minimum distance for accurate depth, on a turntable covering 360°, about 300 frames per tool, and added three sequences of about 1,000 frames taken by a mobile robot looking at unseen tools in clutter. The seven affordances are grasp (handles), cut (blades), scoop (curved surfaces with a mouth, such as trowels), contain (deep cavities), pound (hammer heads), support (flat parts such as turners) and wrap-grasp (the outside of a cup). According to the README, hand labels exist for every third frame; the other frames have automatically generated labels that were not used for training or testing. The label files give the top-ranked affordance per pixel (gt_label, zero for background) and the ranks of all seven (*_label_rank.mat).

IIT-AFF. Nguyen et al. (2017) took a subset of object categories from ImageNet, reusing its boxes (7,866 of the 14,642), and added RGB-D images of cluttered scenes from an Asus Xtion ($640 \times 480$) and a MultiSense-SL camera ($1024 \times 1024$). All images carry pixel-level affordance labels; for the ImageNet images, which have no depth, relative depth maps were estimated. The affordances are contain, cut, display (a monitor screen), engine (a drill’s motor housing), grasp, hit (a racket head), pound, support and w-grasp. AffordanceNet describes the split as 70% for training and 30% for testing, which matches the original split if training and validation are merged.

OPRA. Fang et al. (2018) cut 1,091 videos from six YouTube product-review channels into 11,505 clips, each holding one interaction, and paired them with 2,512 product images found online, one to five per video. The seven actions are dominated by touch (45.12%) and hold (19.22%). The paper’s introduction gives 20,612 sets of clips, while the dataset section gives 20,774 pairs, which its training and test counts and per-action counts both add up to.

AGD20K. Luo et al. (2022) took the exocentric images mainly from HICO and COCO, removed ambiguous interactions by hand, and added 2,112 exocentric and all 3,755 egocentric images from websites with free licences. In the Seen setting training and test images share object categories; in the Unseen setting they do not. The official code lists 36 affordances for Seen and 25 for Unseen (as of November 2026, commit 6bd5638).

Evaluation Metrics

Weighted F-measure (UMD, IIT-AFF). Margolin, Zelnik-Manor and Tal (2014) proposed $F^w_\beta$ for comparing a binary or continuous map with a binary ground truth, to fix three flaws they found in AUC, average precision, $F_\beta$ and the PASCAL overlap: interpolating curves between thresholds, treating pixels as independent, and treating all errors as equally important (the salient object detection article places it among that field’s measures). The UMD code release computes it in WFb.m as follows, for a predicted map $D \in [0, 1]$ and a binary ground truth $G$. The error is $E = |D - G|$. Outside the foreground, each pixel takes the error of its nearest foreground pixel, giving $\tilde E$, which is filtered with a normalized $7 \times 7$ Gaussian of standard deviation 5 and zero padding, $E^A = K * \tilde E$. On foreground pixels the error becomes $\min(E, E^A)$, so isolated misses among correct pixels are forgiven; elsewhere it stays $E$. False positives are weighted by their distance $\Delta$ to the nearest foreground pixel, $B = 2 - e^{\alpha \Delta}$ with $\alpha = \ln(0.5)/5$ (about 1 next to the object, 1.5 at 5 pixels), and $B = 1$ on the foreground. With $E_w = \min(E, E^A) \cdot B$,

$$R^w = 1 - \frac{1}{|G|} \sum_{i \in G} E_w(i), \qquad P^w = \frac{TP_w}{TP_w + FP_w}, \qquad F^w_\beta = \frac{(1 + \beta^2)\, P^w R^w}{\beta^2 P^w + R^w},$$

where $TP_w = |G| - \sum_{i \in G} E_w(i)$, $FP_w = \sum_{i \notin G} E_w(i)$, and $\beta = 1$. The paper sets the Gaussian’s variance $\sigma^2$ to 5 and scales its matrix of pixel dependencies by $1/\sqrt{2\pi\sigma^2}$ over all foreground pixels, while the code uses a standard deviation of 5 and a $7 \times 7$ kernel normalized to sum 1; results reported with the code follow the code. In the UMD protocol (evaluate_wfb_prediction.m), each image is cropped to the bounding box of all labelled pixels, and $F^w_\beta$ is computed for each affordance present in that image’s ground truth, on that affordance’s predicted map. Three consequences follow from the code: false positives outside the object’s box are not counted; an affordance predicted on an object that has none is never scored; and because imfilter pads with zeros, a part that touches the edge of the crop earns some recall even from an empty map, so $F^w_\beta$ does not fall to zero when the affordance is never predicted. The release does not call evaluate_wfb_prediction.m from its other scripts, so how the ground-truth maps passed to it were built is not shown; it returns per-image values and does not show how they were averaged; the tables report a mean per affordance and the mean over affordances. IIT-AFF and AffordanceNet use the same measure with $\beta = 1$ and report it in percent; their evaluation code was not released, so whether they followed the UMD crop is not stated.

Rank-weighted F-measure and Kendall’s $\tau$ (UMD). Because UMD ranks several affordances per pixel, Myers et al. (2015) also define $R^w_\beta = \sum_r w_r F^w_\beta(r)$, a weighted sum over ranks with $\sum_r w_r = 1$ and the largest weight on the top rank, without giving the weights; and Kendall’s $\tau \in [-1, 1]$ between predicted and true rankings at each pixel. The paper reports the average $\tau$ of all pixels over the test images, whereas the released evaluate_model.m pools concordant and discordant pairs over all pixels of all images before forming $(n_c - n_d)/(n_c + n_d)$.

Heatmap measures (OPRA, EPIC, AGD20K). Grounding results are scored with measures from saliency prediction, following Demo2Vec’s use of KL divergence. AGD20K’s test.py resizes prediction and ground truth to $224 \times 224$, rescales the prediction to $[0, 1]$, and computes per image, with $\epsilon = 10^{-12}$ and $\hat D = D / \sum D$, $\hat G = G / \sum G$:

$$\mathrm{KLD} = \sum_i \hat G_i \log\!\left( \frac{\hat G_i}{\hat D_i + \epsilon} + \epsilon \right), \qquad \mathrm{SIM} = \sum_i \min(\hat D_i, \hat G_i), \qquad \mathrm{NSS} = \frac{\sum_i z_i M_i}{\sum_i M_i},$$

then averages over the test images. KLD is the Kullback–Leibler divergence from the ground-truth distribution to the prediction (lower is better), SIM is the histogram intersection of the two normalized maps (1 for identical maps), and NSS averages the prediction’s z-scores $z = (D - \mu_D)/\sigma_D$ over a mask $M$. Saliency’s NSS uses recorded fixation points as $M$; the AGD20K code instead min-max normalizes the ground-truth heatmap and marks the pixels above 0.1. Nagarajan et al. (2019) report AUC-J, an area under the ROC curve from the saliency literature, instead of NSS. Values from different resizing, normalization or mask rules are not comparable, even under the same name.

Applications

The main use is in robot manipulation, where affordance maps tell a robot which part of an object to hold and which part to use; robot perception lists affordances among the high-level understanding that robots were expected to need beyond geometry. Nguyen et al. (2017) turned a detected grasp region into a grasp for the humanoid WALK-MAN by fitting an ellipse to the region: its centre is the grasp point and the eigenvectors of the region’s covariance give the orientation, projected to 3D with the depth camera. With 10 trials for each of six objects in clutter, counting a grasp as successful if the robot lifted the object and held it for 10 seconds, this succeeded in 90.0% of trials on average, against 76.7% for a rectangle fitted to the convex hull of the same region, which the authors found sensitive to outliers. They used the contain region of a bowl for its grasp and treated all affordances as graspable, a simplification they name. AffordanceNet was used on the same robot for grasping, pick-and-place and pouring: the grasp affordance of a bottle tells it where to hold, the contain affordance of a pan where to pour (Do et al., 2018). These are demonstrations rather than benchmarks; robotic grasping explains why grasp success rates depend on objects, clutter and trial protocol.

Grounding from video is motivated by agents that must interact with objects they have not seen used, from household robots to systems that anticipate a person’s next action (Nagarajan et al., 2019; Luo et al., 2022). Learned manipulation policies use their own pixel-wise affordance maps, learned from trials, as described in vision-based robotic manipulation.

Open Problems

• What the label means. Part masks record annotators’ notions of function, heatmaps record where interactions happened, and SayCan’s value functions estimate whether a skill will succeed; results under one definition say little about the others.
• Small, constrained datasets. UMD has 105 tools on a turntable, IIT-AFF ten object categories, and the video datasets record where people touched objects rather than whether an action would succeed. Nagarajan et al. (2019) note that the vocabularies of the segmentation datasets barely overlap those of the video datasets.
• Protocol details hidden behind one name. The weighted F-measure depends on the crop, on probability or label maps, and on the kernel the code uses; the heatmap measures depend on resizing and normalization. Of the code read for this article, only the UMD release and the AGD20K repository contain evaluation code (AffordanceNet’s repository has none), and tables copied between papers have carried mislabelled rows.
• Clutter, occlusion and unseen categories. Accuracy dropped sharply on UMD’s cluttered sequences, object-based methods inherit detector failures, and AGD20K’s best method scores worse on unseen objects than on seen ones on all three measures.
• From maps to actions. A region says where to act, not how: grasp poses, forces and the effect on another object remain to be planned, and the robot demonstrations above treat every affordance as a place to grasp.