Visual Grounding

Visual grounding localizes the part of an image that a piece of text refers to. Given a photograph and “the man in the red hat”, a grounding model returns a box around that man. Object detection returns every instance of fixed categories; grounding returns the region one free-form description picks out, so “the dog on the left” and “the dog on the right” get different answers. In the multimodal learning taxonomy it is explicit alignment between words and regions.

Problem Definition

In referring expression comprehension (REC), the input is an image $I$ and an expression $r$ written to identify exactly one object, and the output is one bounding box $\hat{b}$. A prediction is correct when its intersection over union with the annotated box $b^*$ passes 0.5, and accuracy over $N$ test expressions is

$$\mathrm{Acc@0.5} = \frac{1}{N} \sum_{n=1}^{N} \mathbb{1}\big[\mathrm{IoU}(\hat{b}_n, b^*_n) \ge 0.5\big].$$

Some papers write IoU above 0.5 (Mao et al., 2016), others IoU $\ge 0.5$ (Plummer et al., 2015).

In phrase grounding (or phrase localization), each annotated phrase of a caption must be linked to the regions it mentions: in “a man throws a ball to his dog”, boxes for a man, a ball and his dog. A phrase may refer to several objects (“two children”) and several phrases to one object, which REC rules out by construction.

Input and Output

Ranking methods choose the answer from region proposals $\mathcal{C}(I)$ produced by a detector:

$$\hat{b} = \arg\max_{b \in \mathcal{C}(I)} s(b, r).$$

Mao et al. (2016) ranked by $p(r \mid b, I)$, the likelihood that a captioning-style model generates the expression for that region. Later methods regress the box from fused image and text features, or have a language model write its coordinates as text.

Variants

• Referring expression segmentation returns a pixel mask (Hu, Rohrbach & Darrell, 2016): “two men sitting on the right bench” must select only those two people, which fixed-class semantic segmentation cannot express.
• Open-vocabulary detection is grounding with class names as the text. GLIP (Li et al., 2022) recast detection as phrase grounding, with a prompt such as “person. bicycle. car. …” in place of the classifier. Grounding DINO (Liu et al., 2024) is an open-set detector of this kind that accepts category names or referring expressions.
• Referring expression generation, the dual task, writes an expression that singles out a given object (Mao et al., 2016; Yu et al., 2016).

Challenges

• Ambiguity. An expression must be unique in its image, so datasets keep only expressions with which a second annotator clicked the right object (Kazemzadeh et al., 2014; Mao et al., 2016). The COCO-based datasets chose images with several objects of the same category, so the category alone rarely decides (Yu et al., 2016).
• Relations, viewpoint and counting. “The cup left of the laptop” requires finding a second object and comparing positions, and “left” may be the viewer’s or the object’s. About 41% of RefCOCO and 36% of RefCOCOg expressions contain absolute location words, MAttNet’s authors report (Yu et al., 2018), and ordinals such as “the second man from the left” add counting.
• Small objects. A one-pixel shift costs a small box much of its IoU, and RefCOCOg kept only objects covering at least 5% of the image (Mao et al., 2016).
• Proposal quality. A ranking method cannot recover an object its first stage missed (Yang et al., 2019).

Approaches

Ranking region proposals

The first deep methods followed the two-stage pattern of Faster R-CNN: detect candidate objects, then score each against the expression. MAttNet (Yu et al., 2018) split the expression with language attention into phrase embeddings for a subject module (appearance and attributes), a location module (the box’s corners and area, plus offsets to up to five same-category neighbors) and a relationship module, which scores up to five surrounding objects and keeps the best match, a form of multiple-instance learning; learned weights combine the three scores. On RefCOCO val it reached 85.65% accuracy choosing among ground-truth COCO boxes but 76.40% among boxes detected by Faster R-CNN with the same features (76.65% with the detector of Mask R-CNN) — the cost of the first stage.

One-stage grounding

Yang et al. (2019) fused a BERT embedding of the query, with an 8-channel map of spatial coordinates, into the feature maps of the one-stage detector YOLOv3 at three resolutions and predicted the box directly. They reported it about 10 times faster than two-stage methods, and more accurate.

Transformers

TransVG (Deng et al., 2021) replaced hand-designed fusion modules with transformer encoder layers. Visual tokens and BERT word tokens are joined with a learnable [REG] token and processed by six encoder layers; an MLP on the [REG] output regresses the box, trained with smooth L1 plus generalized IoU losses. This is multimodal fusion by self-attention, without proposals or anchors.

MDETR (Kamath et al., 2021) built grounding into a DETR detector. CNN image features and RoBERTa text features are projected to a shared space, concatenated and fed to the transformer encoder–decoder, so every predicted box is conditioned on the text. Instead of a class, each box predicts the span of text tokens it refers to (soft token prediction), and a contrastive loss aligns object and token embeddings. It was pretrained on 1.3M aligned image–text pairs from about 200,000 COCO, Visual Genome and Flickr30k images, then fine-tuned for REC.

Grounded pretraining

GLIP (Li et al., 2022) replaces a detector’s classification logits with alignment scores between region features $O$ and prompt word features $P$, $S = O P^\top$, so detection and grounding data train one model, and cross-attention fuses image and text in the last encoder layers. With web image–text pairs pseudo-labeled by a grounding teacher, GLIP-L was pretrained on 27M grounding examples (3M human-annotated); the authors reported 49.8 AP on COCO val2017 without COCO images in pretraining. Grounding DINO applies grounded pretraining to the DINO detector, fusing language through a feature enhancer, language-guided query selection and a cross-modality decoder (Liu et al., 2024).

Language models that write coordinates

Vision–language models can output boxes as text. Kosmos-2 (Peng et al., 2023, arXiv version) adds a token for each cell of a $32 \times 32$ grid, writes a box as its top-left and bottom-right cells, and links it to a text span in a Markdown-like [text span](box) form. Shikra (Chen et al., 2023) adds no tokens: boxes are plain numbers, $[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$ normalized by image size with three decimals.

Grounding plus segmentation

Grounded SAM (Ren et al., 2024) chains models: Grounding DINO turns text into boxes, and the boxes prompt SAM for masks, giving text-to-mask segmentation without joint training.

Example: a proposal-ranking grounder on synthetic images

The script draws $64 \times 64$ RGB images of four to six red, green or blue circles, squares and triangles, with invented expressions each true of one object: “the square”, “the red square” or, when a shape repeats, “the circle right of the green square”. Proposals, standing in for a detector, are the true boxes with each corner moved by up to $\pm j$ pixels, each described by its pixels: the foreground’s mean color, fill fraction, share in the top half and corner fullness.

The rules read a color (largest channel) and a shape (full corners: square; little mass on top: triangle; otherwise circle), with thresholds set on tight boxes; for relations they find the landmark and prefer the candidate farthest on the named side. The learned scorer is multinomial logistic regression over an image’s proposals, with a weight vector $\mathbf{u}_w$ per attribute word and $\mathbf{v}_r$ per relation:

$$s_i = \sum_{w \in \text{target}} \mathbf{u}_w^\top \mathbf{g}_i + \max_{j \ne i} \Big( \sum_{w \in \text{landmark}} \mathbf{u}_w^\top \mathbf{g}_j + \mathbf{v}_r^\top \mathbf{h}_{ij} \Big), \qquad p(i \mid r) = \frac{e^{s_i}}{\sum_k e^{s_k}},$$

with $\mathbf{g}_i$ proposal $i$’s pixel features, $\mathbf{h}_{ij} = (\tanh(\Delta x/8), \tanh(\Delta y/8))$ its offset from proposal $j$, and the second term only for relational expressions. The max mirrors MAttNet’s relationship module; the cross-entropy gradient flows only into the chosen landmark and is checked by finite differences. A pick is correct at IoU $\ge 0.5$. It runs in about 18 seconds with NumPy 2.3.

import numpy as np

S, P = 64, 6                                         # image size; at most 6 objects per image
COLORS = {"red": (1.0, 0.2, 0.2), "green": (0.2, 0.9, 0.2), "blue": (0.25, 0.35, 1.0)}
SHAPES = ("circle", "square", "triangle")
ATTR = list(COLORS) + list(SHAPES)                   # the 6 attribute words
RELS = ("left of", "right of", "above", "below")
yy, xx = np.mgrid[0:S, 0:S] + 0.5                    # pixel centers; a box's max is exclusive

def render(objs, rng):
    """Colored shapes on a dark, noisy 64x64 RGB image; returns the image and tight boxes."""
    img = 0.1 + 0.05 * rng.standard_normal((S, S, 3))
    boxes = []
    for color, shape, cx, cy, r in objs:
        x, y = xx - cx, yy - cy
        if shape == "circle":
            m = x**2 + y**2 < r**2
        elif shape == "square":
            m = (abs(x) < 0.85 * r) & (abs(y) < 0.85 * r)
        else:                                        # triangle, apex up
            m = (y < 0.8 * r) & (y > -r + 2 * abs(x))
        img[m] = np.array(COLORS[color]) * rng.uniform(0.7, 1.0)
        rows, cols = np.nonzero(m)
        boxes.append([cols.min(), rows.min(), cols.max() + 1, rows.max() + 1])
    return img, np.array(boxes, float)

def make_scene(rng):
    """4-6 non-overlapping shapes, at least two of the same shape."""
    while True:
        n, objs = rng.integers(4, P + 1), []
        while len(objs) < n:
            cx, cy = rng.uniform(9, S - 9, size=2)
            if all((cx - o[2])**2 + (cy - o[3])**2 > 15**2 for o in objs):
                objs.append((rng.choice(list(COLORS)), rng.choice(SHAPES), cx, cy, rng.uniform(4.5, 6.5)))
        if len({o[1] for o in objs}) < n:
            return objs

def holds(rel, a, b, margin=4.0):
    """Does object a stand in relation rel to object b? (y grows downward)"""
    dx, dy = a[2] - b[2], a[3] - b[3]
    return {"left of": dx < -margin, "right of": dx > margin, "above": dy < -margin, "below": dy > margin}[rel]

def expressions(objs, rng):
    """Invented referring expressions, each true of exactly one object:
    (target, target words, relation or None, landmark words)."""
    out, key = [], [(o[0], o[1]) for o in objs]
    shapes = [k[1] for k in key]
    for t, (c, s) in enumerate(key):
        if shapes.count(s) == 1:
            out.append((t, [s], None, []))                       # "the square"
        elif key.count((c, s)) == 1:
            out.append((t, [c, s], None, []))                    # "the red square"
        if shapes.count(s) < 2:
            continue
        rivals = [i for i in range(len(objs)) if shapes[i] == s]
        for l in rng.permutation(len(objs)):                      # "the square left of the blue circle"
            if shapes[l] == s or key.count(key[l]) > 1:
                continue
            ok = [r for r in RELS if holds(r, objs[t], objs[l]) and sum(holds(r, objs[i], objs[l]) for i in rivals) == 1]
            if ok:
                out.append((t, [s], ok[rng.integers(len(ok))], list(key[l])))
                break
    return out

def say(tw, rel, lw):
    return "the " + " ".join(tw) + (f" {rel} the " + " ".join(lw) if rel else "")

def iou(b, B):
    w = np.clip(np.minimum(b[2], B[:, 2]) - np.maximum(b[0], B[:, 0]), 0, None)
    h = np.clip(np.minimum(b[3], B[:, 3]) - np.maximum(b[1], B[:, 1]), 0, None)
    return w * h / ((b[2] - b[0]) * (b[3] - b[1]) + (B[:, 2] - B[:, 0]) * (B[:, 3] - B[:, 1]) - w * h)

def box_features(img, b):
    """Read a proposal's pixels: mean R, G, B of the foreground, fill, top-half share, corner fill, 1."""
    x1, y1, x2, y2 = b.astype(int)
    crop = img[y1:y2, x1:x2]
    fg = crop.max(2) > 0.45
    h, w = fg.shape
    ch, cw = max(h // 3, 1), max(w // 3, 1)
    corners = np.mean([fg[:ch, :cw].mean(), fg[:ch, -cw:].mean(), fg[-ch:, :cw].mean(), fg[-ch:, -cw:].mean()])
    return np.array([*crop[fg].mean(0), fg.mean(), fg[: h // 2].sum() / fg.sum(), corners, 1.0])

def build(n_scenes, rng, jitter):
    """Proposals are the true boxes with each corner moved by up to +-jitter pixels (a stand-in
    for a detector). Each expression becomes fixed-size arrays over its image's proposals."""
    D = {k: [] for k in ("G", "H", "T", "L", "R", "rel", "gt", "box", "props", "valid", "text", "rivals")}
    for _ in range(n_scenes):
        objs = make_scene(rng)
        img, gt = render(objs, rng)
        n = len(objs)
        prop = np.clip(np.round(gt + rng.uniform(-jitter, jitter, gt.shape)), 0, S)
        g, pb, c = np.zeros((P, 7)), np.zeros((P, 4)), np.zeros((P, 2))
        g[:n] = [box_features(img, b) for b in prop]
        pb[:n] = prop
        c[:n] = np.c_[prop[:, [0, 2]].mean(1), prop[:, [1, 3]].mean(1)]
        h = np.tanh((c[:, None, :] - c[None, :, :]) / 8.0)        # h[i, j] = squashed (dx, dy) of i from j
        for t, tw, r, lw in expressions(objs, rng):
            o = iou(gt[t], prop)
            for k, v in (("G", g), ("H", h), ("box", gt[t]), ("props", pb), ("valid", np.arange(P) < n),
                         ("T", [tw.count(w) for w in ATTR]), ("L", [lw.count(w) for w in ATTR]),
                         ("R", [r == q for q in RELS]), ("rel", r is not None),
                         ("gt", o.argmax() if o.max() >= 0.5 else -1), ("text", say(tw, r, lw)),
                         ("rivals", sum(q[1] == objs[t][1] for q in objs))):
                D[k].append(v)
    D = {k: np.array(v) for k, v in D.items()}
    for k in ("T", "L", "R", "rel"):
        D[k] = D[k].astype(float)
    return D

# ---------- heuristic grounder: hand-written rules ----------
def read_attributes(g):
    """Rules set on tight boxes, where they read every shape correctly."""
    color = list(COLORS)[int(np.argmax(g[:3]))]
    shape = "square" if g[5] > 0.7 else "triangle" if g[4] < 0.35 else "circle"
    return color, shape

def heuristic(D, rng, use_relations):
    picks = []
    for e in range(len(D["T"])):
        n = int(D["valid"][e].sum())
        attrs = [read_attributes(D["G"][e, i]) for i in range(n)]
        match = lambda words: np.array([sum(ATTR[k] in a for k in np.flatnonzero(words)) for a in attrs], float)
        score = match(D["T"][e])
        if use_relations and D["rel"][e]:
            ls = match(D["L"][e])
            lm = rng.choice(np.flatnonzero(ls == ls.max()))          # the landmark
            dx, dy = D["H"][e, :n, lm].T
            d = {"left of": -dx, "right of": dx, "above": -dy, "below": dy}[RELS[int(D["R"][e].argmax())]]
            score += (0.5 * (d > 0) + 0.1 * d) * (np.arange(n) != lm)   # on the named side; the farther the better
        picks.append(rng.choice(np.flatnonzero(score == score.max())))   # ties broken at random
    return np.array(picks)

# ---------- learned grounder: multinomial logistic regression over the proposals ----------
def scores(U, V, D):
    """s_i = sum_w U_w . g_i  +  [relational] max_{j != i} ( sum_w' U_w' . g_j + V_r . h_ij )"""
    At = np.einsum("ew,wd,epd->ep", D["T"], U, D["G"])           # target words vs proposal i
    Al = np.einsum("ew,wd,epd->ep", D["L"], U, D["G"])           # landmark words vs proposal j
    M = Al[:, None, :] + np.einsum("er,rk,eijk->eij", D["R"], V, D["H"])
    M = np.where(np.eye(P, dtype=bool) | ~D["valid"][:, None, :], -np.inf, M)
    jstar = M.argmax(2)                                           # best landmark for each candidate
    m = np.take_along_axis(M, jstar[..., None], 2)[..., 0]
    s = np.where(D["valid"], At + np.where(D["rel"][:, None] > 0, m, 0.0), -np.inf)
    return s, jstar

def loss_grad(U, V, D, lam=1e-3):
    k = D["gt"] >= 0                                              # expressions with a correct proposal
    s, jstar = scores(U, V, D)
    s, jstar, gt = s[k], jstar[k], D["gt"][k]
    T, L, R, G, H, rel = (D[x][k] for x in ("T", "L", "R", "G", "H", "rel"))
    n, rows = len(s), np.arange(len(s))
    p = np.exp(s - s.max(1, keepdims=True))
    p /= p.sum(1, keepdims=True)
    loss = -np.log(p[rows, gt]).mean() + lam * ((U**2).sum() + (V**2).sum())
    dS = p.copy()
    dS[rows, gt] -= 1
    dS /= n                                                       # d loss / d s
    dm = dS * rel[:, None]                                        # flows through the chosen landmark only
    dAl, dM = np.zeros_like(dm), np.zeros((n, P, P))
    np.add.at(dAl, (rows.repeat(P), jstar.ravel()), dm.ravel())
    dM[rows[:, None], np.arange(P)[None, :], jstar] = dm
    dU = np.einsum("ep,ew,epd->wd", dS, T, G) + np.einsum("ej,ew,ejd->wd", dAl, L, G)
    dV = np.einsum("eij,er,eijk->rk", dM, R, H)
    return loss, dU + 2 * lam * U, dV + 2 * lam * V

def train(D, rng, steps=200, lr=0.05):
    U, V = 0.01 * rng.standard_normal((len(ATTR), 7)), 0.01 * rng.standard_normal((len(RELS), 2))
    mom, vel = [0 * U, 0 * V], [0 * U, 0 * V]
    for step in range(1, steps + 1):                              # Adam
        loss, dU, dV = loss_grad(U, V, D)
        for k, (W, g) in enumerate(((U, dU), (V, dV))):
            mom[k] = 0.9 * mom[k] + 0.1 * g
            vel[k] = 0.999 * vel[k] + 0.001 * g**2
            W -= lr * (mom[k] / (1 - 0.9**step)) / (np.sqrt(vel[k] / (1 - 0.999**step)) + 1e-8)
    return U, V, loss

def grad_check(U, V, D):
    _, dU, dV = loss_grad(U, V, D)
    err = 0.0
    for W, dW in ((U, dU), (V, dV)):
        for idx in np.ndindex(W.shape):
            old = W[idx]
            W[idx] = old + 1e-6; lp = loss_grad(U, V, D)[0]
            W[idx] = old - 1e-6; lm = loss_grad(U, V, D)[0]
            W[idx] = old
            err = max(err, abs((lp - lm) / 2e-6 - dW[idx]))
    return err

def hits(picks, D):
    return np.array([iou(D["box"][e], D["props"][e, [p]])[0] >= 0.5 for e, p in enumerate(picks)])

def run(seed, jitter, verbose=False):
    rng = np.random.default_rng(seed)
    tr, te = build(250, rng, jitter), build(250, rng, jitter)
    U, V, loss = train(tr, rng)
    picks = {"attributes only": heuristic(te, rng, False),
             "attributes + relation rule": heuristic(te, rng, True),
             "learned": scores(U, V, te)[0].argmax(1)}
    res = {name: hits(p, te) for name, p in picks.items()}
    res["proposal ceiling"] = te["gt"] >= 0
    if verbose:
        print(f"max |analytic - finite-difference| gradient over all {U.size + V.size} weights: {grad_check(U, V, tr):.1e}")
        print(f"training loss {loss:.3f}; test set: {int((te['rel'] == 0).sum())} attribute and {int(te['rel'].sum())} relational expressions")
        print(f"guessing among same-shaped objects would find {np.mean(1 / te['rivals'][te['rel'] == 1]):.2f} of the relational targets")
        print("learned relation weights on (dx, dy):", ", ".join(f"{r} ({a:+.1f}, {b:+.1f})" for r, (a, b) in zip(RELS, V)))
        for e in range(4):
            print(f"  '{te['text'][e]}': rules {'hit' if res['attributes + relation rule'][e] else 'miss'}, "
                  f"learned {'hit' if res['learned'][e] else 'miss'}")
    return {name: (h[te["rel"] == 0].mean(), h[te["rel"] == 1].mean()) for name, h in res.items()}

if __name__ == "__main__":
    print("seed 0, corners jittered by up to 1 px")
    run(0, 1.0, verbose=True)
    for jitter in (0.0, 1.0, 3.0):
        print(f"accuracy at IoU >= 0.5, seeds 0-4, corners jittered by up to {jitter:.0f} px (min-max over seeds)")
        allres = [run(seed, jitter) for seed in range(5)]
        for name in allres[0]:
            a, b = zip(*(r[name] for r in allres))
            print(f"  {name:27s} attribute {min(a):.2f}-{max(a):.2f}   relational {min(b):.2f}-{max(b):.2f}")

seed 0, corners jittered by up to 1 px
max |analytic - finite-difference| gradient over all 50 weights: 9.9e-11
training loss 0.513; test set: 773 attribute and 529 relational expressions
guessing among same-shaped objects would find 0.43 of the relational targets
learned relation weights on (dx, dy): left of (-2.3, +0.0), right of (+2.4, +0.1), above (-0.1, -2.5), below (-0.0, +2.3)
  'the red circle': rules hit, learned hit
  'the circle right of the green square': rules hit, learned miss
  'the green circle': rules hit, learned hit
  'the blue circle': rules hit, learned hit
accuracy at IoU >= 0.5, seeds 0-4, corners jittered by up to 0 px (min-max over seeds)
  attributes only             attribute 1.00-1.00   relational 0.42-0.48
  attributes + relation rule  attribute 1.00-1.00   relational 1.00-1.00
  learned                     attribute 1.00-1.00   relational 0.99-1.00
  proposal ceiling            attribute 1.00-1.00   relational 1.00-1.00
accuracy at IoU >= 0.5, seeds 0-4, corners jittered by up to 1 px (min-max over seeds)
  attributes only             attribute 0.85-0.88   relational 0.39-0.41
  attributes + relation rule  attribute 0.85-0.89   relational 0.85-0.88
  learned                     attribute 0.96-0.97   relational 0.93-0.94
  proposal ceiling            attribute 1.00-1.00   relational 1.00-1.00
accuracy at IoU >= 0.5, seeds 0-4, corners jittered by up to 3 px (min-max over seeds)
  attributes only             attribute 0.42-0.44   relational 0.20-0.21
  attributes + relation rule  attribute 0.40-0.43   relational 0.40-0.45
  learned                     attribute 0.54-0.56   relational 0.52-0.57
  proposal ceiling            attribute 0.69-0.71   relational 0.68-0.74

With tight boxes the rules read every attribute correctly, yet ignoring the relation leaves 42–48% on relational expressions, near the 0.43 expected (seed 0) from guessing among same-shaped objects; the relation rule reaches 100%. The learned scorer recovers that rule: its relation weights point as the words say. At 1 pixel of jitter the rules fall to 85–89% on attribute expressions; by our reasoning, a box a pixel off pushes the corner and top-half fractions across thresholds set on tight boxes. The learned scorer, fit to the boxes it actually sees, keeps 93–97%. At 3 pixels only 68–74% of targets have any proposal at IoU $\ge 0.5$, and every method falls below that ceiling, the first-stage cap that one-stage methods remove.

Datasets and Benchmarks

• ReferItGame (Kazemzadeh et al., 2014): a two-player game, one player describing an outlined object and the other clicking on it; 130,525 expressions for 96,654 objects in 19,894 photographs from the ImageCLEF IAPR collection.
• RefCOCO and RefCOCO+ (Yu et al., 2016): the same game on COCO images with two or more objects of a category. RefCOCO has 142,209 expressions for 50,000 objects in 19,994 images; RefCOCO+ (141,564 expressions, 49,856 objects) bans location words to focus on appearance. Test sets are split into testA (images with several people) and testB (several objects of other categories).
• RefCOCOg (Mao et al., 2016, “Google Refexp”): written without a game, with longer expressions: 8.43 words on average against 3.61 for RefCOCO (Yu et al., 2016). Mao et al. report 104,560 expressions for 54,822 objects in 26,711 images; Yu et al. used a version with 85,474. The original google split divides objects, so an image can appear in training and validation, and has no released test set (Yu et al., 2018); the umd split of Nagaraja, Morariu and Davis (2016) divides images, and the standard loader recommends it (Yu et al., refer README).
• Flickr30k Entities (Plummer et al., 2015): the 158,915 captions of 31,783 Flickr30k images, with 244,035 coreference chains linking mentions of the same entity and 275,775 boxes; the standard phrase grounding benchmark.
• Visual Genome (Krishna et al., 2017): over 100,000 images with on average 42 region descriptions each, a common source of grounding pretraining data.

Evaluation Metrics

REC is scored as top-1 accuracy at IoU 0.5 (Mao et al. called it precision@1). Phrase grounding reports Recall@K (see precision and recall), the share of phrases with a correct box among the top $K$ (Plummer et al., 2015). For a phrase with several ground-truth boxes, the any-box protocol accepts any of them and the merged-boxes protocol uses their smallest enclosing box; MDETR reported both, naming them in its arXiv appendix (Kamath et al., 2021). Referring segmentation reports mask IoU and precision at several IoU thresholds (Yu et al., 2018).

Reported results

Top-1 accuracy (%) as printed in each paper:

Method	Boxes or pretraining	RefCOCO val / testA / testB	RefCOCO+ val / testA / testB	RefCOCOg val / test
MAttNet (2018)	ground-truth COCO boxes, Faster R-CNN features	85.65 / 85.26 / 84.57	71.01 / 75.13 / 66.17	78.10 / 78.12
MAttNet (2018)	Mask R-CNN detections and features	76.65 / 81.14 / 69.99	65.33 / 71.62 / 56.02	66.58 / 67.27
TransVG, ResNet-101 (2021)	DETR and BERT initialization	81.02 / 82.72 / 78.35	64.82 / 70.70 / 56.94	68.67 / 67.73
MDETR, EfficientNet-B3 (2021)	1.3M grounded pairs	87.51 / 90.40 / 82.67	81.13 / 85.52 / 72.96	83.35 / 83.31
Shikra-7B (2023)	language model, coordinates as text	87.01 / 90.61 / 80.24	81.60 / 87.36 / 72.12	82.27 / 82.19
Grounding DINO T, no REC data (2024)	O365, GoldG	50.41 / 57.24 / 43.21	51.40 / 57.59 / 45.81	67.46 / 67.13
Grounding DINO L, fine-tuned (2024)	O365, OI, GoldG, Cap4M, COCO, RefC	90.56 / 93.19 / 88.24	82.75 / 88.95 / 75.92	86.13 / 87.02

RefCOCOg is the umd split for MAttNet and TransVG; the other papers do not name it, though a test score implies umd, since the google test set is unreleased (our reading). Grounding DINO’s authors flag a possible leak for their large model, whose COCO training data include RefCOCO validation images. Without REC training data their model and GLIP lag far behind (Liu et al., 2024); the arXiv appendix attributes the gap to the format: a RefCOCO expression has one box, while the model tends to predict every matching object. Kosmos-2’s zero-shot results are similar (52.32 on RefCOCO val; Peng et al., 2023). In every row testB scores below testA, and every model trained with RefCOCO data scores lower on RefCOCO+ than on RefCOCO.

Applications

The dataset papers name robots, human–computer interaction and image search (Kazemzadeh et al., 2014; Yu et al., 2016). Grounding with segmentation drives text-based image editing and automatic annotation in the Grounded SAM pipelines (Ren et al., 2024), and grounding models pseudo-label web data for detection pretraining (Li et al., 2022).

Open Problems

• Shortcuts. On RefCOCOg (umd split, ground-truth boxes), the CMN model trained without expressions ranked the target first 41.1% of the time, against 20.4% for random choice and 70.5% for full CMN; shuffling the words cost CMN 3 points (Cirik et al., 2018). The authors trace the bias to data selection: targets came only from categories with several instances in an image, so 43 of 90 COCO categories are targets less than 1% of the times they appear.
• Relations. Keeping only nouns and adjectives, which removes most relation words, cost the two relation-modeling systems Cirik et al. tested only 1.8 and 4.0 points; whether models use relations is still measured indirectly.
• Beyond one object. REC assumes exactly one target; real queries may refer to several objects or none.
• Evaluation hygiene. A detector pretrained on grounding test images leaks them, as MDETR’s authors note for methods built on the bottom-up attention detector (Kamath et al., 2021).