Grounding DINO

Grounding DINO is an open-set object detector: given an image and a text prompt, it returns boxes for the objects the text names, whether the prompt is a list of categories (“cat . person . mouse .”) or a referring expression (“the left lion”). Shilong Liu and eleven co-authors from Tsinghua University, the International Digital Economy Academy (IDEA) and other institutions released it on arXiv in March 2023 and published it at ECCV 2024. It is a transformer detector in the DETR family that adds a BERT text encoder and fuses language with vision at three points.

“DINO” here is the detector DINO: DETR with Improved DeNoising Anchor Boxes (Zhang et al., 2023), a DETR variant with contrastive denoising, mixed query selection and “look forward twice” box refinement, not the self-supervised DINO of Caron et al. (2021) covered under self-supervised learning.

Problem Addressed

A closed-set detector classifies boxes into a fixed list of categories. Open-set detection (the authors treat open-set, open-world and open-vocabulary detection as one task) replaces the classifier with a comparison between region and text features, so a category can be named at test time. ViLD and OV-DETR borrowed semantics from CLIP, which, citing RegionCLIP, the authors argue transfers poorly from images to regions. GLIP (Li et al., 2022) instead recast detection as phrase grounding, scoring each region against each word of a prompt of concatenated category names, so detection, grounding and pseudo-labeled caption data train one model.

Liu et al. divide a detector into a neck, query initialization and a head, and note that GLIP fuses image and text only in the neck and OV-DETR only in the decoder queries. They argue that fusing in all three aligns the modalities better, and that a DETR-style detector, built from layers like a language transformer’s, makes this easy. They also evaluate more strictly than the base/novel splits of earlier open-vocabulary work: “zero-shot” means the training split of the test dataset is never used.

Architecture

Image and text backbones produce “vanilla” features; a feature enhancer fuses them; language-guided query selection picks decoder queries from the fused image features; a cross-modality decoder refines them with both modalities; and the output queries predict boxes scored against the text tokens.

Image and Text Backbones

The image backbone is a Swin Transformer, Swin-T for Grounding DINO T and Swin-L for Grounding DINO L (Liu et al., 2021), giving multi-scale features as in Deformable DETR: three scales from 8× to 32× downsampling plus an extra 64× map for T, four scales from 4× to 32× for L. The text backbone is BERT-base (Devlin et al., 2019), with at most 256 text tokens. Features are projected to $d = 256$.

Feature Enhancer

Each of six enhancer layers applies deformable self-attention (Zhu et al., 2021) to the image tokens, ordinary self-attention to the text tokens, text-to-image and image-to-text cross-attention (bidirectional cross-attention fusion inspired by GLIP), and feed-forward layers.

Language-Guided Query Selection

With image features $X_I \in \mathbb{R}^{N_I \times d}$ (typically more than 10,000 tokens) and text features $X_T \in \mathbb{R}^{N_T \times d}$ ($N_T \le 256$), the indices of the $N_q$ queries are

$$\mathcal{I}_{N_q} = \operatorname{Top}_{N_q}\!\Big( \max_{j} \big( X_I X_T^{\top} \big)_{:,j} \Big),$$

the image tokens whose best-matching text token is most similar, with $N_q = 900$ as in DINO. Following DINO’s mixed query selection, the selected tokens initialize only the positional part of each query, as dynamic anchor boxes; the content part remains a learned embedding. The arXiv appendix shows the selected boxes on one image changing with the prompt (“building”, “river”, “ferris wheel”).

The script below shows the mechanism on a synthetic $128 \times 128$ image of five colored shapes, split into $4 \times 4$ patches (1,024 tokens). In place of a trained backbone and enhancer, each patch feature is the sum of invented embeddings for the color and shape of the object covering it, weighted by coverage, plus noise. The prompt is “red circle . blue square .”; the image also contains a red square and a blue circle, each sharing one word with the prompt. Text tokens are either their word embeddings alone or each word mixed with its phrase, a stand-in for text self-attention restricted to each phrase (see below).

import numpy as np

S, PATCH, d, K = 128, 4, 64, 32               # 128x128 image, 4x4 patches -> 32x32 = 1,024 tokens; keep K
G = S // PATCH
yy, xx = np.mgrid[0:S, 0:S] + 0.5

# Five shapes: two targets, two distractors that share one word each, and an unrelated object.
OBJECTS = [("red", "circle", 34, 34), ("blue", "square", 94, 30), ("red", "square", 32, 94),
           ("blue", "circle", 94, 94), ("green", "triangle", 64, 64)]
RGB = {"red": (0.9, 0.1, 0.1), "blue": (0.1, 0.2, 0.9), "green": (0.1, 0.8, 0.2)}

def shape_mask(shape, cx, cy, r=11):
    x, y = xx - cx, yy - cy
    if shape == "circle":
        return x**2 + y**2 < r**2
    if shape == "square":
        return (abs(x) < 0.9 * r) & (abs(y) < 0.9 * r)
    return (y < 0.8 * r) & (y > -r + 2 * abs(x))   # triangle, apex up

image = np.full((S, S, 3), 0.1)
masks = [shape_mask(s, cx, cy) for _, s, cx, cy in OBJECTS]
for (color, *_), m in zip(OBJECTS, masks):
    image[m] = RGB[color]
cover = np.stack([m.reshape(G, PATCH, G, PATCH).mean((1, 3)) for m in masks])   # (5, 32, 32)
owner = np.where(cover.max(0) > 0.5, cover.argmax(0), len(OBJECTS)).ravel()      # 5 = background

tokens = ["[CLS]", "red", "circle", ".", "blue", "square", ".", "[SEP]"]          # "red circle . blue square ."
phrase = [None, 0, 0, None, 1, 1, None, None]

def run(seed):
    rng = np.random.default_rng(seed)
    vocab = ["red", "blue", "green", "circle", "square", "triangle", "background", ".", "[CLS]", "[SEP]"]
    E = {w: (v := rng.standard_normal(d)) / np.linalg.norm(v) for w in vocab}      # unit-norm embeddings
    # Stand-in for backbone + feature enhancer: a patch's feature is the color and shape embeddings
    # of the object covering it, weighted by coverage, background elsewhere, plus noise.
    obj = np.array([E[c] + E[s] for c, s, _, _ in OBJECTS])
    X_I = np.einsum("ohw,od->hwd", cover, obj) + (1 - cover.sum(0))[..., None] * E["background"]
    X_I = (X_I + 0.05 * rng.standard_normal((G, G, d))).reshape(G * G, d)          # N_I x d
    out = {}
    for contextual in (False, True):
        X_T = np.array([E[t] for t in tokens])                                     # N_T x d
        if contextual:      # each word mixed with its own phrase, as masked text self-attention would
            for p in (0, 1):
                idx = [i for i, q in enumerate(phrase) if q == p]
                X_T[idx] = 0.5 * X_T[idx] + 0.5 * X_T[idx].mean(0)
        top = np.argsort(-(X_I @ X_T.T).max(-1))[:K]                               # Top_K(Max_{-1}(X_I X_T^T))
        out[contextual] = np.bincount(owner[top], minlength=len(OBJECTS) + 1)
    return out

names = [f"{c} {s}" for c, s, _, _ in OBJECTS] + ["background"]
print("patches owned:", ", ".join(f"{n} {int((owner == o).sum())}" for o, n in enumerate(names)))
res = run(0)
for contextual, label in ((False, "word-only text features"), (True, "phrase-contextual features")):
    print(f"seed 0, {label}, top {K}:", ", ".join(f"{n} {c}" for n, c in zip(names, res[contextual])))
hits = {False: [], True: []}
for seed in range(20):
    r = run(seed)
    for c in hits:
        hits[c].append(r[c][:2].sum())
for c, label in ((False, "word-only"), (True, "phrase-contextual")):
    print(f"seeds 0-19, {label}: {min(hits[c])}-{max(hits[c])} of {K} selected on the two named objects "
          f"(mean {np.mean(hits[c]):.1f})")

patches owned: red circle 21, blue square 25, red square 20, blue circle 21, green triangle 12, background 925
seed 0, word-only text features, top 32: red circle 0, blue square 15, red square 0, blue circle 17, green triangle 0, background 0
seed 0, phrase-contextual features, top 32: red circle 8, blue square 24, red square 0, blue circle 0, green triangle 0, background 0
seeds 0-19, word-only: 0-32 of 32 selected on the two named objects (mean 15.4)
seeds 0-19, phrase-contextual: 21-32 of 32 selected on the two named objects (mean 30.5)

For seed 0, no selected token falls on the background or the green triangle either way. With word-only features, the blue circle matches “blue” as well as the blue square does, so noise decides how the selection splits between targets and distractors (0 to 32 of 32 on the targets over 20 seeds). When each word carries its phrase, only the named color–shape pairs match fully, and 30.5 of 32 selected tokens land on the targets on average. Seed 0 also shows one object taking most of a small budget; at full scale, the arXiv appendix reports that 900 queries already cover the objects in COCO and LVIS, with 1,200 or 1,500 queries adding little.

Cross-Modality Decoder

Each of the six decoder layers applies self-attention among the queries, image cross-attention (deformable, as in DINO), a text cross-attention layer that DINO’s decoder lacks, and a feed-forward network. Auxiliary losses are applied after every decoder layer and after the encoder outputs, as in other DETR-like detectors.

Sub-Sentence Text Features

In a concatenated prompt, BERT’s self-attention lets unrelated names influence each other, and the influence changes with their arbitrary order. Encoding each phrase into one feature (sentence level) avoids this but loses per-word features. Grounding DINO keeps per-word features and adds an attention mask that blocks attention between different category names. In the official code the mask is applied inside BERT: tokens attend only within their own name up to and including its “.”, position indices restart for each name, and [CLS] and [SEP] attend only to themselves.

The script below builds that mask for a one-layer toy text encoder with random weights and invented word embeddings. Three stand-in decoder queries (a red circle, a blue square, nothing) are scored against the text tokens as in the classification loss below; a phrase scores its best token’s probability.

import numpy as np

rng = np.random.default_rng(0)
d = 256                                                                 # feature dimension, as in the model
vocab = ["[CLS]", "[SEP]", ".", "red", "blue", "green", "circle", "square", "triangle"]
E = {w: (v := rng.standard_normal(d)) / np.linalg.norm(v) for w in vocab}   # invented word embeddings
P = rng.standard_normal((16, d)) / np.sqrt(d) * 0.3                       # position embeddings
Wq, Wk = (rng.standard_normal((d, d)) / np.sqrt(d) for _ in range(2))
SPECIAL = {"[CLS]", "[SEP]", "."}

def tokenize(categories):
    return ["[CLS]"] + [t for c in categories for t in c.split() + ["."]] + ["[SEP]"]

def masks(tokens, sub_sentence):
    """Full attention, or the sub-sentence mask of the official code: tokens attend only within
    their own category name (up to and including its '.'), positions restart for each name,
    and [CLS] and [SEP] attend only to themselves."""
    T = len(tokens)
    if not sub_sentence:
        return np.ones((T, T), bool), np.arange(T)
    allow, pos, prev = np.eye(T, dtype=bool), np.zeros(T, int), 0
    for col in [i for i, t in enumerate(tokens) if t in SPECIAL]:
        if 0 < col < T - 1:
            allow[prev + 1:col + 1, prev + 1:col + 1] = True
            pos[prev + 1:col + 1] = np.arange(col - prev)
        prev = col
    return allow, pos

def encode(tokens, sub_sentence, return_attention=False):
    """One toy text self-attention layer with a residual connection."""
    allow, pos = masks(tokens, sub_sentence)
    X = np.array([E[t] for t in tokens]) + P[pos]
    A = (X @ Wq) @ (X @ Wk).T / np.sqrt(d)
    A = np.where(allow, A, -np.inf)
    A = np.exp(A - A.max(1, keepdims=True))
    A /= A.sum(1, keepdims=True)
    return (X + A @ X, A) if return_attention else X + A @ X

cats = ["red circle", "blue square"]
tokens = tokenize(cats)
print("tokens:", " ".join(tokens))
# Decoder outputs stand-in: queries that have found a red circle, a blue square, and background.
Q = np.array([E["red"] + E["circle"], E["blue"] + E["square"], np.zeros(d)])
Q += 0.05 * rng.standard_normal(Q.shape)
span = {c: [i for i, t in enumerate(tokens) if t in c.split() and
            sum(tokens[j] == "." for j in range(i)) == k] for k, c in enumerate(cats)}

for sub in (False, True):
    H, A = encode(tokens, sub, return_attention=True)
    logits = Q @ H.T                                     # contrastive logits: N queries x N_T tokens
    prob = 1 / (1 + np.exp(-logits))
    print("sub-sentence mask" if sub else "full attention   ")
    for q, name in enumerate(["red-circle query", "blue-square query", "background query"]):
        print(f"   {name:18s}", "  ".join(f"{c}: {prob[q, span[c]].max():.2f}" for c in cats))
    i = tokens.index("circle")
    h_circle = H[i]
    leak = [h_circle @ E[w] / np.linalg.norm(h_circle) for w in ("blue", "square")]
    print(f"   attention from 'circle' to the tokens of 'blue square .': {A[i, 4:7].sum():.2f}; "
          f"cosine of its feature with 'blue', 'square': {leak[0]:+.2f}, {leak[1]:+.2f}")
    # Does the feature of "circle" depend on the other categories and their order?
    other = tokenize(["green triangle", "blue square", "red circle"])
    H2 = encode(other, sub)
    change = np.linalg.norm(H2[other.index("circle")] - h_circle) / np.linalg.norm(h_circle)
    print(f"   change in 'circle' feature when the prompt is 'green triangle . blue square . red circle .': {change:.2f}")

tokens: [CLS] red circle . blue square . [SEP]
full attention   
   red-circle query   red circle: 0.76  blue square: 0.51
   blue-square query  red circle: 0.59  blue square: 0.78
   background query   red circle: 0.51  blue square: 0.51
   attention from 'circle' to the tokens of 'blue square .': 0.38; cosine of its feature with 'blue', 'square': +0.13, +0.14
   change in 'circle' feature when the prompt is 'green triangle . blue square . red circle .': 0.38
sub-sentence mask
   red-circle query   red circle: 0.83  blue square: 0.45
   blue-square query  red circle: 0.53  blue square: 0.84
   background query   red circle: 0.51  blue square: 0.51
   attention from 'circle' to the tokens of 'blue square .': 0.00; cosine of its feature with 'blue', 'square': +0.04, +0.01
   change in 'circle' feature when the prompt is 'green triangle . blue square . red circle .': 0.00

With full attention, “circle” spends 38% of its attention on “blue square .”, its feature moves toward “blue” and “square”, the blue-square query scores 0.59 against “red circle”, and reordering the prompt with an extra name changes the feature by 38% of its norm. With the mask that attention is zero, the cross-category score drops to 0.53, and the feature no longer depends on the other names or their order.

Training

Data. As in GLIP, detection data become grounding data by concatenating category names, sampled at random during training, into a prompt. Detection data are Objects365 (O365; about 600,000 images in v1 for the T model, 1.7M in v2 for L, per the arXiv appendix), OpenImages (OI) and COCO. Grounding data are GoldG and RefC (RefCOCO, RefCOCO+ and RefCOCOg), both as preprocessed by MDETR (Kamath et al., 2021); GLIP describes GoldG as 0.8M human-annotated grounding examples from Flickr30K, Visual Genome captions and GQA, with COCO images removed. Caption data are GLIP’s pseudo-labeled web pairs: Cap4M (4M pairs boxed by GLIP-T (C)) for the T model and, per the appendix, GLIP-L’s data for L, although the tables label both Cap4M.

Objective. For query $i$ and text token $j$, the classification logit is a dot product, $s_{ij} = \mathbf{q}_i^\top \mathbf{t}_j$, as in GLIP’s alignment scores. The target $y_{ij}$ is 1 when the ground-truth object matched to query $i$ is named by a phrase containing token $j$. Each logit gets a sigmoid focal loss (Lin et al., 2017), and boxes get DETR’s $\ell_1$ and generalized IoU losses:

$$\mathcal{L} = \lambda_{\text{cls}} \sum_{i,j} \operatorname{FL}\big(\sigma(s_{ij}), y_{ij}\big) + \sum_{i\ \text{matched}} \Big[ \lambda_{L1} \lVert b_i - \hat{b}_i \rVert_1 + \lambda_{\text{giou}} \big(1 - \operatorname{GIoU}(b_i, \hat{b}_i)\big) \Big].$$

The same three terms form the cost for the one-to-one Hungarian matching between predictions and ground truth. The main text gives cost weights 2.0, 5.0 and 2.0 and loss weights 1.0, 5.0 and 2.0 for classification, $\ell_1$ and GIoU; the hyperparameter table in the arXiv appendix swaps the two classification weights (cost 1.0, loss 2.0).

Procedure. AdamW with learning rate $10^{-4}$ ($10^{-5}$ for both backbones), weight decay $10^{-4}$ and gradient clipping at 0.1 (arXiv appendix). The T models were trained on 16 V100 GPUs with batch size 32, the L model on 64 A100 GPUs with batch size 64.

Inference

The README suggests separating category names with “.”, as in “chair . person . dog .”. The model outputs 900 boxes, each with a similarity to every text token. The official code keeps boxes whose highest similarity exceeds a box threshold and labels each with the tokens above a text threshold (0.35 and 0.25 in the README example); for referring expressions the authors take the top-scoring box. Grounding DINO T has 172M parameters and ran at 8.37 FPS against 6.11 for GLIP-T (232M parameters) in the arXiv appendix, which does not name the hardware.

Performance

Results the authors reported (AP on COCO val2017, LVIS minival and the 35-dataset ODinW benchmark):

Model	Pretraining data	COCO zero-shot	LVIS zero-shot (AP / AP$_r$)	ODinW zero-shot (mean AP)
GLIP-T (C)	O365, GoldG	46.7	24.9 / 17.7
GLIP-T	O365, GoldG, Cap4M	46.3	26.0 / 20.8	19.6
Grounding DINO T	O365, GoldG	48.1	25.6 / 14.4	20.0
Grounding DINO T	O365, GoldG, Cap4M	48.4	27.4 / 18.1	22.3
GLIP-L	FourODs, GoldG, Cap24M	49.8
Grounding DINO L	O365, OI, GoldG	52.5
Grounding DINO L	O365, OI, GoldG, Cap4M, COCO, RefC	(60.7, COCO seen)	33.9 / 22.2	26.1

GLIP-T’s COCO score is from the GLIP paper, whose LVIS scores match Grounding DINO’s table; all other values are from Grounding DINO’s tables. Fine-tuned on COCO, Grounding DINO L reached 62.6 AP on val2017 and 62.7 on test-dev (63.0 on both with 1.5× larger images). Because O365 nearly covers COCO’s categories, the authors call the COCO setting zero-shot domain transfer, not a test of unseen categories. DetCLIPv2, trained on more data, scored 40.4 on LVIS. On ODinW, Grounding DINO T with Cap4M tied GLIPv2-T’s mean AP (22.3) with a higher median (11.9 against 8.9).

On RefCOCO val, Grounding DINO T scored 50.41% top-1 accuracy without RefC data, 73.98 with RefC in pretraining and 89.19 after fine-tuning; visual grounding compares other methods.

Ablations with Swin-T on O365 (COCO zero-shot / LVIS zero-shot AP): the full model 46.7 / 16.1; without encoder fusion 45.8 / 13.1; static instead of language-guided query selection 46.3 / 13.6; without decoder text cross-attention 46.1 / 14.3; word-level instead of sub-sentence prompts 46.4 / 15.6. The text credits language-guided query selection with +3.0 LVIS AP, but the table shows 2.5; 3.0 is the encoder-fusion gap.

Strengths

• One model handles closed-set detection, open-set detection and referring expressions.
• Against GLIP-T it is smaller, faster and gained more from Cap4M (+1.8 LVIS AP against +1.1).
• It fine-tunes well: 52.1 LVIS AP from O365 and GoldG pretraining, against 50.7 for DetCLIPv2-T.

Limitations

• Rare categories. LVIS rare-category AP trails GLIP’s (14.4 against 17.7 without captions), possibly a limitation of DETR-like detectors, the authors suggest. Specialized categories remain hard: 0.36 AP on ODinW’s PlantDoc against 1.1 for GLIP (arXiv appendix).
• Referring expressions without REC data. Without RefC, the model tends to box every matching object while RefCOCO expects one, which the arXiv appendix gives as a cause of the low zero-shot accuracy.
• No masks. Unlike GLIPv2 it cannot segment, the authors note; they also report false positives.
• Prompt dependence. Wording matters; with a BERT encoder, GLIP found “person. bicycle. car. …” worked better than “Detect: person, bicycle, car, …”. Prompts are truncated at 256 tokens, and a word can split into several tokens.
• Leakage. The large model’s COCO training images include RefCOCO validation images, as the authors flag.

Historical Importance

Grounding DINO combined GLIP’s grounded pretraining with DINO, a closed-set detector that had set a COCO record, and showed that language can enter every stage of a DETR-style detector. Released with code and a checkpoint in March 2023, it became the text-to-box front end of pipelines such as Grounded SAM. Its REC evaluation exposed a large zero-shot gap for open-set detectors.

Variants

• Transfer from DINO. The arXiv appendix trains only the text and fusion blocks on top of a frozen, pretrained DINO, matching the from-scratch model on COCO zero-shot (46.5 against 46.7 AP with O365) and converging faster.
• Grounded SAM (Ren et al., 2024) passes Grounding DINO’s boxes to SAM as box prompts, giving text-prompted masks, and adds tagging, captioning and editing models.
• Grounding DINO 1.5 (Ren et al., 2024), from IDEA Research, describes a scaled-up Pro model with an enhanced backbone, trained on over 20 million images with grounding annotations, and an Edge model optimized for speed on edge devices.

Implementations

The official repository, IDEA-Research/GroundingDINO (Apache 2.0), provides inference code, demos with Stable Diffusion and GLIGEN, and two checkpoints: Swin-T trained on O365, GoldG and Cap4M (48.4 AP zero-shot on COCO, as in the paper), and Swin-B trained on data that include COCO, ODinW-35 and RefCOCO, so by our reasoning its scores on those benchmarks are not zero-shot. There is no Swin-L checkpoint. The configuration defaults to bert-base-uncased, 900 queries and sub-sentence masking. As of October 2026, the README’s to-do list still marked training code as unreleased.