SAM

SAM (Segment Anything Model) is a promptable image segmentation model: given an image and a prompt, such as a click, a box or a rough mask, it returns a mask of the object the prompt indicates, without a class label. Kirillov et al. at Meta AI (FAIR) released it in April 2023, presented it at ICCV 2023, and published with it the SA-1B dataset of 1.1 billion masks. A large Vision Transformer encodes the image once, and a small two-way transformer decoder turns each prompt into masks. Architecture details, most experiments and the responsible-AI analysis are in the ICCV supplement and the arXiv version, with the same numbers.

Problem Addressed

The authors wanted a foundation model for segmentation, applied to new image distributions and tasks by prompting rather than retraining. Two things were missing. Masks are not abundant on the web: the largest existing dataset, Open Images, had about 2.7 million masks. And interactive segmentation aims at a valid mask after enough user input, not a sensible answer to every single click.

They therefore defined the promptable segmentation task: return a valid mask for any prompt, whether foreground and background points, a box, a rough mask or, in initial experiments, free-form text. For an ambiguous prompt, valid means a reasonable mask for at least one of the objects it could refer to; the paper’s example is a point on a shirt, which may mean the shirt or the person wearing it. Other tasks are solved by composition, such as object detector boxes as prompts for instance segmentation.

Architecture

A heavy image encoder runs once per image; a prompt encoder and a lightweight mask decoder then run per prompt on the cached embedding, in about 50 ms on a CPU in a web browser, according to the paper.

Image Encoder

The encoder is a ViT-H/16 pretrained as a masked autoencoder (MAE; He et al., 2022) and adapted to high resolution following ViTDet, with $14 \times 14$ windowed attention and four global-attention blocks. The image is rescaled and padded to $1024 \times 1024$, and two convolutions reduce the $64 \times 64$ grid of patch features to a $256 \times 64 \times 64$ embedding. SAM has about 636M parameters with ViT-H, 308M with ViT-L and 91M with ViT-B.

Prompt Encoder

A point becomes a 256-dimensional token: the positional encoding of its location plus a learned foreground or background embedding. A box becomes two tokens, its top-left and bottom-right corners, each with its own learned embedding. The positional encoding is the random Fourier feature mapping of Tancik et al. (2020); in the official code, a location $\mathbf{p} \in [0, 1]^2$ is encoded as

$$\gamma(\mathbf{p}) = \big[\sin(2\pi\, \mathbf{B}^\top \tilde{\mathbf{p}}),\ \cos(2\pi\, \mathbf{B}^\top \tilde{\mathbf{p}})\big], \qquad \tilde{\mathbf{p}} = 2\mathbf{p} - 1, \qquad \mathbf{B} \in \mathbb{R}^{2 \times 128},\ B_{ij} \sim \mathcal{N}(0, 1),$$

with $\mathbf{B}$ fixed; since $\sin a \sin b + \cos a \cos b = \cos(a - b)$, the dot product of two encodings depends only on the offset between the locations. Text goes through CLIP’s text encoder. A mask prompt, given at $256 \times 256$, is downsampled by strided convolutions to the embedding’s size and added to it; without one, a learned “no mask” embedding is added.

Mask Decoder

The decoder modifies a transformer decoder, following DETR and MaskFormer. Learned output tokens, one for IoU prediction and four for masks, join the prompt tokens, and each of two layers performs self-attention among the tokens, cross-attention from the tokens to the image embedding, an MLP on each token, and cross-attention from the image embedding back to the tokens. Positional encodings are re-added to the image embedding, and the original prompt tokens to the tokens, at every attention layer.

Two $2 \times 2$, stride-2 transposed convolutions then upsample the image embedding to $32 \times 256 \times 256$, the tokens attend to it once more, and an MLP maps each mask token to a vector $\mathbf{w}_k$. Each mask is a per-pixel dot product with the upscaled embedding $\hat{E}$, so the mask tokens act as dynamic linear classifiers:

$$M_k(x, y) = \mathbf{w}_k^\top \hat{E}(x, y).$$

Ambiguity

A single output trained on ambiguous prompts averages the valid masks, so SAM predicts three masks per prompt, enough, the authors observe, for the common nesting of whole, part and subpart, and a head on the IoU token predicts each mask’s IoU so they can be ranked. With several prompts ambiguity is rare, so a fourth mask token produces the single output in that case.

Training

Losses. Each mask is supervised with focal loss (Lin et al., 2017) and Dice loss (Milletari et al., 2016; see Dice coefficient) in a 20:1 ratio, and only the best of the $K$ masks receives gradient:

$$\mathcal{L} = \min_{k}\Big[\, 20\, \mathcal{L}_{\text{focal}}(M_k, G) + \mathcal{L}_{\text{dice}}(M_k, G) \Big] + \frac{1}{K}\sum_{k=1}^{K} \big(\hat{s}_k - \operatorname{IoU}(M_k, G)\big)^2 ,$$

where $G$ is the ground-truth mask and $\hat{s}_k$ the predicted IoU. The paper gives the IoU term as a mean-squared error with weight 1.0 but not whether every score is supervised; the average over $k$ is our reading.

Simulated interaction. Each mask is trained over 11 rounds: a first prompt (a uniformly sampled point, or the mask’s bounding box with noise of standard deviation 10% of its side, at most 20 pixels), then 8 points sampled from the error region of the previous prediction, with the previous mask logits fed back as a mask prompt, and two rounds without new points in which the model refines its own output. The decoder’s small cost, under 1% of the encoder’s, makes this affordable.

Recipe. From the MAE ViT-H, the model was trained for 90k iterations (about 2 epochs of SA-1B) with AdamW, batches of 256 images and no data augmentation; the model card gives 256 A100 GPUs for 68 hours.

Data Engine

SA-1B was collected in three stages, each with the current SAM in the loop:

1. Assisted-manual. Annotators clicked points in a SAM-powered browser tool. As SAM was retrained six times, the time per mask fell from 34 to 14 seconds; 4.3M masks were collected on 120k images.
2. Semi-automatic. Confident masks were pre-filled and annotators labeled the remaining objects: 5.9M more masks on 180k images.
3. Fully automatic. The ambiguity-aware model was prompted with a $32 \times 32$ grid of points and zoomed-in crops; masks were kept if their predicted IoU was at least 0.88 and if they were stable, changing little when the logits were thresholded at $-1$ or $+1$, and duplicates were removed by non-maximum suppression.

SA-1B

The released dataset holds only the automatic masks: 1.1B masks, about 100 per image, on 11M licensed photographs, downsampled to a shorter side of 1,500 pixels, with faces and license plates blurred. Masks carry no category labels. In an audit of 500 images, 94% of automatic masks had an IoU above 90% with annotator-corrected versions. Training on the automatic masks alone cost about 0.5 mIoU against all three stages.

Responsible-AI analysis. From places named in image captions, the authors estimated that 2.8% of the images come from Africa and 0.9% from low-income countries, both underrepresented, as in COCO and Open Images (Europe: 49.8%). SAM’s mIoU on people was similar across perceived gender presentation, age group and skin tone, with overlapping 95% confidence intervals except older against middle-aged people; on clothing, it was higher for people perceived as presenting masculine.

Inference

By design, a single prompt yields three masks with predicted IoUs and several prompts one mask; the official predictor exposes this as a flag, upsamples the $256 \times 256$ logits to the image and thresholds them at 0. “Segment everything” mode prompts a grid of points and applies the data engine’s filters. The SAM 2 authors measured SAM (ViT-H) at 21.7 images per second on one A100 GPU, in bfloat16 with torch.compile and batches of 10 images.

Performance

All results are zero-shot: SAM was not trained on these benchmarks.

• Single point to mask. On 23 datasets, including egocentric, underwater, microscopy and X-ray images, SAM’s most confident mask had a higher mIoU (intersection over union) than RITM on 16, by up to about 47 points; with the best of its three masks chosen against the ground truth (the “oracle”), it was ahead on all 23. On a subset of the datasets, human raters consistently scored SAM’s masks above RITM’s.
• Edge detection (BSDS500): ODS 0.768 from Sobel-filtered mask maps, against 0.788 for HED, trained on BSDS500, and 0.600 for Canny.
• Object proposals (LVIS v1): 59.3 mask AR@1000, against 63.0 for a ViTDet-H detector.
• Instance segmentation with ViTDet-H boxes as prompts: 46.5 mask AP on COCO and 44.7 on LVIS v1, against 51.0 and 46.6 for ViTDet-H. In a human study on LVIS, raters nevertheless preferred SAM’s masks, which the authors take to suggest that ViTDet exploits annotation biases.
• Text to mask: qualitative results only, from a variant trained with CLIP image embeddings as prompts and given CLIP text embeddings at test time.

Strengths

• One model, many tasks, through prompts and composition.
• Ambiguity-aware output: several ranked masks.
• Amortized speed: one encoder pass serves every prompt.
• Open release: code and weights under Apache 2.0; SA-1B for research.

Limitations

SAM, its authors note, can miss fine structures, sometimes hallucinates small disconnected components, and draws boundaries less crisply than slower methods that “zoom in”. Dedicated interactive segmenters should win with many points, text prompts are exploratory, and it is unclear how to prompt SAM for semantic or panoptic segmentation. The pipeline is not real time with the ViT-H encoder, and domain-specific tools are expected to do better in their domains.

Historical Importance

SAM recast interactive segmentation in the foundation-model pattern: a general task, a model trained on it at scale, and a model-in-the-loop data engine yielding 400 times more masks than any earlier dataset.

Variants

• SAM 2 (Ravi et al., ICLR 2025) extends the task to video. An MAE-pretrained Hiera encoder processes frames one at a time, memory attention conditions each frame on a memory bank of past and prompted frames, and a new head predicts whether the object is visible. Its SA-V dataset has 50.9K videos with 35.5M masks. On SAM’s 23 datasets, SAM 2 (Hiera-B+) trained on SA-1B alone reached 58.9 one-click mIoU at 130.1 FPS, against 58.1 for SAM (ViT-H), measured as above; arXiv v2 reports the updated SAM 2.1 checkpoints.
• HQ-SAM (Ke et al., NeurIPS 2023) freezes SAM and adds a high-quality output token that also uses early ViT features, under 0.5% more parameters trained on 44K fine-grained masks, to sharpen boundaries and thin structures.
• MobileSAM (Zhang et al., 2023) distills the ViT-H encoder into a TinyViT of about 5M parameters on one GPU in under a day, reusing SAM’s decoder; the authors report about 10 ms per image on a GPU.
• EfficientSAM (Xiong et al., CVPR 2024) pretrains small ViTs to reconstruct SAM’s image features from masked inputs; with ViTDet boxes, EfficientSAM-S reached 44.4 mask AP on COCO against 46.5 for SAM.
• MedSAM (Ma et al., 2024) fine-tunes SAM ViT-B with box prompts on about 1.57 million medical image–mask pairs from 10 modalities, reporting accuracy on par with or better than modality-specific U-Net and DeepLabV3+ models on 86 internal and 60 external validation tasks.

Implementations

The official repository, facebookresearch/segment-anything, provides the three checkpoints under Apache 2.0, with SamPredictor for prompts and SamAutomaticMaskGenerator for segment-everything mode; SAM 2 is in facebookresearch/sam2.

A Tiny SAM-Like Model

The script below (run with PyTorch 2.14.1) builds SAM’s three parts at toy scale on $48 \times 48$ synthetic images of three gray shapes, the large one containing a small square, so that a click on the square has two valid answers. The “image encoder” is fixed (soft thresholds of the smoothed image plus a Fourier encoding of each pixel position), the click is encoded the same way, and the decoder has an IoU token and two mask tokens that attend to $12 \times 12$ pooled image tokens (without SAM’s image-to-token step). Training uses the 20:1 focal and Dice loss on the lowest-loss mask only, with one valid mask drawn at random as each click’s target.

import math
import numpy as np
import torch
from torch import nn
import torch.nn.functional as F

SEED, STEPS, BATCH = 2, 3000, 16
print(f"torch {torch.__version__}")
torch.set_num_threads(4)                # results depend slightly on the thread count
torch.manual_seed(SEED)
rng = np.random.default_rng(SEED)
S, D, K = 48, 32, 2                  # image size, embedding width, masks per prompt
yy, xx = np.mgrid[:S, :S]

def shape(kind, cy, cx, r):
    if kind == "disk":
        return (yy - cy) ** 2 + (xx - cx) ** 2 <= r * r
    return (abs(yy - cy) <= r) & (abs(xx - cx) <= r)

def make_image(rng):
    """Three objects with distinct gray levels on a noisy background; the first is large and
    contains a small square. Returns the image and (click region, valid masks) pairs."""
    img = np.full((S, S), rng.uniform(0.0, 0.15))
    taken = np.zeros((S, S), bool)
    levels = iter(rng.permutation(np.linspace(0.35, 1.0, 6)))
    items = []
    while len(items) < 4:
        big = not items
        r = rng.integers(10, 13) if big else rng.integers(4, 9)
        cy, cx = rng.integers(r + 1, S - r - 1, size=2)
        kind = rng.choice(["disk", "square"])
        if (shape(kind, cy, cx, r + 2) & taken).any():   # keep a gap between objects
            continue
        m = shape(kind, cy, cx, r)
        taken |= m
        img[m] = next(levels)
        if big:
            inner = shape("square", cy, cx, rng.integers(3, 5))
            img[inner] = next(levels)
            items.append((m & ~inner, [m]))         # click on the large object: the whole object
            items.append((inner, [inner, m]))       # click on the small square: two valid masks
        else:
            items.append((m, [m]))
    img += rng.normal(0, 0.03, img.shape)
    return img.astype(np.float32), items

def clicks(rng, n):
    """n (image, click x, click y, valid masks) records. The click is uniform inside a random
    object, and 20% of the time inside the small square (the ambiguous case)."""
    out = []
    while len(out) < n:
        img, items = make_image(rng)
        region, masks = items[1] if rng.random() < 0.2 else items[rng.integers(len(items))]
        ys, xs = np.nonzero(region)
        j = rng.integers(len(ys))
        out.append((img, (xs[j] + 0.5) / S, (ys[j] + 0.5) / S, masks))
    return out

class FourierPE(nn.Module):
    """Random Fourier features of (x, y) in [0, 1]^2, as in SAM's PositionEmbeddingRandom."""
    def __init__(self, n_freq=D // 2, scale=1.0):
        super().__init__()
        self.register_buffer("B", scale * torch.randn(2, n_freq))
    def forward(self, xy):                           # (..., 2) -> (..., 2 * n_freq)
        p = 2 * math.pi * (2 * xy - 1) @ self.B
        return torch.cat([p.sin(), p.cos()], dim=-1)

class TinySAM(nn.Module):
    def __init__(self):
        super().__init__()
        self.register_buffer("levels", torch.linspace(0.0, 1.1, D))   # frozen encoder thresholds
        self.pe = FourierPE()
        g = (torch.arange(S) + 0.5) / S
        self.register_buffer("grid", torch.stack(torch.meshgrid(g, g, indexing="xy"), -1))  # (S, S, 2)
        self.neck = nn.Linear(2 * D, D)              # mixes appearance and position per pixel
        self.point_embed = nn.Parameter(0.1 * torch.randn(D))       # "foreground point" type
        self.out_tokens = nn.Parameter(0.1 * torch.randn(1 + K, D))  # IoU token + K mask tokens
        self.self_attn = nn.ModuleList(nn.MultiheadAttention(D, 4, batch_first=True) for _ in range(2))
        self.cross_attn = nn.ModuleList(nn.MultiheadAttention(D, 4, batch_first=True) for _ in range(2))
        self.mlp = nn.ModuleList(nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D))
                                 for _ in range(2))
        self.norms = nn.ModuleList(nn.LayerNorm(D) for _ in range(6))
        self.hyper = nn.ModuleList(nn.Sequential(nn.Linear(D, D), nn.GELU(), nn.Linear(D, D))
                                   for _ in range(K))
        self.iou_head = nn.Sequential(nn.Linear(D, D), nn.GELU(), nn.Linear(D, K))

    def embed_image(self, img):
        """Frozen 'image encoder' (soft thresholds of the 3x3-smoothed image, nothing learned)
        plus positional encoding, mixed by a learned 1x1 layer: (B, S, S) -> (B, D, S, S)."""
        smooth = F.avg_pool2d(img[:, None], 3, stride=1, padding=1, count_include_pad=False)
        appearance = torch.tanh(12 * (smooth.permute(0, 2, 3, 1) - self.levels))
        pos = self.pe(self.grid).expand(len(img), -1, -1, -1)
        return self.neck(torch.cat([appearance, pos], -1)).permute(0, 3, 1, 2)

    def forward(self, img, click):                   # click: (B, 2) normalized (x, y)
        E = self.embed_image(img)
        img_tok = F.avg_pool2d(E, 4).flatten(2).transpose(1, 2)        # 12 x 12 = 144 tokens
        img_pos = F.avg_pool2d(self.pe(self.grid).permute(2, 0, 1)[None], 4).flatten(2).transpose(1, 2)
        prompt = (self.pe(click) + self.point_embed)[:, None]          # (B, 1, D)
        t = torch.cat([self.out_tokens.expand(len(img), -1, -1), prompt], 1)
        add = lambda t: t + F.pad(prompt, (0, 0, t.shape[1] - 1, 0))  # re-add the prompt embedding
        for i in range(2):
            q = add(t)
            t = self.norms[3 * i](t + self.self_attn[i](q, q, t)[0])
            t = self.norms[3 * i + 1](t + self.cross_attn[i](add(t), img_tok + img_pos, img_tok)[0])
            t = self.norms[3 * i + 2](t + self.mlp[i](t))
        w = torch.stack([h(t[:, 1 + k]) for k, h in enumerate(self.hyper)], 1)   # (B, K, D)
        logits = torch.einsum("bkd,bdhw->bkhw", w, E)                           # per-pixel dot product
        return logits, self.iou_head(t[:, 0])

def focal_dice(logits, target):
    """SAM's mask loss: focal and dice in a 20:1 ratio, one value per predicted mask."""
    p = logits.sigmoid()
    ce = F.binary_cross_entropy_with_logits(logits, target, reduction="none")
    p_t = p * target + (1 - p) * (1 - target)
    focal = (0.25 * target + 0.75 * (1 - target)) * (1 - p_t) ** 2 * ce
    dice = 1 - (2 * (p * target).sum((-1, -2)) + 1) / (p.sum((-1, -2)) + target.sum((-1, -2)) + 1)
    return 20 * focal.mean((-1, -2)) + dice

def iou(a, b):
    return (a & b).sum((-1, -2)) / (a | b).sum((-1, -2)).clamp(min=1)

def to_tensors(data):
    img = torch.tensor(np.stack([d[0] for d in data]))
    click = torch.tensor([[d[1], d[2]] for d in data], dtype=torch.float32)
    return img, click

model = TinySAM()
opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, STEPS)
for step in range(STEPS):
    data = clicks(rng, BATCH)
    img, click = to_tensors(data)
    # One valid mask per click, as in SA-1B: for an ambiguous click, either reading at random.
    gt = torch.tensor(np.stack([d[3][rng.integers(len(d[3]))] for d in data]), dtype=torch.float32)
    logits, iou_pred = model(img, click)
    loss_k = focal_dice(logits, gt[:, None].expand_as(logits))     # (B, K)
    loss = loss_k.min(1).values.mean()                              # backprop only the lowest loss
    with torch.no_grad():
        actual = iou(logits > 0, gt[:, None].bool())               # IoU of each mask with the target
    loss = loss + F.mse_loss(iou_pred, actual)                     # IoU head, weight 1.0
    opt.zero_grad(); loss.backward(); opt.step(); sched.step()
model.eval()

# Held-out images: one click per object, split into unambiguous clicks and clicks on the square.
test_rng = np.random.default_rng(10_000 + SEED)
plain, ambiguous = [], []
while len(plain) < 400 or len(ambiguous) < 100:
    img, items = make_image(test_rng)
    for region, masks in items:
        ys, xs = np.nonzero(region)
        j = test_rng.integers(len(ys))
        (plain if len(masks) == 1 else ambiguous).append((img, (xs[j] + 0.5) / S, (ys[j] + 0.5) / S, masks))
plain, ambiguous = plain[:400], ambiguous[:100]

with torch.no_grad():
    logits, score = model(*to_tensors(plain))
    gt = torch.tensor(np.stack([d[3][0] for d in plain]))[:, None]
    ious = iou(logits > 0, gt)                                      # (N, K)
    top = ious[torch.arange(len(plain)), score.argmax(1)]
    print(f"unambiguous clicks ({len(plain)}): mIoU {top.mean():.3f} for the top-scored mask, "
          f"{ious.max(1).values.mean():.3f} for the better of the {K} (oracle)")

    logits, score = model(*to_tensors(ambiguous))
    masks = logits > 0
    inner = torch.tensor(np.stack([d[3][0] for d in ambiguous]))[:, None]
    whole = torch.tensor(np.stack([d[3][1] for d in ambiguous]))[:, None]
    i_in, i_wh = iou(masks, inner), iou(masks, whole)               # (N, K)
    print(f"clicks on the small square ({len(ambiguous)}):")
    for k in range(K):
        print(f"  mask {k}: IoU with square {i_in[:, k].mean():.3f}, with whole object "
              f"{i_wh[:, k].mean():.3f}, predicted IoU {score[:, k].mean():.3f}")
    both = (i_in.max(1).values > 0.5) & (i_wh.max(1).values > 0.5)
    print(f"  one mask matches the square and one the whole object (IoU > 0.5): {both.float().mean():.0%}")

torch 2.14.1
unambiguous clicks (400): mIoU 0.761 for the top-scored mask, 0.853 for the better of the 2 (oracle)
clicks on the small square (100):
  mask 0: IoU with square 0.670, with whole object 0.132, predicted IoU 0.322
  mask 1: IoU with square 0.143, with whole object 0.935, predicted IoU 0.645
  one mask matches the square and one the whole object (IoU > 0.5): 82%

Training takes about 40 seconds on a laptop CPU; other hardware may change the last digits. On held-out clicks with one valid answer, the top-scored mask reaches an mIoU of 0.761 and the better of the two 0.853, SAM’s oracle gap in miniature. On clicks on the square, the two mask tokens have specialized unprompted: mask 0 matches the square (IoU 0.670), mask 1 the whole object (0.935), and both readings are present for 82% of these clicks. The predicted IoUs, 0.322 and 0.645, lie between the IoUs each mask attains against the two readings; our interpretation is that, with random targets, the head learns an expected IoU, which ranks the masks but cannot know which reading the user meant. Over seeds 0–4, the top-scored mIoU ranged from 0.745 to 0.850 and both readings were present for 76–92% of square clicks (seed 2 has the median split); which token takes which reading changes with the seed, and in seed 1 the split stayed partial.