Salient Object Detection

Salient object detection finds the object, or the few objects, in an image that a person would notice first, and marks their full extent pixel by pixel. The output is a saliency map: an image-sized map whose values say how likely each pixel is to belong to a salient object, usually thresholded into a binary mask. The task is also called salient object segmentation or salient region detection. It is class-agnostic: the model is not told what kind of object to look for, only that one stands out. This makes it a front end for cropping, retargeting, editing and other steps that need “the subject” of a photo without a fixed label set. It is a special case of image segmentation, and its deep models are close relatives of the fully convolutional networks of semantic segmentation. The saliency maps of neural network interpretability, which show which pixels drove a classifier’s decision, are a different subject that shares the name.

Problem Definition

Borji, Cheng, Hou, Jiang and Li (2019) describe the task as two stages, detecting the most salient object and segmenting its accurate region, and note that models rarely separate them: following Itti et al. (1998) and Liu et al. (2007), they compute one saliency map that does both, and benchmarks score only the result. The ground truth is a binary mask of the object or objects that human annotators chose as salient. Which object that is depends on people, so the task rests on their agreement. On PASCAL-S, where 12 people clicked on the objects they found salient, Li, Hou, Koch, Rehg and Yuille (2014) split the annotators into two halves and scored one half’s thresholded masks against the other’s: the F-measure was 0.972, so people largely agree on what the salient object is.

Salient objects and fixation prediction

The older sense of “saliency” in computer vision is fixation prediction: predicting where people look. In a fixation dataset, observers view each image for a few seconds while an eye tracker records their gaze, and a model produces a probability map that should match the recorded fixation points. In a salient object dataset, annotators draw pixel-accurate outlines of the objects they consider salient, and a model’s map should match that mask (Li et al., 2014). Both kinds of model output one continuous map, and Borji et al. (2019) note that they can be used interchangeably for that reason. Their targets differ: fixation maps are sparse points, usually blurred into a density, while salient object masks are uniform regions with sharp outlines that cover the whole object. Metrics differ accordingly: fixation benchmarks use variants of AUC, such as a shuffled AUC that discounts the tendency of fixations to fall near the image centre (Li et al., 2014), while salient object benchmarks use region and boundary measures (below).

The two are linked. Li et al. recorded eye fixations from 8 people and salient-object choices from 12 on the same 850 PASCAL-S images, reported a strong correlation between where people looked and which objects they chose, and built a salient object model that combined fixation prediction with segmentation. Borji et al. (2019) describe the most salient object in some datasets as the one that receives the peak of the human fixation map.

Input and Output

The input is a single image; variants add a depth map, video frames or a group of related images. The output is a map $S : \Omega \to [0, 1]$ on the pixel grid $\Omega$. Classical methods produce relative saliency values that are normalized to $[0, 1]$; deep models apply a sigmoid to a one-channel output, so the map can be read as a per-pixel probability of “salient”. A binary mask is obtained by thresholding, with a fixed threshold, an image-dependent one, or a segmentation step such as GrabCut initialized from the map (Cheng et al., 2015).

Borji et al. (2019) list three properties a good salient object detector should have: few missed salient regions and few false alarms on the background, high or full resolution so that object boundaries are located accurately, and computational efficiency, because the map is usually an input to further processing.

Variants

• Co-salient object detection. The input is a group of images, and the goal is the salient object they have in common, such as the same object seen from different viewpoints or objects of one category (Borji et al., 2019). Early methods took a pair of images and gave low saliency to regions without a correspondence in the other image (Li and Ngan, 2011), multiplied single-image saliency by a region’s repeatedness across the images (Chang, Liu and Lai, 2011), or carried the global contrast and spatial distribution cues of single-image saliency over to clusters formed across all the images, adding correspondence cues between images (Fu, Cao and Tu, 2013), as the survey summarizes them. Following Fu et al., the survey separates co-saliency from image co-segmentation: co-segmentation may also segment similar but non-salient background, some co-segmentation methods need user input, and co-saliency is expected to be cheap enough to serve as preprocessing. Fan, Lin, Ji, Zhang, Fu and Cheng (2020) argue that earlier co-saliency datasets assume the shared objects look alike, and built CoSOD3k, 3,316 images in 160 groups, where the shared object is often similar only in category.
• Depth, light field and video. Extra cues can separate an object from a background of similar colour: depth from stereo pairs or a Kinect sensor, light-field images, and motion between video frames (Borji et al., 2019).
• Weak supervision. Pixel masks are costly, so some methods learn from cheaper labels. Wang et al. (2017) trained a saliency network with image-level category tags only (see Approaches).
• Fixation prediction. As above, a neighbouring task with its own datasets and metrics.

Challenges

• An ill-defined target. Salience is a judgement by people. Images with one clear object give consistent masks, while scenes with several candidates do not, and datasets have dealt with this by keeping only images with one unambiguous object, by combining several annotators’ choices, or by using fixations to pick the object (Borji et al., 2019).
• Dataset design bias. Li et al. (2014) found that the benchmark built by Achanta et al. (2009) over-expressed a stereotypical notion of saliency: its images stand out for strong object boundaries and colour contrast, because they had been selected so that the salient object was unambiguous. Four algorithms that averaged an F-measure of 0.8341 on that benchmark averaged 0.5530 on PASCAL-S, a drop of 33.70% that the authors note can dwarf four years of progress on the older benchmark.
• Centre and border priors. Many photographs have the subject near the centre and background along the borders. Methods that exploit this fail when the object touches the border. In MSRA-B, Li and Yu (2015) note, most images hold one connected salient region and 98% of border pixels are background, which makes the set easy once such priors are used.
• Low contrast, clutter and several objects. Objects whose colour resembles the background, high-contrast texture in the background, and several separate salient objects all defeat cues based on contrast alone. HKU-IS selected images for exactly these properties (Li and Yu, 2015).
• Boundaries and thin structures. Downsampling in networks and pixel-wise losses give blurry outlines, which motivated boundary-aware models such as BASNet (Qin et al., 2019).

Approaches

The survey by Borji et al. (2019) describes three waves: the attention model of Itti et al. (1998); the redefinition of saliency detection as binary segmentation in the work of Liu et al. (2007) and of Achanta and colleagues (the survey cites here a 2008 paper by Achanta, Estrada, Wils and Süsstrunk, which preceded the frequency-tuned method), followed by many contrast-based methods; and deep networks from 2015, first classifying regions and then, after FCNs, predicting maps end to end.

The fixation lineage

Itti, Koch and Niebur (1998) built a model of bottom-up visual attention on the architecture of Koch and Ullman. The image is decomposed into dyadic Gaussian pyramids of nine scales. Center–surround differences between a fine “center” scale $c \in \{2, 3, 4\}$ and a coarser “surround” scale $s = c + \delta$, $\delta \in \{3, 4\}$, give 42 feature maps: 6 for intensity, 12 for red–green and blue–yellow colour opponency and 24 for orientation from Gabor pyramids. The maps are normalized, combined into conspicuity maps for intensity, colour and orientation, and summed into one saliency map. A winner-take-all network then selects the most salient location, and inhibition of return suppresses it so that attention moves to the next. The model predicts where attention goes, as points, not which object is salient; Borji et al. (2019) describe later fixation studies that used eye movements to test it.

Contrast-based methods

The CVPR papers of Liu et al. (2007) and Achanta et al. (2009) could not be read for this article (the publishers’ and institutional copies do not allow automated access). They are described here through the survey of Borji et al. (2019), the journal paper of Cheng et al. (2015) and, for Achanta et al., the authors’ project page and MATLAB code.

Liu et al. (2007) posed salient object detection as segmenting a salient object from the background and collected the MSRA dataset (below), according to the survey and Cheng et al. (2015). The survey’s summary table lists their model as combining local multi-scale contrast, center–surround contrast and a colour spatial-distribution prior in a conditional random field learned from bounding-box annotations. Multi-scale contrast of this kind sums, at every level $l$ of a Gaussian pyramid $I^{(l)}$, the colour differences between a pixel and its neighbours $N(x)$ (Borji et al., 2019):

$$s(x) = \sum_{l=1}^{L} \sum_{x' \in N(x)} \left\| I^{(l)}(x) - I^{(l)}(x') \right\|^2 .$$

Local contrast fires on edges and texture more than on object interiors, so later methods measured contrast against the whole image.

Frequency-tuned saliency. Achanta, Hemami, Estrada and Süsstrunk (2009) wanted full-resolution maps with well-defined boundaries and took the whole image as the “surround”. The project page describes the method as the Euclidean distance between each pixel’s CIELAB vector in a Gaussian-blurred image and the mean Lab vector of the image:

$$s(x) = \left\| \mathbf{I}_\mu - \mathbf{I}_{\omega_{hc}}(x) \right\| ,$$

where $\mathbf{I}_\mu$ is the mean colour and $\mathbf{I}_{\omega_{hc}}$ the blurred image. The authors’ MATLAB code blurs with a $3 \times 3$ Gaussian filter, computes the mean from the blurred image (its comments say the paper used the unblurred one, with similar results) and returns the squared distance, as does the survey’s formula. The survey credits Achanta et al. with the adaptive threshold used in evaluation (below), and the project page releases their binary masks for 1,000 images, a set later called ASD or MSRA1000.

Global contrast. Cheng, Zhang, Mitra, Huang and Hu (2011; journal version Cheng, Mitra, Huang, Torr and Hu, 2015) defined a pixel’s saliency as its colour contrast to all other pixels. In the histogram-based contrast (HC) method, the saliency of a pixel with colour $c_l$ is

$$S(c_l) = \sum_{j=1}^{n} f_j \, D(c_l, c_j),$$

where $f_j$ is the frequency of colour $c_j$ in the image and $D$ the distance in CIELAB. Pixels of the same colour get the same saliency, and grouping by colour turns an $O(N^2)$ sum over pixel pairs into $O(N) + O(n^2)$. To keep $n$ small, each RGB channel is quantized to 12 levels, only the most frequent colours covering more than 95% of the pixels are kept (typically about 85 colours, the authors report), and the rest are assigned to the nearest kept colour. Each colour’s saliency is then smoothed over its $m = n/4$ nearest colours with weights that fall linearly with colour distance, to reduce quantization artifacts. The region-based contrast (RC) method segments the image with the graph-based method of Felzenszwalb and Huttenlocher and scores each region $r_k$ by its colour contrast to all other regions, weighted by their size $w(r_i)$ and by spatial distance $D_s$:

$$S(r_k) = w_s(r_k) \sum_{r_i \ne r_k} \exp\!\left(-\frac{D_s(r_k, r_i)}{\sigma_s^2}\right) w(r_i) \, D_r(r_k, r_i),$$

with $\sigma_s^2 = 0.4$ in normalized pixel coordinates. The CVPR version has no $w_s$ term; the journal version adds a centre prior $w_s(r_k) = \exp(-9 d_k^2)$, where $d_k$ is the region’s mean distance to the image centre, and sets the saliency of regions that lie largely along the image border to zero. Both versions turn the map into a mask with SaliencyCut, iterated GrabCut initialized from a thresholded saliency map. On the 1,000 images of Achanta et al., the CVPR paper reports average precision 90% and recall 90% for SaliencyCut with RC maps, against 75% and 83% for the previous best result, by Achanta et al. The journal version introduced MSRA10K (below).

Priors. Region-based methods of 2011–2014 added assumptions about where salient objects are, summarized by Borji et al. (2019). A background prior treats a narrow band along the image border as pseudo-background: Yang, Zhang, Lu, Ruan and Yang (2013) ranked superpixels on a graph by their relevance to queries on each image side, then re-ranked against the resulting foreground. A boundary connectivity prior scores regions by the length of their contact with the border relative to their extent, which the survey presents as a remedy for objects that touch the border. Spatial distribution priors penalize colours spread widely over the image. Learned combinations of such cues also appeared, such as DRFI’s random forest over regional features (Borji et al., 2019).

Deep learning

Borji et al. (2019) divide deep models into two families. Classic convolutional network (CCN) based methods cut the image into superpixels or patches and use a CNN to compute features that a multilayer perceptron maps to one saliency value per region. Li and Yu (2015) extracted CNN features from three nested windows: the region’s bounding box, the bounding box of its neighbouring regions, and the whole image. The fully connected layers discard spatial layout, which the survey links to blurred, inaccurate predictions near object boundaries; FCN-based methods predict the whole map in one pass at a much lower time cost. The survey describes CNN models as becoming the mainstream direction; in its table of deep models, the most frequent training sets of that period are 2,500 MSRA-B images and up to 10,000 MSRA10K images, mostly with VGG backbones.

Many FCN-based models are encoder–decoders with skip connections in the style of U-Net; the survey names deep supervision and encoder–decoder designs among the promising choices. Recurring ideas include:

• Deep supervision of side outputs. Hou, Cheng, Hu, Borji, Tu and Torr (2017) started from the holistically-nested edge detector (HED), which attaches a loss to side outputs at every stage of VGG-16, and found its gain for saliency “not obvious”. Their DSS adds short connections from deeper side outputs to shallower ones, so that the shallow, high-resolution outputs receive the location of the object from the deep ones. They reported 0.08 seconds per image.
• Predict, then refine. BASNet (Qin, Zhang, Huang, Gao, Dehghan and Jagersand, 2019) has an encoder–decoder with a ResNet-34 encoder (two extra stages added), supervised at seven side outputs, followed by a residual refinement module, a second small encoder–decoder that predicts a correction added to the coarse map.
• Losses beyond pixels. BASNet’s hybrid loss sums, for each of its eight outputs, binary cross-entropy, an SSIM loss $\ell_{\text{ssim}}$ (one minus the SSIM of corresponding $N \times N$ patches of $S$ and $G$), and an IoU loss:

$$\ell = \ell_{\text{bce}} + \ell_{\text{ssim}} + \left(1 - \frac{\sum_{r,c} S(r,c)\, G(r,c)}{\sum_{r,c} \left[S(r,c) + G(r,c) - S(r,c)\, G(r,c)\right]}\right).$$

The authors describe the three terms as working at pixel, patch and map level: cross-entropy weights all pixels alike, the SSIM loss is largest along boundaries early in training, and the IoU loss emphasizes the foreground. In their ablation on ECSSD, the full loss gave maxF 0.942, relaxed boundary F 0.826 and MAE 0.037, against 0.937, 0.806 and 0.042 with cross-entropy alone. BASNet was trained on the 10,553 DUTS-TR images (doubled by horizontal flips) at $224 \times 224$ crops for 400k iterations, about 125 hours on one GTX 1080 Ti, and runs at 25 frames per second at $256 \times 256$.

• Weak supervision. Wang et al. (2017) pre-trained an FCN to predict image-level tags on ImageNet detection data, with a global smooth pooling layer that lets its feature maps localize the tagged objects and a foreground inference net that turns them into a category-agnostic heat map. They then self-trained that net, alternating between estimating pixel labels, refined with a CRF, and training on them. Their WSS model was trained without any pixel masks and reported an Fβ of 0.657 on the DUTS test set, below three of the seven fully supervised methods in their table, DCL (0.714), RFCN (0.712) and MDF (0.673), but above the other four and above the four unsupervised methods (at most 0.511).

Example: global contrast against a small network

The script below builds $64 \times 64$ synthetic scenes with a known salient object, an ellipse of a colour unlike the background gradient, under three conditions: clean; clutter, with up to 25 small specks of vivid colours on the background; and camouflage, with a large background rectangle of nearly the object’s colour. It computes frequency-tuned (FT) maps, with a Gaussian blur of $\sigma = 1$ pixel, and histogram-contrast (HC) maps as described above, a centred Gaussian that ignores the image, an all-zero map, and the output of a small encoder–decoder trained with binary cross-entropy on 2,001 scenes, 667 of each kind (three training seeds). All maps are scored on 200 test scenes per condition with the measures defined under Evaluation Metrics. The S-measure is printed with two block weightings, because the paper and its official code describe it differently (see below). The FT colour conversion matches skimage.color.rgb2lab exactly. NumPy 2.5.4, SciPy 1.18.1 and PyTorch 2.14.1 on four CPU threads; a run took 3 to 4 minutes on our machine, almost all of it training.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from scipy import ndimage

H = W = 64
EPS = np.finfo(float).eps
torch.set_num_threads(4)


# ---------- synthetic scenes with known salient-object masks ----------
def rgb2lab(rgb):
    """sRGB in [0, 1] to CIELAB (D65), the same constants as skimage.color.rgb2lab."""
    c = np.where(rgb > 0.04045, ((rgb + 0.055) / 1.055) ** 2.4, rgb / 12.92)
    M = np.array([[0.412453, 0.357580, 0.180423],
                  [0.212671, 0.715160, 0.072169],
                  [0.019334, 0.119193, 0.950227]])
    xyz = c @ M.T / np.array([0.95047, 1.0, 1.08883])
    f = np.where(xyz > 0.008856, np.cbrt(xyz), 7.787 * xyz + 16 / 116)
    return np.stack([116 * f[..., 1] - 16, 500 * (f[..., 0] - f[..., 1]),
                     200 * (f[..., 1] - f[..., 2])], -1)


def make_scene(rng, cond):
    Y, X = np.mgrid[0:H, 0:W]
    c0, c1 = rng.uniform(0.15, 0.85, (2, 3))                 # background gradient colours
    th = rng.uniform(0, 2 * np.pi)
    t = np.clip(0.5 + ((X - W / 2) * np.cos(th) + (Y - H / 2) * np.sin(th)) / W, 0, 1)
    img = c0 + t[..., None] * (c1 - c0)
    while True:                                               # object colour unlike the background
        obj = rng.uniform(0.05, 0.95, 3)
        if min(np.linalg.norm(obj - c0), np.linalg.norm(obj - c1)) >= 0.35:
            break
    a, b = rng.uniform(6, 16, 2)
    phi = rng.uniform(0, np.pi)
    cy, cx = rng.uniform(14, 50, 2)
    u = (X - cx) * np.cos(phi) + (Y - cy) * np.sin(phi)
    v = -(X - cx) * np.sin(phi) + (Y - cy) * np.cos(phi)
    mask = (u / a) ** 2 + (v / b) ** 2 <= 1                    # the salient object: an ellipse
    if cond == "camouflage":                                  # a large patch of nearly the object's colour
        h, w = rng.integers(20, 40, 2)
        y0, x0 = rng.integers(0, H - h + 1), rng.integers(0, W - w + 1)
        img[y0:y0 + h, x0:x0 + w] = np.clip(obj + rng.normal(0, 0.05, 3), 0, 1)
    img[mask] = obj
    if cond == "clutter":                                     # small specks of vivid, rare colours
        for _ in range(25):
            s = rng.integers(2, 4)
            y0, x0 = rng.integers(0, H - s + 1, 2)
            if not mask[y0:y0 + s, x0:x0 + s].any():
                img[y0:y0 + s, x0:x0 + s] = rng.choice([0.05, 0.95], 3)
    img = np.clip(img + rng.normal(0, 0.02, img.shape), 0, 1)
    return img, mask


def make_set(seed, conds, n_each):
    rng = np.random.default_rng(seed)
    data = [make_scene(rng, c) for c in conds for _ in range(n_each)]
    return np.stack([d[0] for d in data]), np.stack([d[1] for d in data])


# ---------- global-contrast saliency ----------
def norm01(s):
    return (s - s.min()) / (s.max() - s.min() + EPS)


def ft_saliency(img, squared=False):
    """Frequency-tuned (Achanta et al., 2009): Lab distance of the blurred image to the mean Lab colour."""
    blur = np.stack([ndimage.gaussian_filter(img[..., k], 1.0) for k in range(3)], -1)
    d2 = ((rgb2lab(blur) - rgb2lab(img).reshape(-1, 3).mean(0)) ** 2).sum(-1)
    return norm01(d2 if squared else np.sqrt(d2))


def hc_saliency(img):
    """Histogram contrast (Cheng et al., 2011/2015) with colour-space smoothing."""
    q = np.minimum((img * 12).astype(int), 11)                # 12 levels per RGB channel
    idx = ((q[..., 0] * 12 + q[..., 1]) * 12 + q[..., 2]).ravel()
    bins, inv, counts = np.unique(idx, return_inverse=True, return_counts=True)
    order = np.argsort(-counts, kind="stable")
    n = np.searchsorted(np.cumsum(counts[order]), 0.95 * idx.size) + 1
    keep = order[:n]                                          # most frequent colours, >= 95% of pixels
    lab = rgb2lab(img).reshape(-1, 3)
    mean_lab = np.zeros((len(bins), 3))
    np.add.at(mean_lab, inv, lab)
    mean_lab /= counts[:, None]
    nearest = np.linalg.norm(mean_lab[:, None] - mean_lab[keep][None], axis=-1).argmin(1)
    pix = nearest[inv]                                        # rare colours -> nearest kept colour
    f = np.bincount(pix, minlength=n) / idx.size
    D = np.linalg.norm(mean_lab[keep][:, None] - mean_lab[keep][None], axis=-1)
    S = D @ f                                                 # S(c_l) = sum_j f_j D(c_l, c_j)
    m = min(max(n // 4, 2), n)
    if m >= 2:                                                # smoothing over the m nearest colours
        nn_idx = np.argsort(D, axis=1)[:, :m]
        Dn = np.take_along_axis(D, nn_idx, 1)
        T = Dn.sum(1)
        S = ((T[:, None] - Dn) * S[nn_idx]).sum(1) / ((m - 1) * T + EPS)
    return norm01(S[pix].reshape(H, W))


def centre_gaussian():
    Y, X = np.mgrid[0:H, 0:W]
    return np.exp(-((X - (W - 1) / 2) ** 2 + (Y - (H - 1) / 2) ** 2) / (2 * (0.25 * W) ** 2))


# ---------- a small encoder-decoder ----------
class TinyUNet(nn.Module):
    def __init__(self):
        super().__init__()
        blk = lambda i, o: nn.Sequential(nn.Conv2d(i, o, 3, padding=1), nn.ReLU(),
                                         nn.Conv2d(o, o, 3, padding=1), nn.ReLU())
        self.e1, self.e2, self.e3 = blk(3, 8), blk(8, 16), blk(16, 32)
        self.d2, self.d1 = blk(32 + 16, 16), blk(16 + 8, 8)
        self.out = nn.Conv2d(8, 1, 1)

    def forward(self, x):
        x1 = self.e1(x)
        x2 = self.e2(F.max_pool2d(x1, 2))
        x3 = self.e3(F.max_pool2d(x2, 2))
        y = self.d2(torch.cat([F.interpolate(x3, scale_factor=2), x2], 1))
        y = self.d1(torch.cat([F.interpolate(y, scale_factor=2), x1], 1))
        return self.out(y)


def train_net(Xtr, Ytr, seed, epochs=5, bs=32):
    torch.manual_seed(seed)
    net = TinyUNet()
    opt = torch.optim.Adam(net.parameters(), 1e-3)
    X = torch.tensor(Xtr, dtype=torch.float32).permute(0, 3, 1, 2)
    Yt = torch.tensor(Ytr, dtype=torch.float32)[:, None]
    g = torch.Generator().manual_seed(seed)
    for _ in range(epochs):
        for b in torch.randperm(len(X), generator=g).split(bs):
            loss = F.binary_cross_entropy_with_logits(net(X[b]), Yt[b])
            opt.zero_grad()
            loss.backward()
            opt.step()
    return net


def predict(net, Xte):
    with torch.no_grad():
        return torch.sigmoid(net(torch.tensor(Xte, dtype=torch.float32).permute(0, 3, 1, 2)))[:, 0].double().numpy()


# ---------- evaluation measures ----------
def f_beta(p, r, b2=0.3):
    return np.where(p + r > 0, (1 + b2) * p * r / np.maximum(b2 * p + r, EPS), 0.0)


def adaptive_f(s, g):
    m = s >= 2 * s.mean()                                     # Achanta et al.'s adaptive threshold
    tp = (m & g).sum()
    return f_beta(tp / m.sum() if m.sum() else 0.0, tp / g.sum())


def max_f(S, G):
    """Precision and recall averaged over images at each of 256 thresholds; max of F over thresholds."""
    P, R = np.zeros(256), np.zeros(256)
    for s, g in zip(S, G):
        k = np.round(s * 255).astype(int)
        tp = np.cumsum(np.bincount(k[g], minlength=256)[::-1])[::-1]     # pixels >= threshold
        pos = np.cumsum(np.bincount(k.ravel(), minlength=256)[::-1])[::-1]
        P += np.where(pos > 0, tp / np.maximum(pos, 1), 0)
        R += tp / g.sum()
    return f_beta(P / len(S), R / len(S)).max()


def _ssim(x, y):                                              # as in the official S-measure code
    N = x.size
    mx, my = x.mean(), y.mean()
    sx2 = ((x - mx) ** 2).sum() / (N - 1 + EPS)
    sy2 = ((y - my) ** 2).sum() / (N - 1 + EPS)
    sxy = ((x - mx) * (y - my)).sum() / (N - 1 + EPS)
    a, b = 4 * mx * my * sxy, (mx ** 2 + my ** 2) * (sx2 + sy2)
    return a / (b + EPS) if a != 0 else (1.0 if b == 0 else 0.0)


def s_measure(s, g, weights="area"):
    """Structure-measure (Fan et al., 2017), alpha = 0.5, ported from the official MATLAB code."""
    y = g.mean()
    if y == 0:
        return 1 - s.mean()
    if y == 1:
        return s.mean()

    def obj(x, m):
        v = x[m]
        mu = v.mean()
        return 2 * mu / (mu ** 2 + 1 + v.std(ddof=1) + EPS)

    s_obj = y * obj(np.where(g, s, 0), g) + (1 - y) * obj(np.where(g, 0, 1 - s), ~g)
    tot = g.sum()                                             # centroid, 1-based and rounded as in MATLAB
    X = int(np.floor((g.sum(0) * np.arange(1, W + 1)).sum() / tot + 0.5))
    Yc = int(np.floor((g.sum(1) * np.arange(1, H + 1)).sum() / tot + 0.5))
    blocks = [(slice(0, Yc), slice(0, X)), (slice(0, Yc), slice(X, W)),
              (slice(Yc, H), slice(0, X)), (slice(Yc, H), slice(X, W))]
    if weights == "area":                                     # official code: share of the image area
        w = [g[b].size / g.size for b in blocks]
    else:                                                     # paper's wording: share of the GT foreground
        w = [g[b].sum() / tot for b in blocks]
    s_reg = sum(wk * _ssim(s[b], g[b].astype(float)) for wk, b in zip(w, blocks) if g[b].size)
    return max(0.5 * s_obj + 0.5 * s_reg, 0.0)


def e_measure(s, g):
    """Enhanced-alignment measure (Fan et al., 2018) of the adaptive-threshold binary map."""
    fm, gt = (s >= 2 * s.mean()).astype(float), g.astype(float)
    if gt.sum() == 0:
        enh = 1 - fm
    elif gt.sum() == gt.size:
        enh = fm
    else:
        afm, agt = fm - fm.mean(), gt - gt.mean()
        enh = ((2 * agt * afm / (agt ** 2 + afm ** 2 + EPS)) + 1) ** 2 / 4
    return enh.sum() / (gt.size - 1 + EPS)


def scores(S, G):
    return [np.mean([np.abs(s - g).mean() for s, g in zip(S, G)]),
            np.mean([adaptive_f(s, g) for s, g in zip(S, G)]), max_f(S, G),
            np.mean([s_measure(s, g) for s, g in zip(S, G)]),
            np.mean([s_measure(s, g, "foreground") for s, g in zip(S, G)]),
            np.mean([e_measure(s, g) for s, g in zip(S, G)])]


# ---------- run ----------
conds = ["clean", "clutter", "camouflage"]
Xtr, Ytr = make_set(0, conds, 667)
Xte, Yte = make_set(1, conds, 200)
nets = {seed: train_net(Xtr, Ytr, seed) for seed in (0, 1, 2)}

print(f"{'':16s}{'MAE':>7s}{'adpF':>7s}{'maxF':>7s}{'S(area)':>9s}{'S(fg)':>7s}{'adpE':>7s}")
for ci, cond in enumerate(conds):
    sl = slice(200 * ci, 200 * (ci + 1))
    Xc, Gc = Xte[sl], Yte[sl]
    maps = {"FT": [ft_saliency(x) for x in Xc],
            "FT (squared)": [ft_saliency(x, squared=True) for x in Xc],
            "HC": [hc_saliency(x) for x in Xc],
            "centre Gaussian": [centre_gaussian()] * len(Xc),
            "all zeros": [np.zeros((H, W))] * len(Xc)}
    for seed, net in nets.items():
        maps[f"net, seed {seed}"] = list(predict(net, Xc))
    print(f"{cond} (object covers {Gc.mean():.3f} of the pixels)")
    for name, S in maps.items():
        print(f"  {name:14s}" + "".join(f"{v:7.3f}" if i != 3 else f"{v:9.3f}"
                                         for i, v in enumerate(scores(S, Gc))))

Output:

                    MAE   adpF   maxF  S(area)  S(fg)   adpE
clean (object covers 0.095 of the pixels)
  FT              0.252  0.745  0.767    0.639  0.661  0.880
  FT (squared)    0.134  0.713  0.766    0.779  0.786  0.885
  HC              0.164  0.760  0.821    0.738  0.751  0.896
  centre Gaussian  0.335  0.303  0.309    0.498  0.526  0.777
  all zeros       0.095  0.119  0.120    0.453  0.453  0.250
  net, seed 0     0.115  0.479  0.615    0.576  0.563  0.729
  net, seed 1     0.129  0.526  0.586    0.621  0.616  0.792
  net, seed 2     0.124  0.573  0.645    0.650  0.646  0.825
clutter (object covers 0.093 of the pixels)
  FT              0.194  0.590  0.544    0.625  0.640  0.852
  FT (squared)    0.108  0.619  0.543    0.629  0.627  0.869
  HC              0.142  0.725  0.684    0.714  0.732  0.903
  centre Gaussian  0.337  0.279  0.286    0.496  0.525  0.778
  all zeros       0.093  0.118  0.118    0.453  0.453  0.250
  net, seed 0     0.117  0.440  0.479    0.570  0.558  0.738
  net, seed 1     0.140  0.448  0.512    0.601  0.601  0.766
  net, seed 2     0.143  0.477  0.574    0.624  0.629  0.781
camouflage (object covers 0.094 of the pixels)
  FT              0.448  0.118  0.285    0.427  0.447  0.448
  FT (squared)    0.306  0.272  0.286    0.509  0.534  0.681
  HC              0.307  0.283  0.316    0.513  0.547  0.598
  centre Gaussian  0.335  0.290  0.296    0.499  0.528  0.782
  all zeros       0.094  0.118  0.118    0.453  0.453  0.250
  net, seed 0     0.129  0.342  0.344    0.547  0.541  0.658
  net, seed 1     0.159  0.322  0.295    0.565  0.573  0.667
  net, seed 2     0.162  0.329  0.309    0.571  0.583  0.673

Three results matter for reading papers. First, MAE rewards predicting nothing. The all-zero map has the lowest MAE in every condition, equal to the object’s share of the pixels, because about 90% of each image is background. The F-measures and the S-measure put the same map last in the clean and clutter scenes (its F-measures are above zero because a threshold of zero, which is also twice its mean, marks every pixel as salient): its S-measure of 0.453 comes entirely from the background part of the object term, since its foreground term and its block similarities are zero. Second, global contrast does what its definition says. HC’s maxF falls from 0.821 on clean scenes to 0.684 when specks rarer than the object are added, and to 0.316 when the object’s colour becomes common, close to the centre Gaussian’s 0.296. FT, which compares each pixel only with the mean colour, falls further in clutter, to 0.544. On camouflage scenes the measures disagree: FT scores below the all-zero map on the S-measure, and the centre Gaussian, which never looks at the image, has the highest E-measure of all maps (0.782). Third, protocol details move the numbers. Squaring FT’s distance barely changes maxF but changes MAE and S-measure substantially (MAE 0.252 against 0.134 on clean scenes). In our reading, this is because squaring preserves each image’s ordering of pixels, which thresholding sees, while it pushes the background towards zero, which the continuous measures see. Adaptive F is usually below maxF, but in clutter it is higher for FT and HC, and on camouflage scenes for two of the three network seeds. maxF applies one threshold to the whole dataset and averages precision and recall before combining them, while adaptive F picks a threshold per image and averages per-image scores. The two S-measure weightings differ by up to 0.034 and reverse FT and FT (squared) in clutter (0.625 against 0.629 with area weights, 0.640 against 0.627 with foreground weights). A reimplementation that follows the paper’s wording rather than the code can therefore flip a close ranking.

The small network does not beat the contrast methods on clean scenes. Its maxF over seeds 0–2 is 0.586–0.645, against 0.821 for HC, and the seed alone moves it by about 0.06. It has the highest area-weighted S-measure of all maps on camouflage scenes (0.547–0.571; with foreground weights, seed 0 falls just below HC), but its maxF there (0.295–0.344) is close to HC’s. The network is narrow (8, 16 and 32 channels) and trained for 5 epochs, a size fixed before the first run so that the script runs in a few minutes; we expected it to beat both contrast methods everywhere, and it did not. The example therefore says nothing about deep models at the scale of the benchmarks below. What it shows is how the measures behave on maps whose errors are known.

Datasets and Benchmarks

Dataset	Year	Images	Annotation	Notes
MSRA-A / MSRA-B	2007	20,000+ / 5,000	Bounding boxes by 3 / 9 users	Liu et al.; images chosen to contain one unambiguous object
ASD (MSRA1000)	2009	1,000	Pixel masks	Achanta et al.; objects segmented inside MSRA rectangles
MSRA10K	2013 (described 2015)	10,000	Pixel masks	Cheng et al.; MSRA images with consistent boxes
ECSSD	2013 (CSSD)	1,000	Pixel masks by 5 helpers	Extends the 200-image CSSD of Yan et al. (2013)
DUT-OMRON	2013	5,172	Bounding boxes by 5 users in the paper	Yang et al.; one or two objects per image, according to BASNet
PASCAL-S	2014	850	Per-segment salience from 12 people; fixations from 8	Li et al.; PASCAL VOC 2010 validation images
HKU-IS	2015	4,447	Pixel masks by 3 people	Li and Yu; chosen for low contrast, several objects or objects at the border
DUTS	2017	10,553 train / 5,019 test	Pixel masks by 50 subjects	Wang et al.; ImageNet DET images (train) and ImageNet DET test and SUN images (test)

MSRA. Liu et al. selected more than 20,000 images, mostly $400 \times 300$, from an initial 130,099 so that each held a clear object of interest, and had 3–9 users draw bounding boxes (Cheng et al., 2015; Borji et al., 2019). Achanta et al. segmented the objects inside the rectangles of 1,000 images to give pixel masks, because boxes were too coarse and merged several objects into one. Cheng et al. (2015) and the survey treat this set as drawn from MSRA; the project page names its source as the database of Wang and Li (2008), whose ground truth it describes as user-drawn rectangles. Cheng et al. (2015) segmented 10,000 images whose boxes were consistent across users, choosing them as Liu et al. chose their subset B, to build MSRA10K. The project page of Achanta et al. records an erratum: the precision–recall curve in the paper’s Figure 5 was computed on the first 500 images, not all 1,000.

ECSSD. Yan, Xu, Shi and Jia (2013) built the Complex Scene Saliency Dataset, 200 images with structurally complex foregrounds and backgrounds, because MSRA backgrounds are mostly simple and smooth. The extended version, ECSSD, has 1,000 internet images with masks drawn by five helpers, and accompanies the authors’ journal paper on hierarchical saliency (ECSSD page).

PASCAL-S. Li et al. (2014) used the 850 images of the PASCAL VOC 2010 segmentation validation set. Every object was segmented in full first; 12 people then clicked on the objects they found salient, with deliberately vague instructions, and each segment’s salience is the fraction of people who chose it. The images were thus selected independently of saliency, which is how the authors avoid dataset design bias; fixations from 8 people viewing each image for 2 seconds were recorded as well.

DUT-OMRON and HKU-IS. Yang et al. (2013) introduced DUT-OMRON with 5,172 images; their paper evaluates against bounding boxes from five users, and BASNet’s authors, who evaluate with masks, give 5,168 images. Li and Yu (2015) collected 7,320 candidate images that had several disconnected salient objects, an object touching the border, or low colour contrast between object and surroundings, and had three people mask each; HKU-IS keeps the 4,447 images on which the three masks agree closely (pixels marked by all three over pixels marked by any, at least 0.9), with a majority vote as ground truth.

DUTS and the current protocol. Before DUTS, Wang et al. (2017) note, there was no established split, and methods trained on different sets could not be compared fairly. DUTS has 10,553 training images (DUTS-TR) and 5,019 test images (DUTS-TE). Models are now commonly trained on DUTS-TR and tested on DUTS-TE, ECSSD, DUT-OMRON, PASCAL-S and HKU-IS, as BASNet was. The table below gives a few rows of BASNet’s comparison, computed by Qin et al. from maps released by the authors or by running their released models; training sets differ, and “+” marks CRF post-processing. maxF and MAE follow the protocol described below.

Method	Backbone	Training set	ECSSD maxF / MAE	PASCAL-S maxF / MAE	DUTS-TE maxF / MAE
MDF (Li and Yu, 2015)	R-CNN features	MSRA-B (2,500)	0.832 / 0.105	0.759 / 0.142	0.729 / 0.099
DSS+ (Hou et al., 2017)	VGG-16	MSRA-B (2,500)	0.921 / 0.052	0.831 / 0.093	0.825 / 0.056
PiCANetR	ResNet-50	DUTS-TR (10,553)	0.935 / 0.046	0.857 / 0.076	0.860 / 0.050
BASNet (Qin et al., 2019)	ResNet-34	DUTS-TR (10,553)	0.942 / 0.037	0.854 / 0.076	0.860 / 0.047

The ranking depends on the measure: PiCANetR has the higher maxF on PASCAL-S and equals BASNet on DUTS-TE, while BASNet’s lead in BASNet’s own comparison is largest in the relaxed boundary F-measure (0.758 against 0.696 on DUTS-TE).

Evaluation Metrics

A saliency map $S \in [0, 1]^{H \times W}$ is compared with a binary mask $G$. The measures differ in whether they binarize $S$ first, and in the conventions that turn per-image scores into one number per dataset. Papers that use the same name can therefore report different numbers.

Precision, recall and the F-measure. Thresholding $S$ gives a mask $M$; precision and recall compare it with $G$, and the F-measure combines them:

$$F_\beta = \frac{(1 + \beta^2)\, \text{Precision} \cdot \text{Recall}}{\beta^2\, \text{Precision} + \text{Recall}}, \qquad \beta^2 = 0.3 .$$

The weight $\beta^2 = 0.3$, which favours precision, is used by Cheng et al. “as in Achanta et al.” and by BASNet and the other papers cited here; the reasoning given (Cheng et al., 2015, citing Liu et al.) is that recall matters less, because marking the whole image gives 100% recall. Unlike the F1 score, $F_{0.3}$ is not symmetric in precision and recall. Three thresholding protocols are common (Borji et al., 2019):

• Adaptive threshold. Achanta et al. binarize each map at twice its mean value, $T_a = \frac{2}{WH} \sum_{x,y} S(x, y)$, and average the per-image results.
• All thresholds. Sweeping a threshold over 0–255 gives a precision–recall curve. BASNet’s protocol averages precision and recall over the images at each threshold, computes $F_\beta$ from the averages, and reports the maximum over thresholds as maxF.
• Segmentation. A threshold at 95% recall initializes GrabCut, whose output is scored (Cheng et al., 2011).

Mean absolute error. The MAE averages $|S(x, y) - G(x, y)|$ over pixels and then over images, without any threshold. Borji et al. (2019) recommend it because the overlap measures ignore true negatives and because some applications use the continuous map; later papers, BASNet among them, credit it to Perazzi et al. (2012). MAE is dominated by the background when objects are small, so a map that predicts nothing can score well (see the example).

Weighted F-measure. Margolin, Zelnik-Manor and Tal (2014) argued that the common measures, AUC, average precision, the $F_\beta$-measure and the PASCAL VOC overlap score, do not always evaluate binary or non-binary maps reliably, and proposed $F^\omega_\beta$, which extends true and false positives to continuous values and weights each error by its location and neighbourhood.

S-measure. Fan, Cheng, Liu, Li and Borji (2017) observed that pixel-wise measures ignore structure: two maps with the same counts of true and false positives can preserve an object’s shape or break it apart. Their Structure-measure combines an object-aware term $S_o$ and a region-aware term $S_r$:

$$S = \alpha\, S_o + (1 - \alpha)\, S_r, \qquad \alpha = 0.5 .$$

For $S_r$, both maps are cut into four blocks by a horizontal and a vertical line through the centroid of the ground-truth foreground, and an SSIM-like similarity is computed per block,

$$\text{ssim} = \frac{2 \bar{x} \bar{y}}{\bar{x}^2 + \bar{y}^2} \cdot \frac{2 \sigma_x \sigma_y}{\sigma_x^2 + \sigma_y^2} \cdot \frac{\sigma_{xy}}{\sigma_x \sigma_y},$$

and the blocks’ scores are summed with weights $w_k$. $S_o$ rewards a foreground that is uniformly high and a background that is uniformly low. With $\bar{x}_{FG}$ and $\sigma_{x_{FG}}$ the mean and standard deviation of the map over the ground-truth foreground,

$$O_{FG} = \frac{2 \bar{x}_{FG}}{\bar{x}_{FG}^2 + 1 + 2 \lambda\, \sigma_{x_{FG}}},$$

$O_{BG}$ is the same on $1 - S$ over the background, and $S_o = \mu\, O_{FG} + (1 - \mu)\, O_{BG}$, with $\mu$ the foreground’s share of the image. The paper states $\lambda = 0.5$ for all experiments, although the caption of its Figure 2 says $\lambda = 0.25$; the official code uses $\sigma$ in place of $2\lambda\sigma$, which corresponds to $\lambda = 0.5$. The paper says each block’s weight is proportional to the ground-truth foreground it covers, while the official code weights each block by its share of the image area (its comment repeats the paper’s wording). The code also handles all-background and all-foreground ground truth separately and clips negative scores to 0. Fan et al. tested the measure with meta-measures, criteria that a good measure should satisfy: on 1,000 ASD images, a generic centred map outscored the mean of five methods 11 times with the S-measure, far fewer than with AP and AUC.

E-measure. Fan, Gong, Cao, Ren, Cheng and Borji (2018) designed the enhanced-alignment measure for binary maps, where the S-measure’s uniformity terms are, they argue, not well defined. Each binary map is centred by subtracting its global mean, $\varphi_I = I - \mu_I$, and the pixel-wise alignment of the two centred maps is passed through a convex function:

$$\xi = \frac{2\, \varphi_{GT} \circ \varphi_{FM}}{\varphi_{GT} \circ \varphi_{GT} + \varphi_{FM} \circ \varphi_{FM}}, \qquad E = \frac{1}{wh} \sum_{x, y} \frac{\left(1 + \xi(x, y)\right)^2}{4} .$$

A pixel where both maps lie on the same side of their means contributes a positive alignment, so the term mixes local matching with the image-level statistics in one quantity. The paper binarizes maps with the adaptive threshold. The official code divides by $wh - 1$ rather than $wh$. On four datasets the authors report that, unlike IoU, F1 and $F^\omega_\beta$, the E-measure, like the S-measure, never preferred a Gaussian noise map to the mean of ten state-of-the-art maps.

Boundary measures. BASNet adds a relaxed boundary F-measure: the one-pixel boundaries of the binarized map (threshold 0.5) and of the mask are matched within $\rho = 3$ pixels, and precision and recall of boundary pixels give $F_\beta$ (Qin et al., 2019).

Applications

Borji et al. (2019) list object detection and recognition, image and video compression, video summarization, photo collage, retargeting, cropping and thumbnailing, image quality assessment, segmentation, content-based image retrieval, image editing, visual tracking, object discovery and human–robot interaction. Cheng et al. used their maps for content-aware image resizing and non-photorealistic rendering, and SaliencyCut to extract objects automatically from internet images (Cheng et al., 2015). Co-saliency serves collection-aware cropping, co-segmentation and weakly supervised learning (Fan et al., 2020).

Open Problems

• What counts as salient. With several candidate objects, the most salient one depends on the viewer and the task; datasets that keep only unambiguous images train models for an easier problem than real photographs pose (Li et al., 2014).
• Dataset bias and generalization. Centre bias, object size and the way images were selected all shape benchmark scores, as the PASCAL-S study showed (Li et al., 2014; Borji et al., 2019).
• Which measure to trust. MAE, maxF, adaptive F, mean F, $F^\omega_\beta$, S-measure and E-measure can rank the same methods differently, and the same name hides different protocols. The meta-measure studies of Margolin et al. (2014) and Fan et al. (2017, 2018) are attempts to choose among them.
• Boundaries and fine structure. Thin parts, holes and object outlines remain the main source of error in deep models, which is what losses such as BASNet’s and boundary measures address.
• Relation to semantics. In our reading, deep models learn what kinds of objects people tend to choose. How much of their success comes from contrast and how much from recognizing typical subjects is not settled, and it matters for scenes unlike the training data.