Scene Recognition

Scene recognition assigns a photograph to the kind of place it shows: a kitchen, a bedroom, a forest path, a coast, a highway, an airport terminal. It is also called scene classification or scene categorization, and it is a form of image classification whose label describes the whole environment rather than one object in it. What sets it apart is that the evidence is spread over the image: the layout of large surfaces (sky above, ground below, walls on either side), the textures and materials of those surfaces, and the objects the place contains and where they sit. A street and a highway contain cars; a bookstore and a library contain books; the difference lies in arrangement and in which objects co-occur.

The task shaped several ideas in recognition. Oliva and Torralba’s holistic “spatial envelope” model (2001) and its GIST descriptor showed that a scene’s category can be estimated from coarsely localized texture statistics without segmenting it into objects. Lazebnik, Schmid and Ponce (2006) introduced spatial pyramid matching on a scene benchmark. Indoor scenes, which hand-designed global descriptors handled poorly, motivated representations built from objects. And the gap between networks trained on object-centric data such as ImageNet and on scene-centric data led to Places, whose pretrained networks became the usual starting point for the task.

Problem Definition

Given a fixed list of $K$ scene categories and labelled training photographs, predict the category of a new photograph. Formally this is single-label multi-class classification, so the methods and losses of image classification apply. What makes a category a scene category is set by the datasets that define the task:

• A place one can be in. Xiao et al. (2010) define a scene for the SUN database as a place in which a human can act, or to which a human could navigate. Their list kept building types and the interiors of vehicles but excluded non-navigable views such as a desktop; Places inherited the list.
• Function and entry level. Most category names are basic and entry-level names, the level people use in everyday speech (Zhou et al., 2018), and categories can be separated by what people do there: the Places project page, as the Places article reports, treats a home bedroom, a hotel bedroom and a nursery as different categories.
• Holistic properties. Oliva and Torralba proposed describing a scene by a few perceptual dimensions of its overall shape — the abstract on their project page lists naturalness, openness, roughness, expansion and ruggedness — which they estimated from spectral and coarsely localized image information, and they argued that object identity is not required to categorize a scene.

The standard assumptions are one label per image, a closed set of categories and test images from the training distribution. Each is violated in practice, which is what the variants below address.

Input and Output

The input is usually a single colour photograph taken at eye level, from a few hundred pixels on a side (the 15-Scene images average about 300 × 250 pixels; MIT Indoor-67 and SUN397 images are at least 200 pixels on the shorter side) to the 256 × 256 resized images of Places. Several classical systems worked in grayscale only: Lazebnik et al. (2006) processed all images in grayscale, and LMgist, the official GIST code, averages the colour channels before filtering.

The output is a score per category, from which the top-1 label is taken, or a ranked list of five labels for benchmarks scored by top-5 error. Some systems also output scene attributes (below), or an indoor/outdoor decision, often by mapping the predicted category to a fixed indoor or outdoor label.

Variants

• Indoor/outdoor and superordinate classification. Coarse decisions such as indoor against outdoor, or indoor, outdoor natural and outdoor man-made, the three top nodes of the SUN hierarchy. Xiao et al. (2016, Fig. 19c) report indoor/outdoor classification on SUN with each of their features. Quattoni and Torralba (2009) cite earlier indoor/outdoor work such as Szummer and Picard (1998).
• Indoor scene recognition. Treated as a sub-problem because indoor categories (a bakery against a deli, a bedroom against a hotel room) are separated more by their objects than by their global layout (Quattoni & Torralba, 2009).
• Scene attributes. Describing a scene by many binary properties (materials, surfaces, functions, spatial layout) instead of, or as well as, one category; see Approaches.
• Scene detection and multi-label scenes. Xiao et al. (2010, §5) relaxed the one-image-one-scene assumption by detecting scene categories in sub-windows, so that a photograph of a harbour village can contain both a harbour and a village.
• Scene parsing. Labelling every pixel with a surface or object class, which is semantic segmentation on scene datasets such as ADE20K; the Places team ran scene-parsing challenges alongside its classification challenges.
• Place recognition. Recognizing the particular place (this street corner) rather than the category of place is visual place recognition, a retrieval problem treated with image retrieval methods.
• Event and activity scenes. Datasets such as UIUC Event8 (eight sports-event classes, which Li et al. call UIUC-Sports) label what is happening in a scene; Zhou et al. (2014) and Li et al. (2010) report results on it next to scene benchmarks.

Challenges

• Large variation within a category. Kitchens differ in size, style, viewpoint and clutter. Quattoni and Torralba (2009, Fig. 3) show that the average image of most indoor categories is a nearly uniform field, unlike the averages of 15-Scene or Caltech-101 categories.
• Small differences between categories. SUN’s 397 categories include near neighbours such as abbey, church and cathedral, or dining room, dining car and home dinette. Xiao et al. (2010) wrote explicit rules to keep such definitions from overlapping. In this respect large scene benchmarks share a difficulty with fine-grained image classification, though scene categories differ by function and arrangement rather than by small parts.
• Blurry boundaries and several scenes in one image. Field grades into forest; a photograph can show a beach and a village. Patterson and Hays (2012) list four problems with purely categorical scene labels: they cannot express variation within a category, they impose hard boundaries on a continuous space, they cannot describe images that show several categories, and they do not generalize to unseen kinds of scenes.
• Objects at many scales. A scene contains objects from a few pixels to most of the frame. Herranz et al. (2016) found that networks trained on objects and on scenes work best at different image scales (below).
• Human ambiguity. On SUN397, Amazon Mechanical Turk workers who passed a reliability filter reached 68.5% leaf-level accuracy, and one of the dataset’s authors 70.6% (Xiao et al., 2010, §3). The confusions were mostly between semantically related categories, such as abbey and church or bayou and river.
• Non-visual cues in the data. Quattoni and Torralba (2009, footnote 1) note that the original 15-Scene images have resolutions and aspect ratios that correlate with the categories; they cropped all images to 256 × 256 squares for their comparison.

Approaches

Holistic descriptors: the spatial envelope and GIST

Oliva and Torralba’s 2001 paper modelled scene recognition without segmentation or object detection: a low-dimensional representation of the scene’s overall structure, the spatial envelope, estimated from the image’s spectrum and from coarsely localized energy. The paper itself could not be read for this article (no open copy could be reached: the publisher’s site is not used for this project, and the authors’ copy on cvcl.mit.edu did not respond), so its method is described here through the official code and through papers that describe the descriptor. The sources read here give different counts, which the paper would settle: the abstract on the project page lists five perceptual dimensions, while Patterson and Hays (2012), citing the 2001 paper together with a 2002 workshop paper by the same authors (“Scene-Centered Description from Spatial Envelope Properties”), describe eight spatial-envelope attributes, found by having participants partition a database of eight scene categories. Which paper introduced which set could not be checked.

The descriptor that came out of this work, GIST, is computed by the authors’ MATLAB function LMgist.m in these steps (read line by line from the official code):

1. Convert to grayscale, resize and crop to a square (128 pixels by default; 256 in the page’s examples), and rescale intensities to $[0, 255]$.
2. Prefilter: take $\log(I + 1)$, subtract a Gaussian low-pass version (a whitening step with cut-off parameter $f_c = 4$ cycles per image), and divide by a local standard deviation, $I' = I_w / (0.2 + \sigma_{\text{local}})$, which normalizes local contrast.
3. Filter with a bank of Gabor-like filters defined in the Fourier domain, by default 4 scales with 8 orientations each (32 filters). The filter at scale $i = 0, \dots, 3$ (from high to low frequency) and orientation $\theta_j = \pi j / n_i$, $j = 0, \dots, n_i - 1$, has the transfer function

$$G_{ij}(f, \phi) = \exp\!\left(-3.5\left(\frac{f}{N\,\sigma_i} - 1\right)^{2} - 2\pi \frac{n_i^2}{64}\, (\phi + \theta_j)^2\right), \qquad \sigma_i = \frac{0.3}{1.85^{\,i}},$$

where $f$ is the radial frequency in cycles per image, $\phi$ the orientation of the frequency vector (wrapped to $[-\pi, \pi]$), $N$ the padded image size and $n_i$ the number of orientations at scale $i$ (our transcription of createGabor). 4. Take the magnitude of each filtered image and average it over a $w \times w$ grid of non-overlapping blocks ($w = 4$ by default), giving $32 \times 16 = 512$ numbers.

The result is a vector of oriented, band-pass energies at coarse positions: how much vertical structure at low frequency there is in the top-left block, how much fine horizontal texture in the bottom row, and so on. Two images are compared by squared Euclidean distance, as the code page does, or classified with an SVM. The descriptor’s size varies between papers: Xiao et al. (2016) give $8 \times 4 \times 16 = 512$ dimensions; Xiao et al. (2010) describe a bank of 24 filters at 8 orientations and 4 scales, which do not multiply to 24; and Quattoni and Torralba (2009) use the online code with 384 dimensions, which is 24 filters on a 4 × 4 grid (our arithmetic).

GIST captures layout and texture but not objects. On the 15 scene categories, with images cropped to 256 × 256, Quattoni and Torralba (2009, Fig. 1) report 73.0% for GIST with an SVM, against 73.4% for spatial pyramid matching, 64.1% for a bag of words and 30.6% for raw colour pixels; the GIST average was 62.9% on indoor categories and 78.1% on outdoor ones. On SUN397 with 50 training images per class, GIST reached 16.3%, far below the best single features (Xiao et al., 2016, Fig. 19b).

Bags of features and spatial pyramids

The bag of visual words represents an image by a histogram of quantized local descriptors such as SIFT. Fei-Fei and Perona (2005) applied it to scenes with a Bayesian hierarchical model in which image regions belong to learned “themes”, a form of topic model, on 13 categories; according to Lazebnik et al. (2006), their best result was 65.2%, and they found densely sampled features better than interest points for scenes. Lazebnik et al. adopted dense SIFT on a regular grid, arguing that uniform regions such as sky, calm water or road surface need to be described too.

Spatial pyramid matching adds coarse position: histograms of visual words are computed over the whole image and over $2 \times 2$, $4 \times 4$, … grids, weighted so that matches in finer cells count more, and compared by histogram intersection; the bag-of-visual-words article gives the kernel and its dimensions. Lazebnik et al. relate their method to the older practice of partitioning an image into blocks and pooling statistics in each, which they call “subdivide and disorder” and illustrate with GIST among other descriptors; they suggest that combining several grid resolutions in a principled way may answer the open question of which subdivision to use, and that such descriptors work because they perform approximate geometric matching.

Their results on the 15-Scene benchmark (100 training images per class, the rest for testing, mean per-class recognition rate over ten random splits, grayscale; Table 1) show what the grid adds:

Features	$L = 0$ (bag of features)	Pyramid, $L = 2$	Pyramid, $L = 3$
Weak features: oriented edge points, 2 scales × 8 orientations ($M = 16$)	45.3 ± 0.5	64.7 ± 0.7	66.8 ± 0.6
Dense SIFT, $M = 200$ words	72.2 ± 0.6	81.1 ± 0.3	80.7 ± 0.3
Dense SIFT, $M = 400$ words	74.8 ± 0.3	81.4 ± 0.5	81.1 ± 0.6

The weak features were designed to resemble GIST, and for them the grid adds about 20 points. Doubling the vocabulary helped at $L = 0$ but the difference all but vanished with the pyramid, from which the authors concluded that coarse geometric cues carry more discriminative power than a larger vocabulary. The remaining confusions were among indoor classes (kitchen, bedroom, living room) and among natural ones such as coast and open country.

Objects and regions as features

Indoor categories pushed the field toward objects. Quattoni and Torralba (2009) introduced MIT Indoor-67 and a model built from prototypes: each prototype image has regions of interest (segmented by annotators or by an automatic segmentation), each described by a $2 \times 2$ spatial pyramid of 200 visual words and allowed to shift within a small window, combined with the prototype’s global GIST. With 331 prototypes, combining regions with GIST did best; their bar chart (Fig. 6) shows about 21% for an RBF-kernel SVM on GIST alone and about 26% for regions plus GIST from annotated segmentations (our reading of the bars; Razavian et al., 2014, tabulate the latter as 26.1). The figure’s axis and caption say average precision, while the text describes the measure as average multi-class accuracy, the mean of the confusion matrix’s diagonal.

Object Bank (Li, Su, Xing & Fei-Fei, 2010) represented an image by the responses of 200 pre-trained generic object detectors, each run at 12 scales and pooled over a three-level spatial pyramid. With logistic regression it reported 80.9% on 15-Scene, on par with the 81.1% of spatial pyramid matching, and 37.6% on MIT Indoor-67, against the 26% of Quattoni and Torralba (Table 1). The authors attribute the larger gain on the indoor and event datasets to the semantic content of the object responses.

CNN features: object-centric and scene-centric

The first features from convolutional networks trained on ImageNet transferred well to scenes. DeCAF (Donahue et al., 2014) reached 40.94% on SUN397 with 50 training and 50 test images per class, averaged over five splits, against 38.0% for the multi-feature kernel combination of Xiao et al. (2010); the paper’s text names the SVM on the seventh-layer features as the best method, while its Table 3 and caption put 40.94% under logistic regression on the sixth layer and call the bold entry logistic regression on the seventh. Razavian et al. (2014) reported 58.4% mean accuracy on MIT Indoor-67 with OverFeat features and a linear SVM, and 69.0% with data augmentation.

Zhou et al. (2014) argued that features learned from object-centric images were not the best for scenes, and trained the same network on Places; the Places article covers the dataset and its benchmarks. With linear SVMs on the seventh-layer features of the two networks, the Places-trained network was better on SUN397, MIT Indoor-67, 15-Scene and SUN Attribute, and the ImageNet-trained one on the object benchmarks, on Stanford Action40 and, narrowly, on UIUC Event8 (NIPS 2014, Table 2):

Benchmark (standard protocol)	Places-CNN features	ImageNet-CNN features
SUN397	54.32 ± 0.14	42.61 ± 0.16
MIT Indoor-67	68.24	56.79
15-Scene	90.19 ± 0.34	84.23 ± 0.37
Caltech-101	65.18 ± 0.88	87.22 ± 0.92
Caltech-256	45.59 ± 0.31	67.23 ± 0.27

This is the clearest statement of the object-centric against scene-centric distinction: the same architecture learns different features from different data, and the features suit the kind of image they were learned from. Zhou et al. (2014) also trained a Hybrid-CNN on both datasets together, which the Places article describes.

Herranz, Jiang and Li (2016) traced the distinction to scale. They extracted features from patches at seven scales, from the whole image at 227 × 227 pixels to small patches of an image enlarged to 1827 × 1827, and found that the Places network works best applied to the whole image and degrades quickly on smaller patches, while the ImageNet network does best on patches at intermediate scales and beats the Places network at most local scales; the two curves cross at about a third of the image’s size, which the authors loosely call the border between an object range and a scene range. Combining a Places network on the whole image with ImageNet networks on patches, each at the scale it suits, gave their best results on the two larger benchmarks (Table 1, with an SVM on the concatenated fc7 features): with three VGG-16 networks (Places at 227 and 451 pixels, ImageNet at 899), 86.04% on MIT Indoor-67 and 70.17% on SUN397, and 95.14% on 15-Scene, where two-network combinations reached 95.18%, against a single Places VGG baseline at 92.90%, 80.45% and 66.50%. They compare 70.17% with the human figures of Xiao et al. (2010), 68.5% for good workers and 70.6% for an expert.

The Places TPAMI paper (Zhou et al., 2018, Table 3) repeats the comparison with networks trained on Places365; the Places article gives the SUN397 result of the Places365 VGG features, and on MIT Indoor-67 the same features reached 76.53%, below the 79.76% of the Places205 VGG features.

Scene attributes

Patterson and Hays (2012) built the SUN Attribute database to describe scenes by properties instead of categories. Starting from open-ended descriptions that crowd workers wrote for typical images of 707 SUN categories, they settled on 102 binary attributes: 38 materials (such as vegetation or cement), 11 surface properties (such as rusty), 36 functions or affordances (such as playing or cooking) and 17 spatial-envelope attributes (such as enclosed or symmetric), the last group extended by hand so that it contains the attributes of Oliva and Torralba’s work. They labelled 20 images from each of the 717 SUN categories that had at least 20, 14,340 images in all, each with every attribute. Each label was collected three times, once from the general worker pool and twice from 38 trusted workers selected out of about 800. On manual inspection about 93% of individual labels were reasonable, and 95% of consensus labels (at least two of three votes for present, none for absent).

Two of their results show what attributes add. Classifying the 717 categories from the ground-truth attribute vectors alone, with no visual features, was roughly twice as accurate as visual features with one training image per category, and the two equalized at about 20 per category (Fig. 8), which the authors read as a sign that attributes could support zero-shot recognition of categories with no training images. And attributes can be recognized from images: SVMs on a combination of GIST, HOG 2×2, self-similarity and geometric-context colour histograms reached an average precision of 0.88 over all attributes, with balanced training (300 images) and test (50 images) sets per attribute and five random splits; some surface-property and spatial-envelope attributes were hardest. Zhou et al. (2014) report attribute recognition on this database with Places-CNN features, and the Places demonstration predicts the attributes.

Vision–language models

CLIP classifies scenes without training on scene labels, by comparing an image embedding with embeddings of prompts such as “a photo of a bedroom”. Radford et al. (2021) evaluate SUN397 with 19,850 training and 19,850 test images (50 per class; the paper does not say which of the ten partitions it used). Their largest model, ViT-L/14 at 336 pixels, scored 68.4% zero-shot (Table 11 of the arXiv version), without any SUN397 training image, against 54.32% for the linear SVM on Places-CNN features in 2014, which trained on 50 images per class. The comparison is not controlled, since the two numbers come from different partitions and training data. Radford et al. checked their 400 million training pairs for near-duplicates of the evaluation images, SUN397 among them, and compared zero-shot accuracy on each full evaluation set with accuracy on its overlap-free subset: the largest estimated gain from overlap was 0.6 points, on Birdsnap.

Example: what a spatial grid adds to GIST

The example below tests the claim behind GIST and spatial pyramids: that pooling local texture statistics over a coarse grid recovers layout that a whole-image average discards. It builds synthetic $128 \times 128$ scenes from textures bundled with scikit-image (brick, grass and gravel) and a synthetic sky. Experiment 1 has four classes in two pairs whose members contain the same textures in the same expected proportions, arranged differently: street (brick walls left and right, sky above and gravel below in the middle) against plaza (horizontal bands of sky, brick and gravel), and path (grass on both sides) against lawn (horizontal bands of sky, grass and gravel). Region sizes, texture scale and position, brightness and noise vary from image to image. Two checks use the generator’s region maps: the area fraction of each texture per class, and a nearest-centroid classifier on $4 \times 4$ block averages of the region maps, which sees layout but no image content.

GIST is a NumPy port of LMgist.m (prefilter, the 32 default filters, 32-pixel symmetric padding, block averaging) pooled over $1 \times 1$ (no layout), $2 \times 2$ and $4 \times 4$ grids, each classified with a linear SVM. The bag-of-visual-words baseline quantizes the 32 Gabor magnitudes at every fourth pixel into 64 words learned by k-means, and compares $L = 0$ with a spatial pyramid of $L = 2$ using an intersection-kernel SVM, as Lazebnik et al. did. Each class has 100 training and 100 test images, and the script loops over three seeds. After experiment 1 it prints a diagnostic of orientation energy and runs a second experiment, plaza against a layout with the same bands in another order; both were added after the first run, in response to its result, as described below the output. It takes about two to three minutes (1 min 46 s to 3 min 7 s in our runs; NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2, scikit-learn 1.7.2).

import numpy as np
from skimage import data
from skimage.transform import resize
from scipy.ndimage import gaussian_filter
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC, SVC

N = 128                                    # image side
TEX = {1: data.brick() / 255.0, 2: data.grass() / 255.0, 3: data.gravel() / 255.0}
CLASSES = ["street", "plaza", "path", "lawn"]
SIDE = {"street": 1, "plaza": 1, "path": 2, "lawn": 2, "terrace": 1}   # brick or grass

def texture(rng, k):
    """A 128x128 patch of bundled texture k at a random position and scale."""
    T, s = TEX[k], rng.uniform(0.75, 1.25)
    m = int(round(N / s))
    y, x = rng.integers(0, T.shape[0] - m, 2)
    patch = resize(T[y:y + m, x:x + m], (N, N), anti_aliasing=True)
    return rng.uniform(0.8, 1.2) * patch + rng.uniform(-0.1, 0.1)

def sky(rng):
    y = np.arange(N)[:, None] / N
    low = gaussian_filter(rng.normal(0, 1, (N, N)), 8)
    return 0.75 + 0.15 * (1 - y) + 0.6 * low * np.ones((1, N))

def render(rng, cls):
    """Returns the image and its region map (0 sky, 1 brick, 2 grass, 3 gravel).
    street/path: side walls (or hedges) of width ww, sky above row h, gravel below.
    plaza/lawn: horizontal bands with the same area fractions, drawn from the same
    distribution of (ww, h), so the two classes of a pair share their composition.
    terrace (experiment 2): the plaza's bands in another order, gravel above brick."""
    ww, h = rng.uniform(28, 40), rng.uniform(40, 64)
    side = SIDE[cls]
    lab = np.zeros((N, N), int)
    yy, xx = np.mgrid[0:N, 0:N]
    if cls in ("street", "path"):
        walls = (xx < ww) | (xx >= N - ww)
        lab[walls] = side
        lab[~walls & (yy >= h)] = 3
    elif cls == "terrace":
        f_side = 2 * ww / N
        f_sky = (1 - f_side) * h / N
        a, b = f_sky * N, (1 - f_side) * N
        lab[(yy >= a) & (yy < b)] = 3
        lab[yy >= b] = side
    else:
        f_side = 2 * ww / N
        f_sky = (1 - f_side) * h / N
        a, b = f_sky * N, (f_sky + f_side) * N
        lab[(yy >= a) & (yy < b)] = side
        lab[yy >= b] = 3
    img = sky(rng)
    for k in (1, 2, 3):
        if (lab == k).any():
            img = np.where(lab == k, texture(rng, k), img)
    return img + rng.normal(0, 0.02, img.shape), lab

# ---- GIST: NumPy port of LMgist.m (Oliva & Torralba's official MATLAB code) ----
def create_gabor(ors, n):
    fx, fy = np.meshgrid(np.arange(-n / 2, n / 2), np.arange(-n / 2, n / 2))
    fr = np.fft.fftshift(np.sqrt(fx ** 2 + fy ** 2))
    t = np.fft.fftshift(np.angle(fx + 1j * fy))
    G = []
    for i, o in enumerate(ors):
        for j in range(o):
            p = (0.35, 0.3 / 1.85 ** i, 16 * o ** 2 / 32 ** 2, np.pi / o * j)
            tr = t + p[3]
            tr = tr + 2 * np.pi * (tr < -np.pi) - 2 * np.pi * (tr > np.pi)
            G.append(np.exp(-10 * p[0] * (fr / n / p[1] - 1) ** 2 - 2 * p[2] * np.pi * tr ** 2))
    return np.array(G)

def prefilt(img, fc=4):
    w, s1 = 5, fc / np.sqrt(np.log(2))
    img = np.pad(np.log(img + 1), w, mode="symmetric")
    sn, sm = img.shape
    n = max(sn, sm); n += n % 2
    img = np.pad(img, ((0, n - sn), (0, n - sm)), mode="symmetric")
    fx, fy = np.meshgrid(np.arange(-n / 2, n / 2), np.arange(-n / 2, n / 2))
    gf = np.fft.fftshift(np.exp(-(fx ** 2 + fy ** 2) / s1 ** 2))
    out = img - np.real(np.fft.ifft2(np.fft.fft2(img) * gf))
    localstd = np.sqrt(np.abs(np.fft.ifft2(np.fft.fft2(out ** 2) * gf)))
    out = out / (0.2 + localstd)
    return out[w:sn - w, w:sm - w]

BE = 32
G = create_gabor([8, 8, 8, 8], N + 2 * BE)

def energy_maps(img):
    """The 32 Gabor magnitude maps (32 x 128 x 128) that LMgist pools."""
    img = img - img.min(); img = 255 * img / img.max()
    x = np.pad(prefilt(img), BE, mode="symmetric")
    out = np.abs(np.fft.ifft2(np.fft.fft2(x)[None] * G))
    return out[:, BE:BE + N, BE:BE + N]

def down(maps, w):
    """LMgist's downN: mean over a w x w grid of non-overlapping blocks."""
    e = np.fix(np.linspace(0, N, w + 1)).astype(int)
    return np.array([[maps[:, e[i]:e[i + 1], e[j]:e[j + 1]].mean(axis=(1, 2))
                      for j in range(w)] for i in range(w)]).ravel()

# ---- bag of visual words over local Gabor-energy descriptors, with a spatial pyramid ----
STEP = 4
def descriptors(maps):
    return maps[:, STEP // 2::STEP, STEP // 2::STEP].reshape(32, -1).T   # 1024 x 32

def pyramid(words, k, L):
    g = N // STEP
    r, c = np.divmod(np.arange(g * g), g)
    feats = []
    for l in range(L + 1):
        cells = 2 ** l
        idx = (r * cells // g) * cells + (c * cells // g)
        hist = np.zeros((cells * cells, k))
        np.add.at(hist, (idx, words), 1)
        weight = 1 / 2 ** L if l == 0 else 1 / 2 ** (L - l + 1)
        feats.append(weight * hist.ravel() / len(words))
    return np.concatenate(feats)

def intersection(A, B):
    return np.array([np.minimum(a, B).sum(1) for a in A])

def scores(y, p):
    acc = np.mean([np.mean(p[y == c] == c) for c in np.unique(y)])
    if len(np.unique(y)) == 2:
        return acc, float("nan")
    partner = np.array([1, 0, 3, 2])
    wrong = p != y
    within = np.mean(p[wrong] == partner[y[wrong]]) if wrong.any() else float("nan")
    return acc, within

def evaluate(maps, y, train, rng, seed):
    res = {}
    for w in (1, 2, 4):
        X = np.array([down(m, w) for m in maps])
        sc = StandardScaler().fit(X[train])
        clf = LinearSVC(C=1.0, random_state=0, max_iter=20000).fit(sc.transform(X[train]), y[train])
        res[f"GIST {w}x{w} ({X.shape[1]} dims)"] = scores(y[~train], clf.predict(sc.transform(X[~train])))

    D = [descriptors(m) for m in maps]
    pool = np.concatenate([D[i] for i in np.flatnonzero(train)])
    sample = pool[rng.choice(len(pool), 20000, replace=False)]
    km = KMeans(n_clusters=64, n_init=1, random_state=seed).fit(sample)
    words = [km.predict(d) for d in D]
    for L in (0, 2):
        H = np.array([pyramid(wd, 64, L) for wd in words])
        Ktr = intersection(H[train], H[train]); Kte = intersection(H[~train], H[train])
        svm = SVC(kernel="precomputed", C=1.0).fit(Ktr, y[train])
        res[f"BoW L={L} ({H.shape[1]} dims)"] = scores(y[~train], svm.predict(Kte))
    return res

PER_CLASS = 200                 # 100 training + 100 test images per class
diag = []
results = {}
for seed in range(3):
    rng = np.random.default_rng(seed)
    imgs, labs, y = [], [], []
    for c, cls in enumerate(CLASSES):
        for _ in range(PER_CLASS):
            im, lab = render(rng, cls)
            imgs.append(im); labs.append(lab); y.append(c)
    y = np.array(y)
    train = np.tile(np.r_[np.ones(100, bool), np.zeros(100, bool)], 4)

    # Check 1: composition per class from the region maps (fractions of sky/brick/grass/gravel).
    frac = np.array([[np.mean(l == k) for k in range(4)] for l in labs])
    if seed == 0:
        print("area fractions, mean (sd):  sky        brick      grass      gravel")
        for c, cls in enumerate(CLASSES):
            m, s = frac[y == c].mean(0), frac[y == c].std(0)
            print(f"  {cls:7s}                " + "  ".join(f"{a:.2f} ({b:.2f})" for a, b in zip(m, s)))
    # Check 2: nearest centroid on 4x4 block means of the one-hot region maps (layout only).
    onehot = np.array([np.stack([(l == k) for k in range(4)]).astype(float)
                       .reshape(4, 4, 32, 4, 32).mean(axis=(2, 4)).ravel() for l in labs])
    cent = np.array([onehot[train & (y == c)].mean(0) for c in range(4)])
    pred = np.argmin(((onehot[~train][:, None] - cent[None]) ** 2).sum(-1), 1)
    oracle = scores(y[~train], pred)[0]

    maps = [energy_maps(im) for im in imgs]
    res = {"layout oracle (region maps)": (oracle, float("nan"))}
    res.update(evaluate(maps, y, train, rng, seed))
    # Diagnostic (added after the first run): vertical-to-horizontal energy ratio, coarse scales.
    g1 = np.array([down(m, 1) for m in maps])
    ratio = (g1[:, 16] + g1[:, 24]) / (g1[:, 20] + g1[:, 28])
    diag.append([ratio[y == c].mean() for c in range(4)])
    for name, v in res.items():
        results.setdefault(name, []).append(v)

print(f"\n{'features':28s} accuracy: seed 0, range over seeds 0-2   share of errors within the pair")
for name, v in results.items():
    a = [x[0] for x in v]; wp = [x[1] for x in v]
    print(f"{name:28s} {a[0]:.3f}   {min(a):.3f}-{max(a):.3f}" +
          ("" if np.isnan(wp[0]) else f"          {min(wp):.2f}-{max(wp):.2f}"))

d = np.array(diag)
print("\nvertical/horizontal energy ratio, two coarsest scales (range over seeds):")
print("  " + "   ".join(f"{cls} {d[:, c].min():.2f}-{d[:, c].max():.2f}" for c, cls in enumerate(CLASSES)))

# Experiment 2 (added after the first run): same composition, same boundary orientation,
# different vertical order of the bands. Chance is 0.5.
results2 = {}
for seed in range(3):
    rng = np.random.default_rng(100 + seed)
    imgs, y = [], []
    for c, cls in enumerate(["plaza", "terrace"]):
        for _ in range(PER_CLASS):
            imgs.append(render(rng, cls)[0]); y.append(c)
    y = np.array(y)
    train = np.tile(np.r_[np.ones(100, bool), np.zeros(100, bool)], 2)
    for name, v in evaluate([energy_maps(im) for im in imgs], y, train, rng, seed).items():
        results2.setdefault(name, []).append(v[0])
print("\nplaza vs terrace             accuracy: seed 0, range over seeds 0-2")
for name, a in results2.items():
    print(f"{name:28s} {a[0]:.3f}   {min(a):.3f}-{max(a):.3f}")

Output:

area fractions, mean (sd):  sky        brick      grass      gravel
  street                 0.19 (0.03)  0.53 (0.05)  0.00 (0.00)  0.28 (0.04)
  plaza                  0.19 (0.03)  0.54 (0.05)  0.00 (0.00)  0.27 (0.04)
  path                   0.20 (0.04)  0.00 (0.00)  0.53 (0.05)  0.27 (0.04)
  lawn                   0.20 (0.03)  0.00 (0.00)  0.53 (0.05)  0.27 (0.04)

features                     accuracy: seed 0, range over seeds 0-2   share of errors within the pair
layout oracle (region maps)  1.000   1.000-1.000
GIST 1x1 (32 dims)           0.998   0.988-0.998          1.00-1.00
GIST 2x2 (128 dims)          0.985   0.968-0.985          0.83-0.86
GIST 4x4 (512 dims)          1.000   1.000-1.000
BoW L=0 (64 dims)            0.998   0.998-1.000          1.00-1.00
BoW L=2 (1344 dims)          1.000   1.000-1.000

vertical/horizontal energy ratio, two coarsest scales (range over seeds):
  street 1.95-2.03   plaza 1.07-1.11   path 1.11-1.12   lawn 0.69-0.70

plaza vs terrace             accuracy: seed 0, range over seeds 0-2
GIST 1x1 (32 dims)           0.750   0.730-0.750
GIST 2x2 (128 dims)          1.000   1.000-1.000
GIST 4x4 (512 dims)          1.000   1.000-1.000
BoW L=0 (64 dims)            0.895   0.875-0.955
BoW L=2 (1344 dims)          1.000   1.000-1.000

The composition check confirms the design: within each pair the area fractions of sky, wall or hedge and gravel match to within 0.01 on average, and the layout oracle, which sees only where the regions are, is perfect, so layout alone decides the class. The plan, written before the first run, stated the hypothesis for the whole-image descriptors as “GIST 1x1 and BoW L = 0 identify the pair (texture content) but are near chance within a pair: overall accuracy about 50–65%”. The first run contradicted it: every descriptor in experiment 1 scored 97–100%, including the unlocalized ones. The plan’s other expectations, “GIST 4x4 and SPM L = 2 above 90%” and similar to each other, held, but without the contrast they were meant to show.

The diagnostic line explains why. A whole-image GIST still keeps orientation, and the layouts differ in the orientation of their region boundaries: the side walls of a street or path add long vertical edges, the bands of a plaza or lawn add horizontal ones. At the two coarsest scales the energy of the vertical-structure filter is about twice that of the horizontal one in streets and about 0.7 of it in lawns. This is consistent with Oliva and Torralba’s abstract, which estimates the spatial-envelope dimensions from spectral as well as coarsely localized information, and with the 45.3% that Lazebnik et al.’s orderless weak features reached on 15 real categories, far above the 6.7% of chance.

Experiment 2 asks the question experiment 1 could not: plaza against terrace, the same three bands with gravel above brick, so that composition and boundary orientation are the same and only the vertical order differs. Here the whole-image GIST falls to 73–75%, while any grid, even $2 \times 2$, is perfect, because the grid records which band lies where. The orderless bag of words does better than the orderless GIST (88–96%), presumably because its 64 words, learned on these images, include words for the specific boundary patterns (sky against brick, sky against gravel) whose frequencies differ between the two orders; the spatial pyramid removes the remaining errors. In these synthetic scenes, then, the grid adds most where layouts share their textures and edge directions, and little where the edge statistics already differ.

Two things changed after the first run, both decided after seeing its result and specified in the plan before they ran. The vertical-to-horizontal energy line was added to explain the first result without changing any setting; and experiment 2 was added as a separate question, with the same features, classifiers and sizes, its own random seeds and data generated after experiment 1, so the first experiment’s printed lines are unchanged. Its hypothesis, quoted from the plan, was “GIST 1x1 and BoW L=0 at least 10 points below GIST 4x4 and BoW L=2, which should be near 1.0”; it held for GIST and for two of the three seeds of the bag of words, whose $L = 0$ result is 4.5 to 12.5 points below $L = 2$. The first run took about two minutes for experiment 1 alone, longer than the plan’s estimate of 30–60 seconds; the example was kept at the planned size.

Datasets and Benchmarks

The four benchmarks most often reported, with the protocols their sources define:

Benchmark	Categories	Images	Protocol	Metric
15-Scene (Lazebnik et al., 2006)	15: 8 outdoor categories from Oliva and Torralba, 5 added by Fei-Fei and Perona, 2 by Lazebnik et al.	200–400 per category, about 300 × 250 pixels	100 training images per class, the rest for testing, ten random splits	mean per-class recognition rate
MIT Indoor-67 (Quattoni & Torralba, 2009)	67 indoor categories in five groups (store, home, public spaces, leisure, working place)	15,620; at least 200 pixels on the shorter side	80 training and 20 test images per class	mean of the confusion matrix’s diagonal
SUN397 (Xiao et al., 2010)	the 397 SUN categories with at least 100 unique photographs	108,754	ten published partitions of 50 training and 50 test images per class; results averaged over the partitions, with 1, 5, 10 or 20 training images taken as the first $n$ of the 50	mean per-class accuracy
Places365-Standard (Zhou et al., 2018)	365	about 1.8 million training images; validation and test sets with fixed images per class	official split; test labels withheld	top-1 and top-5 accuracy or error

• 15-Scene (also Scene-15 or the fifteen scene categories) grew in stages: the eight outdoor categories of Oliva and Torralba (tall building, inside city, street, highway, coast, open country, mountain, forest), five more from Fei-Fei and Perona (suburb, office, kitchen, living room, bedroom, completing their 13), and industrial and store from Lazebnik et al. (2006), who name the COREL collection, personal photographs and Google image search as the main sources. Results close to 90% had been reached with hand-designed feature combinations by 2010 (88.1% for the combined kernel of Xiao et al., 2010, Fig. 4a), so the benchmark separates little among modern methods. The image-size cue noted under Challenges is a reason to say whether images were cropped.
• MIT Indoor-67 was collected from Google and Altavista image search, Flickr and LabelMe. Only 80 + 20 images per class enter the standard protocol, 6,700 of the 15,620 images (our arithmetic). Quattoni and Torralba (2009) score by the mean of per-class accuracies, as Razavian et al. (2014) also do, and the benchmark’s difficulty comes from categories that share objects, such as different kinds of shop.
• SUN397 images were gathered by searching WordNet scene terms, keeping colour images of at least 200 × 200 pixels, checking each against a written category definition, and removing duplicates and degenerate images (black and white, very blurry, rotated, aerial). The project page distributes the ten partitions; Xiao et al. used only the first for their results visualization. Papers do not always use all ten: DeCAF averaged five, and CLIP evaluated one without naming it. The SUN database itself has grown: 899 categories and 130,519 images in the 2010 paper, 908 categories and 131,072 images in the 2016 journal version.
• Places365 is the large-scale benchmark; the Places article gives its splits, the disagreement over the validation set size (50 or 100 images per class), the top-5 evaluation code and the 2015–2017 challenges. Test labels are not released, and the Places article found the evaluation server not responding in October 2026, so results are often reported on the validation set.

Other benchmarks appear beside these: the SUN Attribute database (102 attributes on 14,340 images) for attribute recognition, UIUC Event8 for event scenes, and LSUN, whose scene classification challenge used ten scene categories.

Evaluation Metrics

Scene benchmarks report accuracy, in two forms that differ when classes have different numbers of test images:

• Mean per-class accuracy, the average of the confusion matrix’s diagonal, used by 15-Scene and MIT Indoor-67, and by DeCAF for SUN397, where every class has 50 test images and it equals plain accuracy. Lazebnik et al. (2006) chose it because the plain fraction of test images classified correctly is biased when test-set sizes differ between classes; Quattoni and Torralba (2009) give the same reason.
• Top-$k$ error, the fraction of test images whose true category is not among the $k$ highest-scoring predictions, used by Places with $k = 5$ because many scenes fit more than one name (the challenge organizers’ example, reported in the Places article, is a bar that can also be a restaurant). Zhou et al. (2018, §5.1) consider top-1 accuracy possibly ill-defined for many scene categories.

Results are comparable only within one protocol: the same benchmark, number of training images per class, partitions and image preprocessing (cropping, colour or grayscale). Results on classical benchmarks are also sensitive to the pretraining data of the features, ImageNet, Places or both, which several papers vary in one table. Attribute recognition is scored by average precision per attribute, averaged over attributes, and Patterson and Hays (2012) balance positives and negatives so that rare and common attributes are comparable.

Applications

The sources name these uses: context for object detection and search, since a scene’s category predicts which objects are likely and where (Lazebnik et al., 2006, give the example that a highway makes a car likely and a toaster unlikely); place and object recognition in a context-based vision system by Torralba et al., and scene completion in computer graphics by Hays and Efros, both cited by Quattoni and Torralba (2009) as applications of outdoor scene recognition; and features for other scene tasks through transfer learning, the motivation for Places. Scene benchmarks also serve as a test of how well general-purpose image representations transfer, which is why SUN397 appears in the evaluation suites of DeCAF and CLIP.

Open Problems

• One label for a scene with several. All four benchmarks assign one category per image, though Xiao et al. (2010) and Patterson and Hays (2012) argue that the space of scenes is continuous and that images can show several scenes; scene detection and attributes address this, but the common benchmarks do not score it.
• Indoor and functional categories. Categories defined by function (a nursery against a bedroom, a deli against a bakery) depend on small objects and arrangements; across all the results above, indoor and shopping categories remain among the hardest.
• Comparisons across pretraining data. Gains on SUN397 and MIT Indoor-67 mix changes of method with changes of pretraining data (ImageNet, Places, web-scale image–text pairs) and of network depth, as Herranz et al.’s table shows by varying both. In our reading, overlap between web-scale pretraining data and benchmark images makes this harder still; CLIP’s authors measured it, but most papers that report these benchmarks do not.
• Human agreement as a ceiling. Machine accuracy on SUN397 has passed the reported human figures, which were measured with a single prototype image and a text label per category rather than training images (Xiao et al., 2010). By our reading, further gains on the benchmark may reflect learning the dataset’s category boundaries more than better scene understanding.