Image Features

An image feature is a measurement computed from an image that is more useful for a task than the raw pixels: a corner’s position, a vector describing the texture around it, a histogram of the whole image’s colours, or the activations of a trained network. Features are what most vision systems actually compare. Two photographs of the same building share almost no pixel values, but they share corners, edge patterns and gradient statistics that a well-chosen feature keeps and that changes of lighting, zoom or viewpoint leave roughly intact. This article is the overview of the category: what kinds of features there are, the choices that separate them (local or global, sparse or dense, handcrafted or learned, and which changes they ignore), and what they are used for. The methods themselves are in the linked articles, beginning with feature detection and feature descriptors.

Definition

A feature is a function $f$ that maps an image, or a part of one, to a value that can be compared across images: a point, a vector of numbers, or a binary string. Two broad families exist.

• A local feature describes a small region. Tuytelaars and Mikolajczyk (2008) define it as an image pattern that differs from its immediate neighbourhood, usually through a change in intensity, colour or texture; it can be a point, an edge element or a small patch. A local feature has a location, and often a scale and orientation, and a descriptor computed from the region around that location.
• A global feature summarizes the whole image (or a whole detection window) in one vector, such as a colour histogram or the activations a network computes for the whole image. It has no location of its own.

Features in the local sense come in two parts that the literature keeps separate. A detector decides where to measure: it selects keypoints (also called interest points) whose position can be found again in other images. A descriptor decides what to record there: a fixed-length vector that should be similar for the same scene point in two images and different for different points. Some methods, such as SIFT, supply both; others pair one detector with another method’s descriptor.

Intuition

Consider finding a poster in a photograph of a cluttered room. One strategy is to summarize the whole photograph, for example by counting how many pixels have each colour, and compare that summary with the poster’s. The summary is cheap and does not care where the poster hangs or whether the photo is rotated, but it also cannot say where the poster is, and the room’s wallpaper contributes to the count as much as the poster does.

The other strategy is to find a few dozen distinctive spots on the poster, such as the corner of a letter or the tip of a drawn leaf, describe the small patch around each, and look for the same patches in the room photo. If enough of them are found in a consistent arrangement, the poster is there, and their positions say exactly where and how it is rotated. A patch can be found even if the rest of the poster is hidden behind a chair.

These are the global and local strategies. Most of the history of image features is about making the second strategy reliable when the camera moves, zooms and the light changes, and then about learning both kinds from data instead of designing them.

Formal Definition

Write a local feature as a keypoint $k = (\mathbf{x}, \sigma, \theta)$, with position $\mathbf{x}$, scale $\sigma$ and orientation $\theta$, together with a descriptor $\mathbf{d} = f(I, k) \in \mathbb{R}^D$ (or $\{0,1\}^D$ for binary descriptors). A global feature is a single vector $\mathbf{g} = F(I)$. The precise forms of $f$ and of the distances used to compare descriptors are given in feature descriptors.

The language of invariance makes the design goal precise. Let $T$ be a transformation of the image, such as a rotation or a change of brightness. Following Tuytelaars and Mikolajczyk (2008):

• A function is invariant to $T$ if its value does not change: $F(T(I)) = F(I)$.
• A function is covariant with $T$ if transforming the input has the same effect as transforming the output. A detector is covariant with rotation if rotating the image rotates the detected keypoints with it, so the same points are found again.

Detectors aim for covariance and descriptors for invariance: the detector’s position, scale and orientation define a normalized patch, and the descriptor computed on that patch no longer depends on how the image was rotated or scaled.

Properties

Local and global features

Local features tolerate occlusion and clutter, because each is computed from a small region and an object is still found when only some of its features are visible. They also localize: a set of matched features gives the object’s position and pose. Schmid and Mohr (1997) made this argument when they introduced retrieval with local greyvalue invariants computed at Harris interest points. They described the existing appearance-based methods, colour histograms and eigenspace projections, as global, and as having difficulty with partial visibility and extraneous features; their local method retrieved objects from a database of more than a thousand images under partial visibility and similarity transformations.

Global features are compact and cheap to compare, one vector per image, which suits retrieval and classification over large collections. Their weakness is the one in the intuition above: Tuytelaars and Mikolajczyk (2008) note that a global feature cannot distinguish foreground from background and mixes information from both. Colour histograms, the survey’s typical example of a global feature for retrieval, work well for images with distinctive colours when the overall composition of the image is what matters. The bag of visual words bridges the two families: it computes local descriptors and then pools them into a global histogram of quantized descriptors, keeping some of their robustness while producing one vector per image. Learned global vectors are covered under embeddings.

Detectors and descriptors

Detectors look for structure whose position is well defined in both image directions. Corners qualify, and the Harris corner detector scores them from the local gradient structure; blobs qualify at a characteristic scale, which is how SIFT’s difference-of-Gaussians detector finds them; regions with stable boundaries qualify too. Uniform areas and straight edges do not, for the reason given by the aperture problem. Edges are features in their own right: they can be matched by orientation and profile and grouped into contours and lines, and Szeliski (2022, Chapter 7) treats them as complementary to keypoints and region descriptors.

Descriptors range from raw patches compared by normalized correlation, through gradient-orientation histograms such as SIFT’s 128 numbers, to binary strings compared by Hamming distance and vectors produced by neural networks. Their quality depends on the detector: a rotation-normalized descriptor computed with a wrong orientation describes the wrong patch. In Mikolajczyk and Schmid’s (2005) evaluation, however, the ranking of descriptors was mostly independent of the interest region detector they were computed on, and SIFT-based descriptors performed best.

Sparse and dense features

Sparse features are computed only at detected keypoints, typically hundreds or thousands per image; Lowe (2004) reported that a typical $500 \times 500$ image gives about 2,000 stable SIFT features. Sparse features are cheap to store and match and precisely localized, which is what geometric estimation needs.

Dense features are computed on a regular grid or at every pixel, without a detector. The histogram of oriented gradients tiles a whole detection window with gradient histograms, and every channel of a convolutional network’s feature map is a dense feature. Dense sampling covers low-texture regions, such as sky, that detectors skip, and for classification sampling enough patches turned out to matter more than where a detector would have placed them. Nowak, Jurie and Triggs (2006) found that, for bag-of-features image classification with moderate to large numbers of samples, uniform random sampling gave classifiers as good as or better than multiscale interest operators, and that the number of patches sampled from the test image was the single most important factor; the interest operators could not supply enough patches. At the other extreme, dense correspondence assigns a match to every pixel, as optical flow and stereo matching do, and detector-free matchers produce dense matches between arbitrary image pairs.

Handcrafted and learned features

Handcrafted features follow a design: the Harris response, SIFT’s scale space and histograms, HOG’s normalized cells, Haar-like features computed from integral images. Their invariances are built in by construction and can be analysed, and they need no training data.

Learned features are the output of a network trained on data. Razavian et al. (2014) took the 4,096-dimensional activations of one layer of the OverFeat network, trained for ImageNet classification, and fed them to a linear SVM on object, scene, fine-grained and attribute classification datasets. With simple augmentation (crops, rotations and mirror images of each training image), these off-the-shelf features beat the highly tuned non-network systems they compared with on most of their benchmarks (their Tables 1–6). Their abstract calls the results consistently superior, but the tables show exceptions: without augmentation the features reached 58.4% mean accuracy on MIT-67 indoor scenes against 64.0% for the best tuned system, and even with it they trailed one method on across-category attribute detection. Learned local features exist too. SuperPoint (DeTone et al., 2018) computes keypoints and descriptors for a whole image in one forward pass of a fully convolutional network, trained without manual labels. How such networks are trained, with labels, with self-supervised learning or against text, is the subject of representation learning.

The practical difference lies in where invariance comes from. A handcrafted descriptor is invariant to rotation because it is computed in a rotated frame; a learned feature is invariant to what its training data and augmentations varied, and to little else. It follows that learned features can, given suitable training data, cope with changes that have no clean mathematical model, and that their invariances have to be checked rather than assumed.

Invariances and What They Cost

Each invariance is bought with information. Tuytelaars and Mikolajczyk (2008) state the trade-off directly: increasing invariance typically reduces distinctiveness, because some of the image measurements are used up to remove the transformation’s degrees of freedom, and invariance that the application does not need can harm the results. The common invariances:

Change	How features handle it	What it costs
Illumination (gain and offset)	Use gradients, subtract the mean and normalize; SIFT also clips large entries	Absolute brightness and contrast are discarded
Rotation	Assign each keypoint a dominant orientation and describe the patch in that frame	Rotated patterns become identical (a “6” and a “9”); orientation errors under noise
Scale	Detect at a characteristic scale in a scale space	Depends on the scale estimate; fewer features at coarse scales
Viewpoint (affine)	Adapt each region to an ellipse and normalize it	More degrees of freedom removed, so less distinctive; more failure modes

Illumination. A brightness change that multiplies intensities by a gain and adds an offset is removed by working with gradients (which ignore the offset) and normalizing the descriptor to unit length (which removes the gain); Lowe (2004) shows that this makes the SIFT descriptor invariant to affine changes of illumination and adds a clipping step against non-linear effects such as camera saturation. What is lost follows from the construction (our reasoning, not a measurement in Lowe’s paper): the absolute level is gone, so after normalization a dark grey and a light grey surface with the same pattern look alike.

Rotation. A rotation-invariant descriptor is computed in a frame aligned with the patch’s dominant gradient direction. It can no longer tell a pattern from its rotated copy, and it inherits the noise in the orientation estimate. The example in feature descriptors measures this cost: rotation normalization rescues a rotated patch but makes the descriptor move more under noise. When the camera does not rotate about its optical axis, upright descriptors that skip this step can be preferable.

Scale. Scale invariance comes from the detector, which reports each keypoint at the scale where a normalized filter response peaks, and the descriptor is computed on a patch of that size. It is only as good as the scale estimate. Nowak, Jurie and Triggs (2006) also observed that most patches and interest points occur at the finest few scales, and questioned how scale-invariant bag-of-features methods were in practice.

Viewpoint. A change of viewpoint foreshortens a surface differently in different directions. Affine-covariant detectors adapt each region’s shape to compensate (see feature detection), at the cost of more degrees of freedom removed. Without them, viewpoint is handled only partly: Lowe (2004) reported that SIFT-based recognition identified textured planar surfaces reliably over a rotation in depth of up to 50 degrees, but 3D objects only up to about 30 degrees, with illumination change more disruptive for them.

Changes that are small or hard to model, such as noise, blur and compression, are handled by robustness instead: making the feature less sensitive rather than exactly invariant (Tuytelaars & Mikolajczyk, 2008). SIFT’s pooling of gradients over cells, which lets them shift by a few pixels without changing the descriptor much, is an example.

Historical Development

Interest points for tracking (1980s). Moravec’s interest operator, developed for a robot rover that navigated by vision, compared a window with copies shifted by one pixel horizontally, vertically and diagonally, and marked points where even the smallest change was a local maximum above a threshold (as described by Harris and Stephens). Harris and Stephens cite it from Moravec’s 1980 report; Tuytelaars and Mikolajczyk (2008) trace it to his conference papers of 1977 to 1981. Harris and Stephens (1988), working on 3D interpretation of image sequences, argued that tracking needs discrete features rather than a continuum of edge pixels, and replaced Moravec’s discrete shifts with an analytic expansion based on image gradients, which removed the operator’s dependence on direction. Tuytelaars and Mikolajczyk call the result, also known as the Plessey detector, probably the best known interest point detector.

Global appearance features (around 1990). Retrieval and recognition used global descriptions of appearance. Swain and Ballard’s colour indexing (1991) compared colour histograms, and Turk and Pentland’s eigenfaces (1991) and later Murase and Nayar (1995) projected whole images onto principal components; Tuytelaars and Mikolajczyk (2008) describe these appearance-based results as striking compared with the geometry-based approaches before them, but limited by clutter and occlusion to clean backgrounds or segmented objects.

Local invariant features (late 1990s and 2000s). Schmid and Mohr (1997) computed rotation-invariant differential descriptors at Harris points, with a multi-scale scheme for scale changes, and used them to retrieve images by voting; Tuytelaars and Mikolajczyk (2008) count it among the seminal works that turned sets of local features into an image representation. Lowe’s SIFT (1999, in full 2004) combined a scale-space detector with a gradient-histogram descriptor, and Lowe (2004) used it to recognize objects in clutter and under occlusion. Scale- and affine-covariant detectors, faster binary descriptors, and systematic evaluations followed, among them Mikolajczyk and Schmid’s (2005) comparison of descriptors.

Holistic, dense and pooled features (2000s). Alongside local matching, recognition used whole-image and pooled representations: Oliva and Torralba (2001) categorized scenes from a holistic representation, built from the energy spectrum and a coarsely localized windowed Fourier transform, without recognizing objects; HOG (2005) described detection windows densely; and visual words quantized local descriptors into histograms for retrieval and classification, where dense random sampling matched or beat keypoint detection once enough patches were sampled (Nowak et al., 2006).

Learned features (2010s onward). After a convolutional network won the ImageNet classification challenge in 2012 (see AlexNet), their activations proved to be strong generic features (Razavian et al., 2014), and instance retrieval shifted from handcrafted to CNN-based representations (Szeliski, 2022, Section 7.1.4). Learned local detectors, descriptors and matchers were developed as well; Szeliski (Section 7.1.2) notes that since about 2015 most new descriptors have been built with deep learning. Self-supervised training now produces features without labels. In our assessment, handcrafted local features remain a reasonable choice where speed, the lack of training data, or exact geometric behaviour matters.

Example

The example contrasts a global descriptor with local features on synthetic scenes. Two textured objects, a bright object A and a dark object B, are pasted into the same cluttered background with a known position and rotation. The global descriptor is a 32-bin grey-level histogram of the whole image, used raw and after standardizing the image to zero mean and unit variance, which cancels a global gain and offset. Each histogram is compared with that of a reference image (A, unrotated, at position (60, 80)) by the total variation distance, which runs from 0 for identical histograms to 1 for histograms with no overlap. The local features are scikit-image’s SIFT keypoints and descriptors: a cut-out of A is matched into each scene with the ratio test, and RANSAC fits a similarity transform to the matches, giving the object’s position and rotation.

import numpy as np
from scipy import ndimage
from skimage.feature import SIFT, match_descriptors
from skimage.measure import ransac
from skimage.transform import SimilarityTransform, warp

def shapes(seed, size, lo, hi):
    """Random filled rectangles and disks with grey values drawn from [lo, hi]."""
    rng = np.random.default_rng(seed)
    img = np.full((size, size), (lo + hi) / 2)
    yy, xx = np.mgrid[:size, :size]
    for _ in range(40):
        cy, cx = rng.integers(0, size, 2)
        r = rng.integers(4, size // 6)
        g = rng.uniform(lo, hi)
        if rng.random() < 0.5:
            img[(yy - cy) ** 2 + (xx - cx) ** 2 < r**2] = g
        else:
            img[max(cy - r, 0):cy + r, max(cx - r // 2, 0):cx + r // 2] = g
    img += rng.normal(0, 0.1, img.shape)  # fine surface texture
    return ndimage.gaussian_filter(img, 1.0)

obj_a = shapes(1, 128, 0.5, 1.0)  # a bright object
obj_b = shapes(2, 128, 0.0, 0.5)  # a dark object
background = shapes(3, 320, 0.2, 0.8)

def scene(obj, angle_deg, shift, gain=1.0, offset=0.0):
    """Paste obj into the background with a known rotation and shift, then relight."""
    T = SimilarityTransform(rotation=np.deg2rad(angle_deg), translation=shift)
    pasted = warp(obj, T.inverse, output_shape=background.shape, cval=-1.0)
    img = np.where(pasted >= 0, pasted, background)
    return gain * img + offset, T

def global_hist(img, standardize, bins=32):
    """Global descriptor: grey-level histogram of the whole image. Standardizing
    first removes a global gain and offset (an exposure change)."""
    if standardize:
        img = (img - img.mean()) / img.std()
        h, _ = np.histogram(img, bins=bins, range=(-3, 3))
    else:
        h, _ = np.histogram(img, bins=bins, range=(-0.2, 1.2))
    return h / h.sum()

def tv(p, q):
    return 0.5 * np.abs(p - q).sum()  # total variation distance, in [0, 1]

def locate(query, img, seed=0):
    """Local features: SIFT, ratio-test matching, RANSAC similarity transform."""
    feats = []
    for im in (query, img):
        s = SIFT()
        s.detect_and_extract(im)
        feats.append((s.keypoints[:, ::-1], s.descriptors))  # (row, col) -> (x, y)
    (k1, d1), (k2, d2) = feats
    m = match_descriptors(d1, d2, max_ratio=0.8, cross_check=True)
    if len(m) < 3:
        return len(m), 0, None
    model, inl = ransac((k1[m[:, 0]], k2[m[:, 1]]), SimilarityTransform, 2, 2.0,
                        max_trials=2000, rng=seed)
    return len(m), int(inl.sum()), model

reference, _ = scene(obj_a, 0, (60, 80))  # database photo for the global query
refs = [global_hist(reference, z) for z in (False, True)]

tests = {  # (object, rotation in degrees, object position, gain, offset)
    "A moved":           (obj_a, 0, (180, 170), 1.0, 0.0),
    "A rotated, relit":  (obj_a, 30, (150, 40), 0.7, 0.1),
    "B, same place":     (obj_b, 0, (60, 80), 1.0, 0.0),
    "B rotated, relit":  (obj_b, 30, (150, 40), 0.7, 0.1),
}
print(f"{'image':<18}{'raw d':>7}{'std d':>7}{'matches':>9}{'inliers':>9}"
      f"{'true x, y, angle':>19}{'found x, y, angle':>20}")
for name, (o, a, s, g, b) in tests.items():
    img, T = scene(o, a, s, g, b)
    d = [tv(global_hist(img, z), r) for z, r in zip((False, True), refs)]
    n, k, model = locate(obj_a, img)
    true = f"{T.translation[0]:.0f}, {T.translation[1]:.0f}, {np.rad2deg(T.rotation):.0f}"
    found = "no object"
    if k >= 4:
        found = (f"{model.translation[0]:.1f}, {model.translation[1]:.1f}, "
                 f"{np.round(np.rad2deg(model.rotation), 1) + 0.0:.1f}")
    print(f"{name:<18}{d[0]:>7.3f}{d[1]:>7.3f}{n:>9}{k:>9}{true:>19}{found:>20}")

Output (NumPy 2.3.5, SciPy 1.16.3, scikit-image 0.25.2; the script is deterministic, and the RANSAC sampling uses a fixed seed):

image               raw d  std d  matches  inliers   true x, y, angle   found x, y, angle
A moved             0.048  0.064       92       91        180, 170, 0   180.0, 170.0, 0.0
A rotated, relit    0.422  0.041       30       24        150, 40, 30   150.2, 39.7, 30.1
B, same place       0.158  0.451        5        2          60, 80, 0           no object
B rotated, relit    0.472  0.438        5        2        150, 40, 30           no object

The first row shows what a global descriptor cannot do. Moving object A by 150 pixels leaves both histograms almost unchanged, because a histogram records how much of each grey level the image contains and not where; the local features find the object at its new position to within a tenth of a pixel.

The two global columns show invariance as a design choice. The raw histogram calls the rotated and relit image of A less similar to the reference than the image of B in the same place, because the exposure change shifted every grey value; standardizing removes that change, and then both images of A are close to the reference and both images of B are far from it. The standardized histogram separates the images correctly, but its distances for B (about 0.44 to 0.45) are larger than the object’s share of the image alone would explain (the raw histogram puts the same replacement at 0.158, close to the object’s 16% of the image area): replacing the object changed the whole image’s mean and spread, so the standardization shifted the background’s histogram too. Foreground and background are mixed in one vector, as Tuytelaars and Mikolajczyk describe.

The local features needed no global normalization. SIFT keypoints are detected covariantly and described in a rotated, normalized frame, so 24 of the 30 matches into the rotated and relit scene agree on one similarity transform, which recovers the rotation and position within a few tenths of a degree and pixel. For object B only a handful of matches survive the ratio test; two matches always fit a similarity transform exactly, so two inliers are no evidence, and the code requires at least four before reporting an object.

Common Misconceptions

• “Features are corners.” Corners are one kind of local feature. Blobs, regions, edges, dense grids of descriptors, whole-image histograms and network activations are all features; which kind fits depends on whether the task needs positions, statistics or a single summary.
• “More invariance is better.” Every invariance discards information and adds a failure mode, such as a wrong orientation estimate. Tuytelaars and Mikolajczyk (2008) warn that invariance not adapted to the application can harm results; a tracker between video frames needs almost none.
• “A good detector makes a good feature for every task.” Detectors optimize repeatability, which matters for matching. For classification, Nowak, Jurie and Triggs (2006) found that dense random sampling matched or beat interest operators once enough patches were used.
• “Learned features are invariant to everything.” A network’s features ignore the variations its training covered. Rotation invariance, for example, is not guaranteed unless rotations were in the training data or built into the architecture.
• “Global features are obsolete.” The global vectors of modern retrieval and classification are learned embeddings, but they are still global: one vector per image, with the same blindness to position, which is why retrieval pipelines can re-rank their results with local features and a geometric check.

Where It Is Used

• Matching and alignment. Feature matching finds corresponding keypoints by descriptor distance and verifies them with RANSAC; the correspondences give homographies for panorama stitching and the transforms of feature-based image registration.
• Retrieval. Image retrieval ranks a collection against a query image, with local descriptors pooled into bags of visual words or with learned global embeddings, searched at scale with approximate nearest neighbor search.
• Recognition and detection. Lowe’s SIFT system could recognize a specific object from as few as three matched features; HOG and Haar-like features drove window-based object detection and face detection; convolutional networks now compute the features for image classification and detection.
• Tracking. Feature tracking follows corners through video with a local search such as Lucas–Kanade. Szeliski (2022, Section 7.1) contrasts the two ways of finding correspondences: tracking suits nearby viewpoints and video, while independent detection and matching suits large motion and appearance change.
• 3D vision. Structure from motion, SLAM and visual odometry turn matched or tracked features into camera poses and triangulated 3D points, and augmented reality relies on them to anchor virtual content to the scene.