Pixel

A pixel, short for picture element, is one sample of a digital image: the value recorded at one position of the image’s grid. The same word also names the light-sensitive cells of an image sensor and the light-emitting cells of a display, and these meanings do not count the same things. Vision code must also decide where, in continuous coordinates, a pixel sits.

Definition

In an image, a pixel is the sample at one grid position, holding one value per channel. Digital Image defines the image as such an array and explains why a pixel is a point sample rather than a small square of colour, the argument of Alvy Ray Smith’s 1995 memo. On a sensor, “pixel” names a photosite, the cell that converts light to charge. On a display, it names the element that shows one image value; Lyon reports that the display industry calls the separate red, green and blue elements of a panel sub-pixels.

Where the Word Came From

Richard Lyon’s history of the term (2006) is the source for this section; the works he cites have not been read here, so each statement is as Lyon reports it. The phrase “picture element” came from early television, and by the 1940s it already had two senses: Albert Rose took it as an element of arbitrary size, Otto Schade as the smallest detail an imaging process can resolve. Lyon notes that the tension between an arbitrary element and a resolution element persists with “pixel”.

The Oxford English Dictionary, in the 2002 online edition Lyon cites, dated “pixel” to a 1969 Science paper on Mariner 6 images. Lyon found earlier uses: Fred Billingsley of Caltech’s Jet Propulsion Laboratory used it in two papers given at SPIE meetings in 1965 (Lyon’s abstract says both were published that year, but his reference list dates the proceedings volume of the second paper January 1967), and Billingsley recalled that the word came from a subcontractor, probably the Link Division of General Precision. The alternative “pel” was coined in 1964 at the request of William Schreiber of MIT, who published it in the Proceedings of the IEEE in 1967. Both terms circulated in image processing and video coding for more than a decade before three textbooks titled Digital Image Processing spread “pixel” in the late 1970s: Gonzalez and Wintz (1977), Pratt (1978) and Castleman (1979). Lyon argues that the image processing and artificial intelligence communities propagated the word, not computer graphics, where “picture element” already meant a drawn primitive such as a line or a polygon.

Sensor and Display Pixels

The sensor meaning came later. Lyon reports that the unit cell of a solid-state image sensor was called a picture element by RCA authors in 1969, and that a 1975 patent application by Lampe and White of Westinghouse is the first use he found of “pixel” for an element of sensor hardware; through the 1970s the word mostly meant the information element, not the cell that measures it. In 1982 Aoki et al. of Hitachi applied “pixel” to the single-colour elements of a sensor under a colour filter array, the usage Lyon called prevalent in 2006. A pixel of a Bayer-filtered sensor therefore measures one colour, while a pixel of the RGB file made from it by demosaicing holds three.

Displays have their own unit. Smith’s memo points out that the colour triads of a display screen are not pixels and do not map one-to-one onto them. Lyon’s conclusion records a trend among small LCD panels to count each red, green or blue element as a pixel, instead of calling them sub-pixels, and recommends resisting it.

Counting Pixels

A pixel count is the number of samples, and megapixels are millions of them. For a Bayer sensor, Lyon notes, the single-colour sensor elements happen to correspond one-to-one with the RGB pixels of the output file, so the two counts agree and cameras could be labelled by either. Two designs broke the coincidence, as he describes them. Fujifilm’s SuperCCD placed its sensor elements on a checkerboard, so the natural output had twice as many pixels as the sensor had elements; under pressure from the industry Fujifilm began to label its cameras primarily by sensor elements. The Foveon X3 sensor stacks three colour-sensing layers at each position: the Sigma SD9 and SD10 produced files of 3.4 million RGB pixels from 10.2 million elements. Lyon wrote as a Foveon employee, and the view that such a sensor should be rated by its total of 10.2 MP, described as 3.4 MP each of red, green and blue, is his and Foveon’s. His summary of practice in the camera field is that a camera’s pixel count is its number of sensor elements, and the output file’s pixel count is a separate specification.

Pixel Geometry

Three numbers describe a sensor pixel’s geometry. The pixel pitch is the centre-to-centre spacing of photosites. It is not the size of the light-sensitive area: the fill factor, the light-sensitive fraction of the pixel, belongs to Image Sensor, which treats it with microlenses and quantum efficiency. The pixel aspect ratio is the ratio of the pitches in $x$ and $y$; when it is not 1, the focal lengths in pixels differ, $f_x = f/p_x \neq f_y = f/p_y$, as in Camera Parameters. In our derivation, by the small-angle approximation a pixel of pitch $p$ behind a lens of focal length $f$ focused at infinity subtends an angle of about $p/f$ radians, the reciprocal of the focal length in pixels; Focal Length and Field of View turns the same ratio into the angle of view.

Where Is the Pixel Centre?

To compute with positions between samples, code must place the samples in continuous coordinates. One convention, OpenCV’s as Camera Parameters notes and the one used here, puts the centre of pixel $k$ at $x = k$, so that the pixel’s area is $[k - \tfrac12, k + \tfrac12]$ and an image of $N$ pixels spans $[-\tfrac12, N - \tfrac12]$. Digital Image notes that libraries also differ on whether centres sit at integers or half-integers; Smith argues for integers. Even with centres at integers, a resize from $N$ to $N'$ pixels needs a second choice, of which positions to keep aligned:

$$x' = (x + \tfrac12)\,\frac{N'}{N} - \tfrac12 \quad \text{(outer pixel edges aligned)}, \qquad x' = x\,\frac{N' - 1}{N - 1} \quad \text{(first and last pixel centres aligned)}.$$

They agree at the image centre, $x = (N - 1)/2$, and differ linearly away from it, by $\pm(s - 1)/2$ output pixels at the first and last pixel centres for a factor $s = N'/N$. Camera Parameters uses the first rule to rescale the principal point. In SciPy 1.16.3, scipy.ndimage.zoom chooses with its grid_mode argument: by default (False) it zooms the distance between pixel centres, so that, in the docstring’s example, a signal of 5 samples has length 4; with True it uses the full pixel extent, length 5. In the source, the input-to-output factor is $(N - 1)/(N' - 1)$ in the first case and $N/N'$ in the second. scikit-image 0.25.2’s skimage.transform.resize calls zoom with grid_mode=True, after an optional Gaussian prefilter that it applies by default when shrinking a non-boolean image (but not an integer image resized with order=0), with $\sigma = \max(0, (r - 1)/2)$ for the input-to-output size ratio $r = N/N'$, so $\sigma = 0$ when enlarging.

Example

The script below renders one Gaussian blob ($\sigma = 1.5$ pixels) on a $64 \times 64$ image at the known position $(x_0, y_0) = (12.3, 50.6)$, with $x$ the column and $y$ the row, measured from the centre of the top-left pixel. Each pixel’s value is the blob’s exact integral over that pixel’s area. The image is resized by factors 2, 3 and 0.5 with zoom in both grid modes and with resize with anti-aliasing off and on, and the blob is located after each resize by its intensity centroid, the ratio of first to zeroth image moments. The blob sits near opposite edges in $x$ and $y$, away from the centre where the rules agree.

The interpolation orders are each library’s default: cubic splines for zoom, linear for resize. The hypotheses, with their tolerances, were written before any script ran. H1: zoom(grid_mode=True) and both resize variants match the edge-aligned prediction within 0.01 pixel on both axes at every factor. H2: default zoom matches the corner-aligned prediction within 0.01 pixel. H3: the measured shift between the two zoom modes equals the predicted difference within 0.01 pixel. The checks: C1, the rendered image sums to 1 and has its centroid at $(x_0, y_0)$ within $10^{-9}$; C2, the output sizes are 128, 192 and 32; C3, resize equals a hand-written bilinear interpolation at the edge-aligned coordinates, and C4, each zoom mode equals map_coordinates at that mode’s coordinates, within $10^{-12}$. Runs use NumPy 2.3.5, SciPy 1.16.3 and scikit-image 0.25.2, and take well under a second.

import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ[v] = "1"                    # one BLAS thread, set before NumPy is imported
import numpy as np
from scipy import ndimage
from scipy.special import ndtr
from skimage.transform import resize

N, SIGMA, X0, Y0 = 64, 1.5, 12.3, 50.6     # x = column, y = row, origin at the top-left pixel centre
FACTORS = (2, 3, 0.5)

def box_gauss(u, c):
    """Gaussian of width SIGMA centred at c, integrated over the pixel [u - 1/2, u + 1/2]."""
    return ndtr((u + 0.5 - c) / SIGMA) - ndtr((u - 0.5 - c) / SIGMA)

def centroid(img):
    i, j = np.indices(img.shape)
    return np.array([(j * img).sum(), (i * img).sum()]) / img.sum()   # (x, y)

px = np.arange(N)
img = np.outer(box_gauss(px, Y0), box_gauss(px, X0))   # exact integral over each pixel's area

def half(x, s, n):   return (x + 0.5) * s - 0.5          # pixel edges map to pixel edges
def corner(x, s, n): return x * (n - 1) / (N - 1)         # first and last centres map to each other

METHODS = {   # name: (resize function, convention it should follow)
    "zoom, grid_mode=False": (lambda a, s, n: ndimage.zoom(a, s), corner),
    "zoom, grid_mode=True": (lambda a, s, n: ndimage.zoom(a, s, grid_mode=True, mode="grid-constant"), half),
    "resize, anti_aliasing=False": (lambda a, s, n: resize(a, (n, n), anti_aliasing=False), half),
    "resize, anti_aliasing=True": (lambda a, s, n: resize(a, (n, n), anti_aliasing=True), half),
}

# ---- hand-written references for the library cross-checks (C3, C4) ----
def mirror(u):
    u = np.where(u < 0, -u, u)
    return np.where(u > N - 1, 2 * (N - 1) - u, u)

def bilinear(a, u):   # separable linear interpolation at input coordinates u on both axes, mirror boundary
    out = a
    for axis in (0, 1):
        v = mirror(u)
        j0 = np.clip(np.floor(v).astype(int), 0, N - 2)
        t = (v - j0).reshape((-1, 1) if axis == 0 else (1, -1))
        out = (1 - t) * np.take(out, j0, axis=axis) + t * np.take(out, j0 + 1, axis=axis)
    return out

def spline_at(a, u, mode):
    gy, gx = np.meshgrid(u, u, indexing="ij")
    return ndimage.map_coordinates(a, [gy, gx], order=3, mode=mode)

c0 = centroid(img)
ok = {"C1": abs(img.sum() - 1) < 1e-9 and np.abs(c0 - (X0, Y0)).max() < 1e-9}
print(f"C1 input: sum - 1 = {img.sum() - 1:.1e}, centroid error ({c0[0] - X0:.1e}, {c0[1] - Y0:.1e})")
ok.update(C2=True, C3=True, C4=True, H1=True, H2=True, H3=True)

for s in FACTORS:
    n = int(round(N * s))
    print(f"\ns = {s}: {N} -> {n} px; predicted x' half-pixel ({half(X0, s, n):.4f}, {half(Y0, s, n):.4f}), "
          f"corner ({corner(X0, s, n):.4f}, {corner(Y0, s, n):.4f})")
    cents = {}
    for name, (fn, rule) in METHODS.items():
        out = fn(img, s, n)
        ok["C2"] &= out.shape == (n, n)
        c = centroid(out)
        cents[name] = c
        err = c - (rule(X0, s, n), rule(Y0, s, n))
        key = "H2" if rule is corner else "H1"
        ok[key] &= np.abs(err).max() < 0.01
        print(f"  {name:28s} {str(out.shape):11s} centroid ({c[0]:8.4f}, {c[1]:8.4f})  "
              f"error vs {rule.__name__:6s} ({err[0]:+.1e}, {err[1]:+.1e})")
    pred = np.array([half(X0, s, n) - corner(X0, s, n), half(Y0, s, n) - corner(Y0, s, n)])
    meas = cents["zoom, grid_mode=True"] - cents["zoom, grid_mode=False"]
    ok["H3"] &= np.abs(meas - pred).max() < 0.01
    print(f"  grid_mode True - False: predicted ({pred[0]:+.4f}, {pred[1]:+.4f}), measured "
          f"({meas[0]:+.4f}, {meas[1]:+.4f}), discrepancy ({meas[0] - pred[0]:+.1e}, {meas[1] - pred[1]:+.1e})")
    # library cross-checks against coordinates computed by hand
    u_half = (np.arange(n) + 0.5) * N / n - 0.5
    u_corner = np.arange(n) * (N - 1) / (n - 1)
    d3 = np.abs(resize(img, (n, n), anti_aliasing=False) - bilinear(img, u_half)).max()
    d4a = np.abs(ndimage.zoom(img, s) - spline_at(img, u_corner, "constant")).max()
    d4b = np.abs(ndimage.zoom(img, s, grid_mode=True, mode="grid-constant")
                 - spline_at(img, u_half, "grid-constant")).max()
    ok["C3"] &= d3 < 1e-12
    ok["C4"] &= d4a < 1e-12 and d4b < 1e-12
    print(f"  C3 resize vs hand bilinear at half-pixel coords: max |diff| {d3:.1e}")
    print(f"  C4 zoom False vs map_coordinates at corner coords: {d4a:.1e}; "
          f"zoom True vs map_coordinates at half-pixel coords: {d4b:.1e}")

print()
for k in ("C1", "C2", "C3", "C4", "H1", "H2", "H3"):
    print(f"{k}: {'held' if ok[k] else 'failed'}")

Output:

C1 input: sum - 1 = 0.0e+00, centroid error (0.0e+00, -7.1e-15)

s = 2: 64 -> 128 px; predicted x' half-pixel (25.1000, 101.7000), corner (24.7952, 102.0032)
  zoom, grid_mode=False        (128, 128)  centroid ( 24.7952, 102.0032)  error vs corner (+5.9e-07, -5.9e-07)
  zoom, grid_mode=True         (128, 128)  centroid ( 25.1000, 101.7000)  error vs half   (-1.4e-11, +8.9e-12)
  resize, anti_aliasing=False  (128, 128)  centroid ( 25.1000, 101.7000)  error vs half   (-9.3e-13, +5.7e-13)
  resize, anti_aliasing=True   (128, 128)  centroid ( 25.1000, 101.7000)  error vs half   (-9.3e-13, +5.7e-13)
  grid_mode True - False: predicted (+0.3048, -0.3032), measured (+0.3048, -0.3032), discrepancy (-5.9e-07, +5.9e-07)
  C3 resize vs hand bilinear at half-pixel coords: max |diff| 1.4e-17
  C4 zoom False vs map_coordinates at corner coords: 1.6e-16; zoom True vs map_coordinates at half-pixel coords: 0.0e+00

s = 3: 64 -> 192 px; predicted x' half-pixel (37.9000, 152.8000), corner (37.2905, 153.4063)
  zoom, grid_mode=False        (192, 192)  centroid ( 37.2905, 153.4063)  error vs corner (+5.0e-07, -4.8e-07)
  zoom, grid_mode=True         (192, 192)  centroid ( 37.9000, 152.8000)  error vs half   (-2.2e-11, +1.4e-11)
  resize, anti_aliasing=False  (192, 192)  centroid ( 37.9000, 152.8000)  error vs half   (-1.3e-12, +7.7e-13)
  resize, anti_aliasing=True   (192, 192)  centroid ( 37.9000, 152.8000)  error vs half   (-1.3e-12, +7.7e-13)
  grid_mode True - False: predicted (+0.6095, -0.6063), measured (+0.6095, -0.6063), discrepancy (-5.0e-07, +4.8e-07)
  C3 resize vs hand bilinear at half-pixel coords: max |diff| 1.7e-16
  C4 zoom False vs map_coordinates at corner coords: 1.2e-16; zoom True vs map_coordinates at half-pixel coords: 3.8e-16

s = 0.5: 64 -> 32 px; predicted x' half-pixel (5.9000, 25.0500), corner (6.0524, 24.8984)
  zoom, grid_mode=False        (32, 32)    centroid (  6.0523,  24.8985)  error vs corner (-3.7e-05, +5.3e-05)
  zoom, grid_mode=True         (32, 32)    centroid (  5.9000,  25.0500)  error vs half   (+7.0e-06, -3.7e-06)
  resize, anti_aliasing=False  (32, 32)    centroid (  5.9000,  25.0500)  error vs half   (+2.8e-06, -1.5e-06)
  resize, anti_aliasing=True   (32, 32)    centroid (  5.9000,  25.0500)  error vs half   (+1.6e-06, -8.5e-07)
  grid_mode True - False: predicted (-0.1524, +0.1516), measured (-0.1523, +0.1515), discrepancy (+4.4e-05, -5.7e-05)
  C3 resize vs hand bilinear at half-pixel coords: max |diff| 1.7e-18
  C4 zoom False vs map_coordinates at corner coords: 2.4e-17; zoom True vs map_coordinates at half-pixel coords: 0.0e+00

C1: held
C2: held
C3: held
C4: held
H1: held
H2: held
H3: held

C1 held: the integrated blob’s centroid is where it was placed, to rounding error. C2, C3 and C4 held: resize is bilinear interpolation at the edge-aligned coordinates and each zoom mode is spline interpolation at its own coordinates. H1 and H2 held at every factor on both axes: the largest error against the matching rule is $5.3 \times 10^{-5}$ pixels, at $s = 0.5$, more than two orders of magnitude inside the tolerance. H3 held: the two zoom modes place the same blob $+0.3048$, $+0.6095$ and $-0.1523$ pixels apart in $x$ at the three factors, within $6 \times 10^{-5}$ of the prediction, with the opposite sign in $y$, where the blob lies on the other side of the centre. The spline zoom and the bilinear resize give the same centroid to four decimals whenever they share a convention: in this example the convention moves the result, and the interpolation order does not. Anti-aliasing changes nothing when enlarging, where its $\sigma$ is zero, and when shrinking ($\sigma = 0.5$) changes only digits far below the tolerance; in our reading, because a Gaussian kernel is symmetric about zero, and convolving with such a kernel leaves a centroid in place apart from boundary effects.

All bars were written before the planning script that computes the values a correct method reaches, so the outcomes are tests rather than illustrations; the batch plan gave no tolerance for H3, and its 0.01 pixel is ours. Process note: before any script ran, the hand-written bilinear reference was changed from clipping coordinates outside the image to mirroring them, the boundary rule that scikit-image’s mode='reflect' maps to in SciPy. The example departs from the batch plan’s description in six ways, each logged before the pilot: (1) the image size and blob position, which the plan left open, were fixed as above; (2) resize is run with anti-aliasing both off and on, since the plan only asked for it to be set explicitly; (3) interpolation orders and modes are library defaults, except that zoom(grid_mode=True) uses mode='grid-constant', the mode SciPy recommends with that setting; (4) H1–H3 are checked on both axes, and H3 has the tolerance above; (5) the checks C1–C4 were added; (6) the centroid is taken over the whole output image. The pilot ran this exact script, and the first run repeated it unchanged with identical output.

Sub-Pixel Positions

Because a pixel is a sample of a smooth signal, positions can be estimated more finely than the grid spacing, by interpolating between samples or by weighting coordinates with intensities as the example does. The Harris corner detector can refine its corners to sub-pixel accuracy, feature trackers refine displacements iteratively by solving a small linear system whose solution is not restricted to whole pixels, and image interpolation supplies the values between samples. Sub-pixel results are only meaningful when the coordinate convention is known: an estimate accurate to a hundredth of a pixel is offset by tenths after a resize under the other convention, as the example shows.

Common Misconceptions

• “A camera’s megapixels are the pixels in its files.” For Bayer sensors the counts agree by construction; Lyon’s SuperCCD and Foveon cases show that they need not.
• “Resize conventions only matter at the border.” The two rules differ everywhere except the image centre.
• “A display’s red, green and blue elements are pixels.” Smith argues that display triads are not pixels, and Lyon recommends calling a panel’s colour elements sub-pixels.

Where It Is Used

The intrinsic matrix converts metric image positions to pixel positions through the pitch (camera parameters). Image warping, image pyramids and every resize must fix the centre convention, and sampling and aliasing explains what a pixel’s area does to the signal before it is sampled. Detection and segmentation pipelines that resize boxes, masks or keypoints between the network’s input and the original image inherit the same choice, as do bounding box coordinate conventions.