Digital Image

A digital image is a finite array of numbers that records how much light reached each point of a regular grid. Almost everything in computer vision, from image gradients to deep networks, takes this array as its input, so the conventions for how it is sampled, quantized, laid out in memory, and indexed affect nearly every piece of vision code.

Definition

A digital image is a function of two discrete spatial coordinates whose values are drawn from a finite set. It is produced from a continuous light distribution by two steps: sampling, which measures the scene only at grid positions, and quantization, which rounds each measurement to one of a fixed number of levels. Each sample is a pixel (picture element). A pixel may hold one value (a grayscale image), three values (a color image), or more (a multispectral image); each value is a channel.

Intuition

Picture a camera sensor as a sheet of tiny light buckets laid out in rows and columns. During an exposure each bucket collects light, and afterwards the camera reads out a number for each one. The grid spacing decides how fine a detail the image can record. The number of levels in the readout decides how finely brightness differences are distinguished. The image is simply the table of those readings.

Formal Definition

Let $f(x, y)$ be the continuous irradiance on the image plane. Sampling on a grid with spacing $\Delta$ gives

$$I[i, j] = Q\big(f(j\Delta,\; i\Delta)\big), \qquad 0 \le i < H,\; 0 \le j < W,$$

where $i$ is the row index, $j$ the column index, $H$ and $W$ the height and width in pixels, and $Q$ a quantizer that maps real values to $2^b$ levels for a bit depth of $b$. In practice each sample integrates light over a small area of the sensor rather than reading $f$ at a single point, which acts as a low-pass filter before sampling. A color image adds a channel index, $I[i, j, c]$.

Properties

Pixels Are Samples

A pixel is a sample at a point, not a small square of color. Alvy Ray Smith made this argument in a 1995 Microsoft technical memo: an image is an array of point samples, and a continuous picture is recovered from them with a reconstruction filter. The squares seen when zooming into an image are an artifact of the display replicating each sample, not a property of the pixel. The distinction matters whenever an image is resized, rotated, or warped: new values must be interpolated between samples, and the result depends on the interpolation method.

Spatial Resolution

Spatial resolution is the number of samples, usually written as width × height (for example 1920 × 1080). A pixel count says nothing by itself about how much detail is captured; that also depends on the optics, focus, and motion blur. Sampling too coarsely relative to the detail in the scene causes aliasing, in which fine patterns masquerade as coarser ones. Sampling and aliasing covers when a grid is fine enough and how to prefilter before reducing resolution.

Bit Depth and Quantization

Bit depth is the number of bits per channel. Eight bits give 256 levels and are the norm for displayed images; camera raw files and medical or scientific images commonly use 10 to 16 bits, and high-dynamic-range work uses floating point. Too few levels produce visible banding (false contours) in smooth gradients. Quantization also adds a small error, often modeled as one source of image noise. The distribution of values over the available levels is summarized by the image histogram.

Channels

A grayscale image has one channel. A color image usually has three, red, green, and blue, whose meaning is fixed by a color space such as sRGB. Some formats add a fourth channel for transparency (alpha). Multispectral and hyperspectral images record many bands, including infrared, for remote sensing and material analysis, and depth images store distance rather than light.

Coordinates and Indexing

Two conventions coexist. Mathematical and geometric code writes positions as $(x, y)$, with $x$ increasing to the right. Arrays are indexed by $(i, j)$, row first. Most image libraries place the origin at the top-left corner with rows increasing downward, so $y$ points down, unlike the usual mathematical axes. NumPy arrays follow the array convention: image[row, column], that is, image[y, x]. Swapping the two is a frequent source of bugs. A second, subtler choice is whether a pixel’s center lies at integer coordinates or at half-integers; libraries differ, and the difference shifts results by half a pixel when resampling. Smith’s memo argues for placing samples at the integers.

Memory Layout and Data Types

A color image stored as height × width × channels is in HWC layout (also called channels-last); this is what most image I/O libraries return. Deep-learning frameworks such as PyTorch expect CHW (channels-first), with a batch dimension in front (NCHW). Converting between them is a transpose, not a reshape.

Images are usually stored as unsigned 8-bit integers (uint8, 0 to 255) but processed as floating point. Floats avoid two traps of integer arithmetic: overflow, where 250 + 10 wraps around to 4 in uint8, and truncation in division and filtering. The common convention divides by 255 to get values in $[0, 1]$; neural networks often go further and subtract a per-channel mean and divide by a standard deviation. Whichever range a function expects should be checked, because passing a $[0, 255]$ float array to code that expects $[0, 1]$ fails silently.

How a Camera Produces an Image

Light from the scene is focused by the lens onto a sensor, typically CMOS, whose photosites convert photons to electric charge. The charge is amplified and digitized by an analog-to-digital converter. How the lens maps scene points to sensor positions is the subject of image formation and camera models, covered separately.

A photosite measures only how much light arrived, not its color. Most color cameras therefore place a color filter array over the sensor so that each photosite sees only red, green, or blue light. The most common arrangement is the Bayer pattern, patented by Bryce Bayer of Eastman Kodak in 1976, in which green filters occupy every other position (half of all photosites) and red and blue share the rest. The patent gives the denser grid to the green, luminance-sensitive elements because human vision resolves luminance detail better than color detail. Each raw pixel thus holds one color value, and the two missing values are interpolated from neighbors in a step called demosaicing. The camera’s image signal processor then applies white balance, color correction, tone mapping, and gamma encoding before saving the result. A raw file stores the sensor values before most of these steps.

File Formats

Formats differ mainly in compression. Lossless formats such as PNG and TIFF reproduce the stored array exactly. Lossy formats such as JPEG discard information the eye is unlikely to notice, trading accuracy for size and introducing artifacts such as blocking and ringing near edges, which can accumulate when an image is re-encoded repeatedly. For datasets used to train or evaluate vision systems, compression is part of the data: a model trained on clean images can behave differently on heavily compressed ones.

Example

The NumPy example below builds a small RGB image, shows the indexing convention, converts between uint8 and float, and changes the layout.

import numpy as np

# An RGB image 4 pixels high and 6 wide, stored height x width x channels
img = np.zeros((4, 6, 3), dtype=np.uint8)
x, y = 4, 1                      # column 4, row 1
img[y, x] = [255, 0, 0]          # NumPy indexes [row, column]
print(img.shape, img.dtype)
print(img[y, x])

try:
    img[x, y]                    # (x, y) order: row 4 does not exist
except IndexError as e:
    print("IndexError:", e)

# uint8 -> float32 in [0, 1], and back
f = img.astype(np.float32) / 255.0
print(f[y, x], f.dtype)
back = np.round(f * 255.0).astype(np.uint8)
print(np.array_equal(back, img))

# uint8 arithmetic wraps around; float arithmetic does not
print(img[y, x] + np.uint8(10), f[y, x] + 10 / 255)

# HWC -> CHW, the layout PyTorch expects
chw = f.transpose(2, 0, 1)
print(chw.shape)

Output:

(4, 6, 3) uint8
[255   0   0]
IndexError: index 4 is out of bounds for axis 0 with size 4
[1. 0. 0.] float32
True
[ 9 10 10] [1.0392157  0.03921569 0.03921569]
(3, 4, 6)

Adding 10 to the saturated red value wraps it to 9 in uint8, while the float version exceeds 1.0 and can be clipped deliberately.

Common Misconceptions

• “A pixel is a little square.” It is a sample. Treating it as a square of constant color is one particular, crude reconstruction filter.
• “More megapixels means a sharper image.” Sharpness is limited by the lens, focus, motion, and noise as well as by the sampling grid.
• “Pixel values measure light linearly.” Most stored 8-bit images are gamma-encoded, so a value of 128 is far less than half the light of 255. Arithmetic that assumes linearity, such as averaging or blending, should first convert to linear values (color spaces explains the sRGB transfer function).
• “(x, y) and [i, j] are the same thing.” They are transposed: image[y, x] in NumPy.
• “A color camera measures three colors at every pixel.” Most measure one per pixel and interpolate the other two.

Where It Is Used

Every vision algorithm consumes digital images, and many design decisions trace back to the properties above. Linear filters such as convolution and image gradients operate on the sample grid; multi-scale methods such as the image pyramid change its resolution and must respect sampling and aliasing; geometric transformations rely on interpolating between samples; and the statistics of pixel values underlie the image histogram and models of image noise. Deep networks add their own conventions on layout and normalization, which must match between training and deployment.