Positional Encoding

A positional encoding tells a model where each token sits. Self-attention treats its input as an unordered set: shuffle the patches of an image and every output is shuffled the same way, so a transformer cannot tell a face from its patches rearranged. Vaswani et al. (2017) added fixed sinusoids to the input embeddings; vision models have since learned one vector per patch, encoded rows and columns separately, biased attention by the offset between patches, or rotated queries and keys by position. Fourier features of continuous coordinates, a related idea, help coordinate networks such as NeRF represent fine detail.

Definition

A positional encoding maps a token’s position $p$ (a sequence index, a row and column in a patch grid, or a continuous image coordinate) to a vector $\mathbf{e}(p) \in \mathbb{R}^d$ that is combined with the token’s content. Absolute encodings attach a vector to each position, usually by addition; relative encodings modify attention between two tokens according to their offset $p_j - p_i$.

Intuition

Drop the $14 \times 14$ patches of an image into a bag: attention can compare their appearance but not tell which was above which. A positional encoding writes an address on each patch first. Good addresses are similar for nearby patches and relate the same way everywhere, so what is learned at one place transfers. Sinusoids of several wavelengths act like clock hands: fast ones resolve small offsets, slow ones large ones.

Formal Definition

Why order must be injected. Stack $n$ tokens as the rows of $X \in \mathbb{R}^{n \times d}$, and let $f(X) = \operatorname{softmax}(XW_Q W_K^\top X^\top/\sqrt{d_k})\,XW_V$ with a row-wise softmax. For a permutation matrix $P$, the scores of $PX$ are $P S P^\top$, permuting rows and columns alike; the row-wise softmax commutes with this, and the values become $PXW_V$. Hence

$$f(PX) = P\,\operatorname{softmax}(S)\,P^\top P X W_V = P f(X),$$

since $P^\top P = I$. The MLP and layer normalization act on each token separately, so a whole encoder is permutation-equivariant and mean or max pooling over tokens is invariant. Adding a fixed encoding $E \in \mathbb{R}^{n \times d}$ breaks this, since the shuffled input is $PX + E$, not $P(X + E)$.

Sinusoidal encoding. Vaswani et al. (2017) used

$$PE_{(pos,\,2i)} = \sin\!\big(pos / 10000^{2i/d}\big), \qquad PE_{(pos,\,2i+1)} = \cos\!\big(pos / 10000^{2i/d}\big),$$

so each channel pair is a sinusoid of frequency $\omega_i = 10000^{-2i/d}$, with wavelengths from $2\pi$ to $10000 \cdot 2\pi$. The authors chose it because $PE_{pos+k}$ is a linear function of $PE_{pos}$ for any fixed offset $k$, which they hypothesized would make attending by relative position easy. The linear map rotates each pair:

$$\begin{pmatrix} \sin \omega_i(pos+k) \\ \cos \omega_i(pos+k) \end{pmatrix} = \begin{pmatrix} \cos \omega_i k & \sin \omega_i k \\ -\sin \omega_i k & \cos \omega_i k \end{pmatrix} \begin{pmatrix} \sin \omega_i pos \\ \cos \omega_i pos \end{pmatrix}.$$

The block-diagonal matrix $M_k$ of these rotations does not depend on $pos$, and $PE_{p} \cdot PE_{q} = \sum_i \cos \omega_i (p - q)$ depends only on the magnitude of the offset.

2D encodings for images. DETR encodes the row with $d/2$ sine and cosine channels and the column with the other $d/2$, independently, and concatenates them (Carion et al., 2020, arXiv appendix A.4). The official code first divides the 1-based indices by the height or width of the unpadded map and scales them to $(0, 2\pi]$, so the encoding describes relative location in the image.

Relative position representations. Shaw, Uszkoreit and Vaswani (2018) add learned vectors, indexed by the clipped offset, to the keys (and values) when token $i$ attends to token $j$:

$$e_{ij} = \frac{x_i W^Q \big(x_j W^K + a^K_{ij}\big)^\top}{\sqrt{d_z}}, \qquad a^K_{ij} = w^K_{\operatorname{clip}(j - i,\, k)}, \qquad \operatorname{clip}(x, k) = \max(-k, \min(k, x)),$$

so only $2k + 1$ offsets are learned. Swin Transformer instead adds a learned scalar bias per head inside each $M \times M$ window, $\operatorname{softmax}(QK^\top/\sqrt{d} + B)V$, with $B$ read from a table $\hat{B} \in \mathbb{R}^{(2M-1) \times (2M-1)}$ indexed by the vertical and horizontal offsets (Liu et al., 2021).

Rotary position embedding (RoPE). Su et al. (2021) seek query and key maps whose inner product depends only on content and the offset. Their solution rotates each channel pair of the projected query and key by an angle proportional to position, $f_{q}(x_m, m) = R_{\Theta, m} W_q x_m$, with frequencies $\theta_i = 10000^{-2(i-1)/d}$. Since $R_{\Theta,m}^\top R_{\Theta,n} = R_{\Theta,n-m}$,

$$q_m^\top k_n = x_m^\top W_q^\top R_{\Theta,\,n-m} W_k x_n ,$$

an absolute encoding of each token that yields a purely relative score.

Fourier features of coordinates. Tancik et al. (2020) map a continuous coordinate $\mathbf{v}$ to $\gamma(\mathbf{v}) = [\cos(2\pi B\mathbf{v}), \sin(2\pi B\mathbf{v})]$, with entries of $B \in \mathbb{R}^{m \times d}$ drawn from $\mathcal{N}(0, \sigma^2)$, before a multilayer perceptron. NeRF uses deterministic frequencies $2^0\pi, \dots, 2^{L-1}\pi$ per coordinate, with $L = 10$ for location and $L = 4$ for viewing direction (Mildenhall et al., 2020).

Properties

Fixed or learned matters little. Vaswani et al. (2017) found learned embeddings and sinusoids nearly identical in translation quality and kept the sinusoids for their untested promise of extrapolation (see the paper article). The Vision Transformer (ViT) learns one vector per patch in raster order. In its ablation (Dosovitskiy et al., 2021, arXiv v2, Appendix D.4: ViT-B/16, ImageNet 5-shot linear evaluation), accuracy was 61.4% without position embeddings, 64.2% with 1D embeddings, and 64.0% with either 2D embeddings (learned row and column halves, concatenated) or relative ones. The authors suggest that a $14 \times 14$ grid is easy to represent with any scheme, and show that learned 1D embeddings encode 2D topology anyway, with row and column structure.

Where it enters matters. DETR adds its fixed 2D encodings at every attention layer. In its ablation (ResNet-50, 300 epochs, median AP over the last 10 epochs on COCO 2017 validation), this reached 40.6 AP; passing sine encodings once at the input, as in the original transformer, gave 39.2, learned encodings at every layer 39.6, and no spatial encodings 32.8 (Carion et al., 2020, Table 3).

Relative encodings suit dense prediction. For Swin-T, the relative position bias raised ImageNet-1K top-1 accuracy by 1.2 points over no encoding and 0.8 over ViT-style absolute embeddings, and COCO box AP by 1.3 and 1.5 (Liu et al., 2021, Table 4). Absolute embeddings alone helped classification (+0.4%) but hurt object detection and semantic segmentation, which the authors take as evidence that some translation invariance remains useful for dense tasks.

Resolution changes. When ViT is fine-tuned at a higher resolution with the same patch size, the pre-trained embeddings are resampled by 2D interpolation according to their location in the image (Dosovitskiy et al., 2021, §3.2); the official JAX code uses scipy.ndimage.zoom with linear interpolation and keeps the class token’s embedding apart. Swin resizes its bias table bicubically for a different window size; clipped relative offsets apply to any length.

Extrapolation. Press, Smith and Lewis (2022) found that language models with sinusoids extrapolate poorly beyond their training length; their ALiBi instead penalizes each attention score linearly with query–key distance. Heo et al. (2024) applied 2D RoPE to ViT and Swin, either axial (frequencies split between column and row) or RoPE-Mixed (learnable frequencies combining both axes, so diagonal offsets are represented). ViTs trained on ImageNet-1k at $224 \times 224$ with either form outperformed learned absolute embeddings at larger test resolutions.

Fourier features and spectral bias. A multilayer perceptron fed raw coordinates learns fine detail very slowly; Tancik et al. (2020) explain why with neural tangent kernel theory, and show that random Fourier features give a stationary kernel whose bandwidth is set by $\sigma$: too small underfits, too large overfits, and the scale matters much more than the distribution’s shape. SAM’s prompt encoder uses this mapping for clicks and box corners (Kirillov et al., 2023), with $\sigma = 1$ on coordinates scaled to $[-1, 1]$ in the official code.

Examples

Permutation and position on image patches. The NumPy code below cuts a $32 \times 32$ synthetic image into 16 patches of $8 \times 8$ pixels and applies one self-attention layer with random weights to the patches in order and shuffled, with and without sinusoids added per slot.

import numpy as np

rng = np.random.default_rng(0)

# A 32x32 synthetic image: a bright square and a darker disc on a gradient background.
yy, xx = np.mgrid[0:32, 0:32]
img = 0.2 + 0.3 * xx / 31
img[4:14, 18:28] = 1.0                                   # square, top right
img[(yy - 22) ** 2 + (xx - 9) ** 2 < 36] = 0.6           # disc, bottom left

# Cut into a 4x4 grid of 8x8 patches, row by row: 16 tokens of 64 pixels.
patches = img.reshape(4, 8, 4, 8).transpose(0, 2, 1, 3).reshape(16, 64)

d = 32
W_e = rng.normal(0, 1 / 8, (64, d))                      # patch embedding
W_q, W_k, W_v = (rng.normal(0, 1 / np.sqrt(d), (d, d)) for _ in range(3))

def attention(X):
    Q, K, V = X @ W_q, X @ W_k, X @ W_v
    S = Q @ K.T / np.sqrt(d)
    A = np.exp(S - S.max(1, keepdims=True))
    return (A / A.sum(1, keepdims=True)) @ V

def sinusoid(n, d):
    """Vaswani et al. (2017): PE[pos, 2i] = sin(pos / 10000^(2i/d)), PE[pos, 2i+1] = cos(...)."""
    pos = np.arange(n)[:, None]
    omega = 1.0 / 10000 ** (np.arange(0, d, 2) / d)
    pe = np.zeros((n, d))
    pe[:, 0::2], pe[:, 1::2] = np.sin(pos * omega), np.cos(pos * omega)
    return pe

perm = rng.permutation(16)                               # shuffle the patches
X = patches @ W_e
E = sinusoid(16, d)                                      # one encoding per slot

out, out_shuffled = attention(X), attention(X[perm])
print("no positions:   max |f(PX) - P f(X)| =", f"{np.abs(out_shuffled - out[perm]).max():.1e}")
print("                mean-pooled output changes by", f"{np.abs(out_shuffled.mean(0) - out.mean(0)).max():.1e}")

# With positions: each slot keeps its encoding, so a patch moved to slot j gets E[j].
out, out_shuffled = attention(X + E), attention(X[perm] + E)
print("with positions: max |f(PX) - P f(X)| =", f"{np.abs(out_shuffled - out[perm]).max():.1e}")
print("                mean-pooled output changes by", f"{np.abs(out_shuffled.mean(0) - out.mean(0)).max():.1e}")
print("                (output entries have standard deviation", f"{out.std():.2f})")

no positions:   max |f(PX) - P f(X)| = 3.3e-16
                mean-pooled output changes by 2.2e-16
with positions: max |f(PX) - P f(X)| = 1.5e-01
                mean-pooled output changes by 5.4e-02
                (output entries have standard deviation 0.66)

Without encodings the identity holds to rounding error and the pooled descriptor is unchanged. With them, a moved patch is processed as if it belonged to its new slot.

Sinusoids on a patch grid. Continuing, the code builds the 1D sinusoid for 196 patches in raster order and two 2D encodings for the $14 \times 14$ grid, compares neighbor similarities, and checks $PE_{pos+k} = M_k PE_{pos}$ numerically.

def sine_2d_at(y, x, d, temperature=10000):
    """2D sine encoding at row coordinates y and column coordinates x: d/2 channels encode
    the row, d/2 the column, concatenated as in DETR. Returns (len(y), len(x), d)."""
    omega = 1.0 / temperature ** (np.arange(0, d // 2, 2) / (d // 2))
    def enc(t):
        e = np.zeros((len(t), d // 2))
        e[:, 0::2], e[:, 1::2] = np.sin(t[:, None] * omega), np.cos(t[:, None] * omega)
        return e
    return np.concatenate([np.repeat(enc(y)[:, None], len(x), 1), np.repeat(enc(x)[None, :], len(y), 0)], -1)

def sine_2d(h, w, d, normalize=True):
    """normalize=True scales coordinates to (0, 2*pi], as DETR's PositionEmbeddingSine(normalize=True);
    normalize=False uses integer patch indices, like the 1D sinusoid."""
    y, x = np.arange(h) + 1.0, np.arange(w) + 1.0
    if normalize:
        y, x = y / h * 2 * np.pi, x / w * 2 * np.pi
    return sine_2d_at(y, x, d)

def cos_sim(a, b):
    return (a * b).sum(-1) / (np.linalg.norm(a, axis=-1) * np.linalg.norm(b, axis=-1))

d, g = 64, 14
pe1 = sinusoid(g * g, d)                                  # 1D, patches in raster order
offsets = (0, 1, 2, 4, 8, 14, 28, 56)
print("1D sinusoid over 196 raster positions: cosine similarity between positions 98 and 98 + k")
print("  k:  " + " ".join(f"{k:5d}" for k in offsets))
print("  sim:" + " ".join(f"{cos_sim(pe1[98], pe1[98 + k]):5.2f}" for k in offsets))
print("  largest |sim(98, 98 + k) - sim(98, 98 - k)|:",
      f"{max(abs(cos_sim(pe1[98], pe1[98 + k]) - cos_sim(pe1[98], pe1[98 - k])) for k in range(1, 98)):.1e}")

print("\nSimilarity of patch (7, 7) to its neighbors      right  below  diagonal")
for name, pe in [("1D sinusoid, raster order", pe1.reshape(g, g, d)),
                 ("2D sine, normalized (DETR-style)", sine_2d(g, g, d)),
                 ("2D sine, integer coordinates", sine_2d(g, g, d, normalize=False))]:
    c = pe[7, 7]
    print(f"  {name:34s}  {cos_sim(c, pe[7, 8]):5.2f}  {cos_sim(c, pe[8, 7]):5.2f}  {cos_sim(c, pe[8, 8]):6.2f}")

pe2 = sine_2d(g, g, d, normalize=False)
print("\n2D sine, integer coordinates: similarity to (7, 7) at offset (k, k) and (0, k)")
print("  k:       " + " ".join(f"{k:5d}" for k in range(7)))
print("  diagonal:" + " ".join(f"{cos_sim(pe2[7, 7], pe2[7 - k, 7 - k]):5.2f}" for k in range(7)))
print("  row:     " + " ".join(f"{cos_sim(pe2[7, 7], pe2[7, 7 - k]):5.2f}" for k in range(7)))

# Relative-shift property: PE(pos + k) = M_k PE(pos), M_k block-diagonal 2x2 rotations.
omega = 1.0 / 10000 ** (np.arange(0, d, 2) / d)
def M(k):
    m = np.zeros((d, d))
    for i, w in enumerate(omega):
        c_, s_ = np.cos(w * k), np.sin(w * k)
        m[2 * i:2 * i + 2, 2 * i:2 * i + 2] = [[c_, s_], [-s_, c_]]   # acts on each (sin, cos) pair
    return m
print()
for k in (1, 14, 50):
    err = np.abs(pe1[k:] - pe1[:-k] @ M(k).T).max()
    print(f"k = {k:2d}: max |PE(pos + k) - M_k PE(pos)| over all positions = {err:.1e}")

1D sinusoid over 196 raster positions: cosine similarity between positions 98 and 98 + k
  k:      0     1     2     4     8    14    28    56
  sim: 1.00  0.97  0.88  0.75  0.70  0.63  0.49  0.55
  largest |sim(98, 98 + k) - sim(98, 98 - k)|: 6.7e-16

Similarity of patch (7, 7) to its neighbors      right  below  diagonal
  1D sinusoid, raster order            0.97   0.63    0.61
  2D sine, normalized (DETR-style)     1.00   1.00    0.99
  2D sine, integer coordinates         0.98   0.98    0.96

2D sine, integer coordinates: similarity to (7, 7) at offset (k, k) and (0, k)
  k:           0     1     2     3     4     5     6
  diagonal: 1.00  0.96  0.86  0.77  0.73  0.74  0.74
  row:      1.00  0.98  0.93  0.88  0.86  0.87  0.87

k =  1: max |PE(pos + k) - M_k PE(pos)| over all positions = 1.7e-14
k = 14: max |PE(pos + k) - M_k PE(pos)| over all positions = 1.6e-14
k = 50: max |PE(pos + k) - M_k PE(pos)| over all positions = 1.4e-14

Similarity falls over the first few offsets, then levels off non-monotonically (0.49 at 28, 0.55 at 56), and ignores the sign of the offset. In raster order the patch below is 14 positions away, so the 1D sinusoid rates it like a patch 14 columns away (0.63 against 0.97 for the right neighbor); 2D encodings treat both axes alike. With DETR’s normalization the fastest channel completes one period across the image, so neighbors are nearly indistinguishable by cosine similarity; by our reasoning, learned projections must amplify the small differences. $M_k$ reproduces every shift to rounding error.

Interpolating a position grid. Continuing, the integer-coordinate $14 \times 14$ grid stands in for the learned embeddings of a ViT trained at $224 \times 224$ with $16 \times 16$ patches, resampled to the $24 \times 24$ grid of $384 \times 384$ inputs. Because the stand-in is a known function, the result can be compared with the exact encoding.

from scipy.ndimage import zoom

grid14 = sine_2d(14, 14, d, normalize=False)     # stands in for a learned 14x14 position grid (224 px, 16 px patches)
# Fine-tuning at 384 x 384 with 16 px patches gives a 24 x 24 grid. zoom maps new index j to old index j * 13/23.
coord = 1 + np.arange(24) * 13 / 23
exact = sine_2d_at(coord, coord, d)

def similarity_matrix(pe):
    f = pe.reshape(-1, pe.shape[-1])
    f = f / np.linalg.norm(f, axis=1, keepdims=True)
    return f @ f.T

S_exact = similarity_matrix(exact)
for order, name in [(1, "bilinear"), (3, "bicubic")]:
    grid24 = zoom(grid14, (24 / 14, 24 / 14, 1), order=order)        # interpolate rows and columns, not channels
    rel = np.linalg.norm(grid24 - exact) / np.linalg.norm(exact)
    S = similarity_matrix(grid24)
    r = np.corrcoef(S.ravel(), S_exact.ravel())[0, 1]
    print(f"{name}: grid {grid24.shape}, relative error {rel:.1e}, "
          f"max |similarity error| {np.abs(S - S_exact).max():.1e}, correlation of 576x576 similarities {r:.5f}")
    norms = np.linalg.norm(grid24, axis=-1) / np.sqrt(d // 2)      # every exact encoding has norm sqrt(d/2)
    print(f"   vector norm relative to exact: mean {norms.mean():.4f}, min {norms.min():.4f}")
    c = grid24[12, 12]
    print(f"   similarity of (12, 12) to right / below / diagonal neighbor: "
          f"{cos_sim(c, grid24[12, 13]):.3f} / {cos_sim(c, grid24[13, 12]):.3f} / {cos_sim(c, grid24[13, 13]):.3f}")
c = grid14[7, 7]
print(f"original 14x14 grid, (7, 7) to right / below / diagonal: "
      f"{cos_sim(c, grid14[7, 8]):.3f} / {cos_sim(c, grid14[8, 7]):.3f} / {cos_sim(c, grid14[8, 8]):.3f}")

bilinear: grid (24, 24, 64), relative error 2.3e-02, max |similarity error| 2.9e-02, correlation of 576x576 similarities 0.99795
   vector norm relative to exact: mean 0.9931, min 0.9892
   similarity of (12, 12) to right / below / diagonal neighbor: 0.994 / 0.994 / 0.988
bicubic: grid (24, 24, 64), relative error 1.3e-02, max |similarity error| 1.3e-02, correlation of 576x576 similarities 0.99969
   vector norm relative to exact: mean 0.9997, min 0.9936
   similarity of (12, 12) to right / below / diagonal neighbor: 0.993 / 0.993 / 0.986
original 14x14 grid, (7, 7) to right / below / diagonal: 0.979 / 0.979 / 0.957

Both interpolations keep the similarity structure, and bicubic halves the error. Bilinear interpolation also shortens the vectors slightly; Touvron et al. (2021) found that this norm loss hurt DeiT’s accuracy without fine-tuning, and interpolate bicubically before fine-tuning. Neighbors become more similar (0.994 against 0.979) because they are 13/23 of an old patch apart.

Fourier features for one image row. Finally, a linear model predicts one row of a synthetic image from the pixel coordinate, fitted by ridge regression on the even pixels and tested on the odd ones, using the raw coordinate or 256 random Fourier features.

# One row of a synthetic image: a shading ramp, a dark object with sharp edges, a striped
# texture (period 6 px) and a thin bright line. Coordinates x in [0, 1).
n = 256
x = np.arange(n) / n
row = 0.3 + 0.2 * x
row[40:90] = 0.1                                                      # dark object
row[120:200] += 0.25 * (np.sin(2 * np.pi * np.arange(80) / 6) > 0)    # stripes
row[220:222] = 1.0                                                    # thin line
train, test = np.arange(0, n, 2), np.arange(1, n, 2)                  # fit even pixels, predict odd ones
stripes = test[(test >= 120) & (test < 200)]

def fit_predict(features, lam=0.1):
    """Ridge regression of pixel value on the features (plus a bias), fitted on the even pixels."""
    F = np.column_stack([np.ones(n), features])
    A = F[train]
    w = np.linalg.solve(A.T @ A + lam * np.eye(A.shape[1]), A.T @ row[train])
    err = (F @ w - row) ** 2
    return err[train].mean(), err[test].mean(), err[stripes].mean()

def rff(x, m, sigma, rng):
    """Random Fourier features (Tancik et al., 2020): [cos(2 pi b x), sin(2 pi b x)], b ~ N(0, sigma^2)."""
    b = rng.normal(0, sigma, m)
    return np.column_stack([np.cos(2 * np.pi * np.outer(x, b)), np.sin(2 * np.pi * np.outer(x, b))])

print(f"{'input':24s} {'train MSE':>9s} {'test MSE':>9s} {'stripes':>9s}")
print(f"{'raw coordinate x':24s}" + "".join(f"{e:10.4f}" for e in fit_predict(x)))
for sigma in (1, 10, 20, 50, 300):
    errs = fit_predict(rff(x, 256, sigma, np.random.default_rng(15)))
    print(f"{f'256 RFF, sigma = {sigma}':24s}" + "".join(f"{e:10.4f}" for e in errs))

input                    train MSE  test MSE   stripes
raw coordinate x            0.0175    0.0219    0.0346
256 RFF, sigma = 1          0.0079    0.0095    0.0193
256 RFF, sigma = 10         0.0056    0.0079    0.0200
256 RFF, sigma = 20         0.0011    0.0052    0.0100
256 RFF, sigma = 50         0.0000    0.0091    0.0163
256 RFF, sigma = 300        0.0000    0.0463    0.0479

On the raw coordinate a linear model can only fit a ramp; Fourier features make the same regression a sum of sinusoids. At $\sigma = 20$ (frequencies of about 20 cycles per row, against 43 for the stripes) the test error falls fourfold overall and more than threefold on the stripes. Larger $\sigma$ fits the training pixels exactly but interpolates worse, the underfitting-to-overfitting curve Tancik et al. report for MLPs. Over 20 draws of the frequencies, the test error was 0.0046–0.0081 at $\sigma = 20$ and 0.0052–0.099 at $\sigma = 50$; this draw is typical.

Common Misconceptions

• “Sinusoids let attention see relative position directly.” Their own dot products depend only on the offset, but attention compares $(x_i + e_i)W_Q$ with $(x_j + e_j)W_K$, mixing content and position terms. Relative and rotary schemes put the offset into the score explicitly.
• “Learned embeddings fix the resolution.” ViT and DeiT interpolate them to the new grid and then fine-tune.
• “CNNs know nothing about absolute position.” Islam, Jia and Bruce (2020) read absolute position out of standard CNN features and traced much of it to zero padding at the border; VGG16 without padding encoded much less (see convolutional neural network and receptive field).

Where It Is Used

• Backbones. ViT uses learned absolute embeddings, Swin a relative position bias, and 2D RoPE has been applied to both.
• Detection and segmentation. DETR adds 2D sine encodings at every attention layer, with learned object queries as output positional encodings; SAM encodes image positions and prompts with random Fourier features.
• Coordinate networks. NeRF and other networks mapping pixel or 3D coordinates to values use Fourier features, not to supply order but to fit high-frequency functions, as Mildenhall et al. (2020) note; see also the Fourier transform.