Vision Systems & Engineering
Edge AI Accelerators
The inside of the neural processing units and other fixed-function accelerators that run vision models on devices, covering arrays of multiply-accumulate units, systolic arrays, weight- and output-stationary dataflows, the on-chip memory hierarchy, tiling a convolution onto a fixed array, utilization lost to shape mismatch and fill and drain, operator coverage, and why a peak TOPS figure is not a throughput.
intermediate
An edge AI accelerator is a processor built to run the arithmetic of neural networks, mostly matrix products and convolutions in low-precision integers, at a higher rate per watt than a general-purpose CPU, on the device that captures the images. The family includes neural processing units (NPUs), digital signal processors with machine-learning extensions, and named blocks such as Google’s EdgeTPU and Apple’s Neural Engine. The edge inference article places them among the other classes of edge hardware, gives the roofline model that tells whether a layer is limited by arithmetic or by memory traffic, and describes the runtimes that dispatch work to them; quantization gives the integer arithmetic they compute. This article looks inside the accelerator: the array of multiply-accumulate units at its centre, the order in which data moves through that array (its dataflow), the on-chip memories around it, how a convolution is cut into pieces that fit it, and how much of its peak rate a layer actually uses.
Definition
An edge AI accelerator, in the sense used here, is a fixed-function or narrowly programmable processor whose datapath is an array of processing elements (PEs, as the Gemmini README calls them; Chen et al. say processing engines), each able to perform a multiply-accumulate (MAC), fed from explicitly managed on-chip memories and connected to main memory through a direct memory access (DMA) engine. Chen, Emer and Sze (2016) call accelerators of this kind spatial architectures: arrays of relatively simple PEs that exploit high parallelism through direct communication with one another, and that suit applications whose dataflow has producer–consumer relationships or can share data efficiently among a region of PEs. A systolic array, in our description, is a regular such design: a two-dimensional grid of identical PEs in which data passes only between neighbours, in step with the clock. At the heart of the Gemmini accelerator generator (ucb-bar, University of California), which is used as the running example below, lies such an array performing matrix multiplications (Gemmini README, version 0.7.2).
Intuition
The arithmetic of a convolutional layer is cheap; fetching its operands is not. Chen et al. (2016, §VI-C, Table IV) give the energy of one access at each level of an accelerator’s storage hierarchy, relative to a MAC, from a commercial 65 nm process:
| Level | Energy per access, relative to a MAC |
|---|---|
| Off-chip DRAM | 200× |
| Global buffer (more than 100 kB) | 6× |
| Array (from one PE to another, 1–2 mm) | 2× |
| Register file in the PE (0.5 kB) | 1× |
The model compression article quotes figures of the same kind for a 45 nm process. By that table, a MAC whose two operands both came from DRAM would spend hundreds of times more energy on the fetches than on the arithmetic (our arithmetic). An accelerator therefore tries to fetch each value from a distant memory once and use it many times from a near one. A 3 × 3 convolution offers plenty of reuse: each weight is applied at every output position, each input pixel contributes to nine output positions of every output channel, and each output sums contributions from every input channel and kernel offset. Chen et al. call the reuse of a weight across output positions and of an input pixel across kernel offsets convolutional reuse, the reuse of an input pixel across output channels ifmap reuse (input feature map), and the reuse of a weight across the images of a batch filter reuse; the outputs are built by accumulating partial sums (psums). No single arrangement captures all of them in the smallest memory, so designs choose which value stays put in each PE and which values move: the dataflow.
A systolic array makes the choice concrete. Picture a grid of PEs with one weight parked in each. Input values enter at the left edge and march one PE to the right per cycle; partial sums march one PE down per cycle, each PE adding its own product as they pass. A column then emits a finished dot product at the bottom every cycle. Nothing is fetched twice from far away while it is needed: each weight is read into its PE once, each input is read into the array once and then handed from neighbour to neighbour, and each partial sum passes from PE to neighbouring PE rather than to a buffer and back.
Formal Definition
Lower a convolution to a matrix product by im2col: each row of holds the input values under one output position, each column of holds one filter, and has one row per output position ( rows) and one column per output channel ( columns). Gemmini’s array computes on blocks of a fixed size, for a array, so a larger product is cut into tiles: , and along the three dimensions, with the matrices padded with zeros to and (and , here equal to ).
Weight-stationary (WS) array. PE holds the weight of one tile. Row of the array receives , element of input row , at cycle ; the input moves right and reaches PE at cycle . The partial sum moves down:
with , so that at cycle is , the contribution of this tile to . The delay of cycles on row is the skew that makes each input meet the right partial sum; Gemmini’s MeshWithDelays module implements it with shift registers at the array’s inputs (Gemmini README, “Systolic Array and Transposer”). Streaming rows through a loaded tile takes cycles: the array fills along a diagonal for cycles at the start and drains for at the end, during which some PEs are idle.
Output-stationary (OS) array. PE keeps the accumulator of one output . Row receives at cycle moving right, column receives at cycle moving down, and the two meet in PE at cycle . Streaming the whole reduction dimension takes cycles, after which the finished outputs are shifted out.
Utilization. With useful MACs and a run of cycles on PEs, the PE utilization is
In our derivation, if each tile adds a fixed overhead of cycles to a useful stream of length ( in WS, where all input rows stream through each weight tile; in OS), then
the product of a padding factor, which is below 1 whenever a dimension is not a multiple of (the factor belongs to OS, which tiles ; in WS, which streams the rows unpadded, it is 1), and a fill-and-drain factor . With the conventions of the example below (a -cycle weight preload in WS, a -cycle drain in OS, both not overlapped with the next tile), in both dataflows (our derivation: cycles at the edge plus of skew).
Peak rate. A array clocked at performs at most MACs per second. Counting a MAC as two operations (a multiplication and an addition), the peak in tera-operations per second (TOPS) is (our arithmetic; a vendor’s figure should be read with its own counting convention, number format and clock). A layer achieves times the peak at best, and the roofline of edge inference lowers that further when the layer is memory-bound. A TOPS figure is therefore a ceiling, not a throughput.
Properties
Dataflows
Chen et al. (2016, §IV) classify the dataflows of the accelerators before theirs by their data handling, chiefly what stays in each PE’s register file:
- Weight stationary (WS). Each filter weight stays in the register file of a PE, which runs through all the operations that use it before the weight is replaced. Weights are reused maximally; partial sums are passed to the neighbouring PE or stored back in the global buffer, and in Chen et al.’s implementation of WS the PE array works as a systolic array with little local control (§VI-A).
- Output stationary (OS). The accumulation of each output pixel stays in a PE, so partial sums never leave the register file until they are finished. Chen et al. distinguish three variants by whether the array covers one or many output channels and one or many pixels of an output plane (SOC-MOP, MOC-MOP and MOC-SOP). They note that one MOC-MOP design from the literature does not exploit convolutional reuse because it treats the convolution as a plain matrix multiplication (§IV-B, page 5 of the copy read), as, in our reading, an im2col lowering does: im2col copies each input pixel into up to rows of .
- No local reuse (NLR). PEs have no register file; all data is stored in a larger global buffer, and the array is used for ifmap reuse and psum accumulation only.
- Row stationary (RS), Chen et al.’s contribution. The convolution is broken into one-dimensional primitives, each convolving one row of filter weights with one row of input pixels into one row of partial sums, and each primitive is mapped to a PE, so that weights, input pixels and partial sums are all reused in the register file.
Chen et al. compare the dataflows with an analysis framework under equal area and parallelism: accesses at each level of the hierarchy are counted exactly for real layer shapes and weighted by Table IV. For the convolutional layers of AlexNet, with simulated arrays of 256, 512 and 1,024 PEs and batch sizes 1, 16 and 64, RS was 1.4 to 2.5 times more energy-efficient than the other dataflows, and in the fully connected layers (1,024 PEs) at least 1.3 times at a batch size of 16 (§VII-B, §VII-C); the abstract states the fully connected result “for batch size larger than 16”. WS, in their analysis, maximizes weight reuse but sacrifices ifmap reuse, because the number of filters loaded on chip at a time is limited by the global buffer’s room for partial sums and by the size of the PE array, which leads to high DRAM accesses. They built RS into the Eyeriss chip: 168 PEs with 0.5 kB of register file each, a 108 kB global buffer, 16-bit fixed-point arithmetic and a 200 MHz clock in 65 nm CMOS (Fig. 4). In their energy breakdown for RS, the convolutional layers spend most of their energy in register-file accesses and the fully connected layers in DRAM accesses, for lack of convolutional reuse.
Gemmini supports both WS and OS in one array and lets software pick between them at run time, or fixes one when the hardware is generated. In WS mode the values (the weights) are preloaded into the array while and are fed through; in OS mode is preloaded and and are fed through. Because weight-stationary dataflows require an accumulator outside the array, Gemmini adds an SRAM bank with adders (README, “Architecture”), and in WS mode is usually zero, since an 8-bit cannot hold partial sums accurately; the accumulator (32-bit in the default configuration) sums them instead (README, “Systolic Array and Transposer”). In our reading, the need comes from the WS array emitting the partial sums of only one tile at a time.
The memory hierarchy
The array is the smallest part of the design’s memory story. Gemmini, as its README describes it, stores the array’s inputs and outputs in an explicitly managed scratchpad of banked SRAMs, not a cache: software decides what lives there. A DMA engine moves data between main memory and the scratchpad, and between the accumulator and main memory, operating on virtual addresses through the accelerator’s own translation lookaside buffer. Each row of the scratchpad and of the accumulator is as wide as the array: in the default configuration, a array with 8-bit inputs and 32-bit accumulators, a scratchpad row holds bits and an accumulator row 512 bits. Values read out of the accumulator can be scaled from 32 bits back to 8 (by default a multiplication by a 32-bit floating-point scale) and passed through an activation function, the requantization step that the quantization article derives. The four levels of Chen et al.’s accelerator model are similar in kind: an off-chip DRAM, a global buffer (around 100–300 kB in the CNN accelerators of the time, they write), the network between PEs, and a register file, typically below 1 kB per PE (§II).
Since the array consumes data every cycle, the memories must deliver it in time. Gemmini decouples data movement from computation with separate queues for load, store and execute instructions, and its loop instructions for large matrix products and convolutions double-buffer tiles, so that the next tile is loaded while the current one is computed (README, “Matmul and Conv Loop Unrollers”). Chen et al. likewise find prefetching, double buffering, caching and pipelining quite effective at hiding the latency of data movement in CNN acceleration, so that they do not expect it to affect overall throughput significantly (§VI-B).
Tiling and shape mismatch
An array of fixed size only computes blocks of its own size. Gemmini’s array “can only operate on matmuls that are up to DIMxDIM elements large”, and larger products are tiled into a sequence of such blocks, either by the programmer or by the accelerator’s own loop instructions (README). When a layer’s dimensions are not multiples of , the last tile along that dimension is partly padding and the PEs that would have held it do no useful work. Small layers suffer in a second way: each tile pays the fill and drain of the array, and the shorter the stream through a loaded tile, the larger that fixed cost is in proportion. Both losses follow directly from the utilization formula above, and both depend on the layer’s shape; in our reading, this is one reason a network designed for one accelerator can run at a low fraction of peak on another. Chen et al.’s energy-delay comparison makes the same point about shape: at batch size 1, the number of active PEs in two of the OS variants is capped by the size of an output plane or by the number of output channels (§VII-B).
Operator coverage and fallback
An accelerator implements matrix products, convolutions and a few fused operations around them; the rest of a network runs elsewhere. Gemmini, for example, has peripheral circuitry for ReLU and ReLU6, for scaling results down by powers of two in quantized workloads, and for transposing matrices, performs max-pooling while it moves results out to main memory, and wraps its instructions in a C library for matrix multiplications, convolutions with or without pooling, and matrix additions (README). It is attached to a RISC-V CPU core and driven by custom instructions from it; in our reading, whatever the accelerator does not implement is left to that core. On phones, the edge inference article describes how an operation an accelerator does not support runs elsewhere, usually on the CPU, at the cost of a transfer, and how early phone accelerators handled only quantized or only floating-point models. Which operators an accelerator supports can decide a network’s design: as the MobileNet article reports, the MobileNetV4 authors found squeeze-and-excitation poorly supported on DSPs and slow on accelerators.
Examples
A systolic array, cycle by cycle
The example simulates a systolic array of int8 multipliers with int32 accumulators, stepping it one clock cycle at a time, in weight-stationary and output-stationary modes. The simulator is our reproduction of a documented mechanism, following the Gemmini v0.7.2 README’s description of MeshWithDelays (a mesh computing one row per cycle, inputs skewed by shift registers, weights preloaded in WS mode, an int32 accumulator outside the array) and Chen et al.’s definitions of WS and OS; it is not Gemmini’s RTL. Gemmini’s transposer, which its OS mode uses to reorder rows read from the scratchpad, is not modelled, and neither is its two-level hierarchy, in which each tile of PEs is fully combinational and only the mesh of tiles has pipeline registers between tiles: our simulation registers every PE, as if each tile held one PE. The outcomes test our reproduction, not Gemmini.
The workload is one convolution with zero padding of one pixel on a synthetic image: each channel is a horizontal ramp ( for column ) plus 100 inside a rendered shape (a disk, a rectangle and a triangle, one per channel), shifted by −128 into int8. im2col turns it into with output positions and ; the weights are seeded random int8 values, and the number of output channels is 8, 12, 16 or 20, on arrays of and , so that divides in some conditions and not in others. is padded to 32 in every condition.
The modelling choices that the README leaves open are fixed as follows:
- WS: for each tile and, inside it, each tile, the weight block is preloaded one row per cycle from the top ( cycles), then all rows of stream through ( cycles); the partial sums leaving the bottom of the array are added into the accumulator, and .
- OS: for each tile and tile, all tiles of the reduction stream through back to back, accumulating in place in the PEs ( cycles), as Gemmini’s
matmul.compute.accumulatedinstruction allows; then the results drain out one row per cycle ( cycles). - Counting: traffic is counted where values cross the array’s edge: weight values entering it, input values entering it, and partial sums leaving it for the accumulator (WS) or drained out (OS). Padded zeros count, since the hardware moves them; moves from one PE to its neighbour inside the array are not counted.
Whether the preload (WS) or the drain (OS) overlaps the previous tile is one of the open choices; since the README says Gemmini’s loop instructions double-buffer tiles, the code prints the utilization both ways: with these edge cycles counted, and with them hidden. The convolution’s output is checked against a direct integer convolution (C0), and every counter against its closed form (C1). The closed forms (the counts, the cycles per tile, , the ratios behind H1–H4b) were derived before any code ran and printed by a separate planning script before the pilot, so the example illustrates the dataflows’ accounting and checks it cycle by cycle against those formulas; it is not an empirical finding. The planned outcomes were:
- H1: in WS each real weight enters the array exactly once over the whole layer; H1b: in OS each enters times, once per tile.
- H2: WS moves exactly times as many partial sums as OS.
- H3: within each array size and dataflow, exactly: output-channel counts that does not divide lose utilization by the padding fraction and nothing else.
- H4: fill and drain lower utilization most for the smallest layer, ; H4b: OS loses more to fill and drain than WS.
The script uses NumPy 2.3.5 and scikit-image 0.25.2 (Python 3.13.9), runs in about a second, and printed identical output on two runs.
import os
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["VECLIB_MAXIMUM_THREADS"] = "1"
from fractions import Fraction as Fr
import numpy as np
from skimage.draw import disk, polygon, rectangle
# --- Data: a synthetic 3-channel 32x32 image of rendered shapes, as int8 ---
H = W = 32
masks = np.zeros((3, H, W), bool)
masks[0][disk((16, 12), 8, shape=(H, W))] = True # channel 0: disk
rr, cc = rectangle((5, 18), end=(26, 28), shape=(H, W))
masks[1][rr, cc] = True # channel 1: rectangle
rr, cc = polygon([4, 28, 28], [16, 4, 28], shape=(H, W))
masks[2][rr, cc] = True # channel 2: triangle
x = np.arange(W)[None, None, :]
img = ((4 * x + 100 * masks).astype(np.int16) - 128).astype(np.int8) # ramp + shape, shifted to int8
ok = img.dtype == np.int8 and img.shape == (3, H, W) and np.array_equal(
img.astype(np.int64), 4 * x + 100 * masks.astype(np.int64) - 128)
print(f"D0 data: int8 {img.shape}, every pixel = 4x + 100*mask - 128: {ok}, "
f"mask areas {masks.sum((1, 2)).tolist()}, range [{img.min()}, {img.max()}]")
M, K = H * W, 27 # GEMM rows (output pixels) and depth (3 channels x 3 x 3)
def im2col(img):
"""Rows m = y*32 + x, columns k = c*9 + dy*3 + dx; zero padding of 1."""
p = np.pad(img.astype(np.int32), ((0, 0), (1, 1), (1, 1)))
return np.stack([p[c, dy:dy + H, dx:dx + W].reshape(-1)
for c in range(3) for dy in range(3) for dx in range(3)], axis=1)
def direct_conv(img, w):
"""Direct 3x3 cross-correlation in int64 by loops over the nine offsets (no im2col)."""
p = np.pad(img.astype(np.int64), ((0, 0), (1, 1), (1, 1)))
out = np.zeros((w.shape[0], H, W), np.int64)
for dy in range(3):
for dx in range(3):
out += np.einsum("nc,chw->nhw", w[:, :, dy, dx].astype(np.int64),
p[:, dy:dy + H, dx:dx + W])
return out.reshape(w.shape[0], -1).T # (M, N), rows m = y*32 + x
def skew_rows(X, P):
"""S[t, i] = X[t - i, i] (zero outside): row i of the array gets column i of X, delayed i cycles."""
L = X.shape[0]
S = np.zeros((L + P - 1, P), X.dtype)
for i in range(P):
S[i:i + L, i] = X[:, i]
return S
def run_ws(A, Wm, P, N):
"""Weight-stationary: preload a PxP weight block, stream all M rows of A, sums flow down."""
Kp, Np = A.shape[1], Wm.shape[1]
C = np.zeros((M, Np), np.int32) # accumulator SRAM (adds in place)
cnt = dict(weights=0, inputs=0, psums=0, edge=0, stream=0)
per_w = np.zeros((K, N), np.int64) # loads of each real weight
for n0 in range(0, Np, P):
for k0 in range(0, Kp, P):
blk = Wm[k0:k0 + P, n0:n0 + P]
w = np.zeros((P, P), np.int32)
for c in range(P): # preload: one row enters the top per cycle
w[1:] = w[:-1].copy()
w[0] = blk[P - 1 - c]
cnt["weights"] += P; cnt["edge"] += 1
kk, nn = np.meshgrid(np.arange(k0, k0 + P), np.arange(n0, n0 + P), indexing="ij")
real = (kk < K) & (nn < N)
np.add.at(per_w, (kk[real], nn[real]), 1)
S = skew_rows(A[:, k0:k0 + P], P)
a = np.zeros((P, P), np.int32); s = np.zeros((P, P), np.int32)
for t in range(M + 2 * P - 2):
a[:, 1:] = a[:, :-1].copy() # inputs move right
a[:, 0] = S[t] if t < S.shape[0] else 0
if t < M + P - 1:
cnt["inputs"] += int(np.count_nonzero((t - np.arange(P) >= 0) & (t - np.arange(P) < M)))
prod = a * w
s_new = np.empty_like(s)
s_new[0] = prod[0]
s_new[1:] = s[:-1] + prod[1:] # partial sums move down
s = s_new
m = t - (P - 1) - np.arange(P) # row of C leaving each column's bottom
v = (m >= 0) & (m < M)
C[m[v], n0 + np.arange(P)[v]] += s[P - 1, v]
cnt["psums"] += int(v.sum()); cnt["stream"] += 1
return C, cnt, per_w
def run_os(A, Wm, P, N):
"""Output-stationary: each PE keeps one output; A moves right, W moves down; then drain."""
Kp, Np = A.shape[1], Wm.shape[1]
C = np.zeros((M, Np), np.int32)
cnt = dict(weights=0, inputs=0, psums=0, edge=0, stream=0)
per_w = np.zeros((K, N), np.int64)
ar = np.arange(P)
for m0 in range(0, M, P):
for n0 in range(0, Np, P):
SA = skew_rows(A[m0:m0 + P, :].T, P) # SA[t, i] = A[m0+i, t-i]
SB = skew_rows(Wm[:, n0:n0 + P], P) # SB[t, j] = W[t-j, n0+j]
a = np.zeros((P, P), np.int32); b = np.zeros((P, P), np.int32)
acc = np.zeros((P, P), np.int32)
for t in range(Kp + 2 * P - 2):
a[:, 1:] = a[:, :-1].copy(); a[:, 0] = SA[t] if t < SA.shape[0] else 0
b[1:] = b[:-1].copy(); b[0] = SB[t] if t < SB.shape[0] else 0
kin = t - ar # k index entering each row / column
vin = (kin >= 0) & (kin < Kp)
cnt["inputs"] += int(vin.sum()); cnt["weights"] += int(vin.sum())
real = vin & (kin < K) & (n0 + ar < N)
np.add.at(per_w, (kin[real], (n0 + ar)[real]), 1)
acc += a * b
cnt["stream"] += 1
for r in range(P): # drain: bottom row leaves each cycle
C[m0 + P - 1 - r, n0:n0 + P] = acc[P - 1]
acc[1:] = acc[:-1].copy(); acc[0] = 0
cnt["psums"] += P; cnt["edge"] += 1
return C, cnt, per_w
def closed(flow, P, N):
Tm, Tk, Tn = M // P, -(-K // P), -(-N // P)
Kp = Tk * P
if flow == "WS":
T, L = Tk * Tn, M
f = dict(weights=T * P * P, inputs=T * M * P, psums=T * M * P)
else:
T, L = Tm * Tn, Kp
f = dict(weights=T * P * Kp, inputs=T * P * Kp, psums=T * P * P)
f.update(edge=T * P, stream=T * (L + 2 * P - 2))
return f, T, L, Tk, Tn, Tm
res = {}
print("\n P flow Cout Tk Tn tiles cyc/tile cycles U U_hidden F pad(K,N) "
"C0 max|d| C1 weights inputs psums per-weight[min,max]")
for P in (8, 16):
for flow in ("WS", "OS"):
for N in (8, 12, 16, 20):
w = np.random.default_rng(0).integers(-128, 128, size=(N, 3, 3, 3)).astype(np.int8)
Tk, Tn = -(-K // P), -(-N // P)
A = np.zeros((M, Tk * P), np.int32); A[:, :K] = im2col(img)
Wm = np.zeros((Tk * P, Tn * P), np.int32); Wm[:K, :N] = w.reshape(N, -1).T
C, cnt, per_w = (run_ws if flow == "WS" else run_os)(A, Wm, P, N)
d = int(np.abs(C[:, :N].astype(np.int64) - direct_conv(img, w)).max())
f, T, L, Tk, Tn, Tm = closed(flow, P, N)
c1 = all(cnt[k] == f[k] for k in f)
tot = cnt["edge"] + cnt["stream"]
U = Fr(M * K * N, P * P * tot); Uh = Fr(M * K * N, P * P * cnt["stream"])
F = Fr(T * L, tot); pad = Fr(K, Tk * P) * Fr(N, Tn * P)
res[(P, flow, N)] = dict(U=U, F=F, Np=Tn * P, psums=cnt["psums"], Tk=Tk, Tm=Tm,
pw=(int(per_w.min()), int(per_w.max())))
print(f"{P:2d} {flow} {N:3d} {Tk:2d} {Tn:2d} {T:6d} {tot // T:8d} {tot:8d} {float(U):.6f} "
f"{float(Uh):.6f} {float(F):.6f} {float(pad):.6f} {d:9d} {str(c1):5s} "
f"{cnt['weights']:7d} {cnt['inputs']:7d} {cnt['psums']:6d} "
f"[{per_w.min()}, {per_w.max()}]")
if not c1:
print(" C1 mismatch:", {k: (cnt[k], f[k]) for k in f if cnt[k] != f[k]})
print("\nH1 WS per-weight loads [min, max] (expect [1, 1]):",
{(P, N): res[(P, "WS", N)]["pw"] for P in (8, 16) for N in (8, 12, 16, 20)})
print("H1b OS per-weight loads [min, max] (expect [Tm, Tm]):",
{(P, N): res[(P, "OS", N)]["pw"] + (res[(P, "OS", N)]["Tm"],) for P in (8, 16) for N in (8, 12, 16, 20)})
print("H2 psums WS / OS (expect Tk):",
{(P, N): (res[(P, "WS", N)]["psums"] / res[(P, "OS", N)]["psums"], res[(P, "WS", N)]["Tk"],
res[(P, "WS", N)]["psums"] == res[(P, "WS", N)]["Tk"] * res[(P, "OS", N)]["psums"])
for P in (8, 16) for N in (8, 12, 16, 20)})
for P in (8, 16):
for flow in ("WS", "OS"):
ref = res[(P, flow, 16)]["U"]
print(f"H3 P={P} {flow}: U/U(16) vs N/N_pad:",
[(N, f"{float(res[(P, flow, N)]['U'] / ref):.6f}", f"{N / res[(P, flow, N)]['Np']:.6f}",
res[(P, flow, N)]["U"] / ref == Fr(N, res[(P, flow, N)]["Np"])) for N in (8, 12, 16, 20)])
for P in (8, 16):
for flow in ("WS", "OS"):
loss = {N: 1 - res[(P, flow, N)]["F"] for N in (8, 12, 16, 20)}
strict = all(loss[8] > loss[N] for N in (12, 16, 20))
print(f"H4 P={P} {flow}: 1-F by Cout", {N: f"{float(v):.6f}" for N, v in loss.items()},
f"largest strictly at Cout=8: {strict}")
for P in (8, 16):
fw, fo = res[(P, "WS", 8)]["F"], res[(P, "OS", 8)]["F"]
same = all(res[(P, fl, N)]["F"] == res[(P, fl, 8)]["F"] for fl in ("WS", "OS") for N in (12, 16, 20))
print(f"H4b P={P}: F_WS {float(fw):.6f} F_OS {float(fo):.6f} F_OS < F_WS: {fo < fw} "
f"(F equal across Cout: {same})")
It printed:
D0 data: int8 (3, 32, 32), every pixel = 4x + 100*mask - 128: True, mask areas [193, 242, 313], range [-128, 84]
P flow Cout Tk Tn tiles cyc/tile cycles U U_hidden F pad(K,N) C0 max|d| C1 weights inputs psums per-weight[min,max]
8 WS 8 4 1 4 1046 4184 0.826004 0.832370 0.978967 0.843750 0 True 256 32768 32768 [1, 1]
8 WS 12 4 2 8 1046 8368 0.619503 0.624277 0.978967 0.632812 0 True 512 65536 65536 [1, 1]
8 WS 16 4 2 8 1046 8368 0.826004 0.832370 0.978967 0.843750 0 True 512 65536 65536 [1, 1]
8 WS 20 4 3 12 1046 12552 0.688337 0.693642 0.978967 0.703125 0 True 768 98304 98304 [1, 1]
8 OS 8 4 1 128 54 6912 0.500000 0.586957 0.592593 0.843750 0 True 32768 32768 8192 [128, 128]
8 OS 12 4 2 256 54 13824 0.375000 0.440217 0.592593 0.632812 0 True 65536 65536 16384 [128, 128]
8 OS 16 4 2 256 54 13824 0.500000 0.586957 0.592593 0.843750 0 True 65536 65536 16384 [128, 128]
8 OS 20 4 3 384 54 20736 0.416667 0.489130 0.592593 0.703125 0 True 98304 98304 24576 [128, 128]
16 WS 8 2 1 2 1070 2140 0.403738 0.409867 0.957009 0.421875 0 True 512 32768 32768 [1, 1]
16 WS 12 2 1 2 1070 2140 0.605607 0.614801 0.957009 0.632812 0 True 512 32768 32768 [1, 1]
16 WS 16 2 1 2 1070 2140 0.807477 0.819734 0.957009 0.843750 0 True 512 32768 32768 [1, 1]
16 WS 20 2 2 4 1070 4280 0.504673 0.512334 0.957009 0.527344 0 True 1024 65536 65536 [1, 1]
16 OS 8 2 1 64 78 4992 0.173077 0.217742 0.410256 0.421875 0 True 32768 32768 16384 [64, 64]
16 OS 12 2 1 64 78 4992 0.259615 0.326613 0.410256 0.632812 0 True 32768 32768 16384 [64, 64]
16 OS 16 2 1 64 78 4992 0.346154 0.435484 0.410256 0.843750 0 True 32768 32768 16384 [64, 64]
16 OS 20 2 2 128 78 9984 0.216346 0.272177 0.410256 0.527344 0 True 65536 65536 32768 [64, 64]
H1 WS per-weight loads [min, max] (expect [1, 1]): {(8, 8): (1, 1), (8, 12): (1, 1), (8, 16): (1, 1), (8, 20): (1, 1), (16, 8): (1, 1), (16, 12): (1, 1), (16, 16): (1, 1), (16, 20): (1, 1)}
H1b OS per-weight loads [min, max] (expect [Tm, Tm]): {(8, 8): (128, 128, 128), (8, 12): (128, 128, 128), (8, 16): (128, 128, 128), (8, 20): (128, 128, 128), (16, 8): (64, 64, 64), (16, 12): (64, 64, 64), (16, 16): (64, 64, 64), (16, 20): (64, 64, 64)}
H2 psums WS / OS (expect Tk): {(8, 8): (4.0, 4, True), (8, 12): (4.0, 4, True), (8, 16): (4.0, 4, True), (8, 20): (4.0, 4, True), (16, 8): (2.0, 2, True), (16, 12): (2.0, 2, True), (16, 16): (2.0, 2, True), (16, 20): (2.0, 2, True)}
H3 P=8 WS: U/U(16) vs N/N_pad: [(8, '1.000000', '1.000000', True), (12, '0.750000', '0.750000', True), (16, '1.000000', '1.000000', True), (20, '0.833333', '0.833333', True)]
H3 P=8 OS: U/U(16) vs N/N_pad: [(8, '1.000000', '1.000000', True), (12, '0.750000', '0.750000', True), (16, '1.000000', '1.000000', True), (20, '0.833333', '0.833333', True)]
H3 P=16 WS: U/U(16) vs N/N_pad: [(8, '0.500000', '0.500000', True), (12, '0.750000', '0.750000', True), (16, '1.000000', '1.000000', True), (20, '0.625000', '0.625000', True)]
H3 P=16 OS: U/U(16) vs N/N_pad: [(8, '0.500000', '0.500000', True), (12, '0.750000', '0.750000', True), (16, '1.000000', '1.000000', True), (20, '0.625000', '0.625000', True)]
H4 P=8 WS: 1-F by Cout {8: '0.021033', 12: '0.021033', 16: '0.021033', 20: '0.021033'} largest strictly at Cout=8: False
H4 P=8 OS: 1-F by Cout {8: '0.407407', 12: '0.407407', 16: '0.407407', 20: '0.407407'} largest strictly at Cout=8: False
H4 P=16 WS: 1-F by Cout {8: '0.042991', 12: '0.042991', 16: '0.042991', 20: '0.042991'} largest strictly at Cout=8: False
H4 P=16 OS: 1-F by Cout {8: '0.589744', 12: '0.589744', 16: '0.589744', 20: '0.589744'} largest strictly at Cout=8: False
H4b P=8: F_WS 0.978967 F_OS 0.592593 F_OS < F_WS: True (F equal across Cout: True)
H4b P=16: F_WS 0.957009 F_OS 0.410256 F_OS < F_WS: True (F equal across Cout: True)
Checks. D0 held: the image has the planned pixel values and the three shapes cover 193, 242 and 313 pixels. C0 held: the simulator’s output equals the direct convolution exactly in all 16 rows, so the skew, the movement of inputs and partial sums, the preload, the drain and the accumulation across tiles put every product in the right place. C1 held: every counter equals its closed form in all 16 rows.
Weights, inputs and partial sums. H1 held: in every weight-stationary row each real weight entered the array exactly once. H1b held: in output-stationary mode each weight entered 128 times on the array and 64 times on the array, once per tile of 8 or 16 output positions. For eight output channels on the smaller array, that is 256 weight values loaded against 32,768 streamed. H2 held: WS sent four times as many partial sums to its accumulator as OS drained on the array and twice as many on the array, the number of tiles in each case, because a WS tile sees only a slice of the reduction and must hand its sums to the accumulator, while an OS PE finishes its output before it leaves. Input traffic is the same in both dataflows in every row: in our derivation from the closed forms, each padded element of enters the array once per tile in either dataflow. Each dataflow keeps one operand class at home and pays for it in another, which is Chen et al.’s point about dataflows that optimize one kind of reuse; the counts here are numbers of transfers, not energies, and their cost depends on which memory each operand comes from.
Cycles and utilization. OS took more cycles than WS in every row (6,912 against 4,184 for eight output channels on the array). Of the 54 cycles of an OS tile on the array, only 32 stream the padded reduction (); the other 22 are the overhead of cycles (our arithmetic and derivation), whereas a WS tile streams all 1,024 input rows past the same overhead. This is H4b, which held: the fill-and-drain factor was 0.592593 for OS against 0.978967 for WS at , and 0.410256 against 0.957009 at . The effect belongs to this layer’s shape, with its short reduction of : in our derivation from the formula for , a layer with a long reduction would give OS a long stream too.
H3 held in all 16 rows: dividing each utilization by that of gives exactly . Twelve output channels on the array need two tiles, the same as sixteen, so they take the same cycles and reach three-quarters of the utilization; on the array, eight channels leave half the columns idle and reach exactly half the utilization of sixteen. The padding of from 27 to 32 caps every row at a padding factor of 0.843750, even when divides . A larger array is not faster on a layer too small to fill it: for eight output channels, utilization falls from 0.826004 on the array to 0.403738 on the array (WS).
H4, that fill and drain lower utilization most for the smallest layer, failed in all four groups of array size and dataflow, by a four-way tie in each: the loss was 0.021033 for every output-channel count with WS at , 0.407407 with OS at , 0.042991 with WS at and 0.589744 with OS at . H4b’s lines print that is equal across the output-channel counts, and the derivation written before any script ran had predicted the tie.
The open modelling choice. Hiding the preload or drain behind the previous tile, as double buffering would, changes WS little (0.826004 against 0.832370 for eight channels on the array) and OS more (0.500000 against 0.586957; 0.346154 against 0.435484 for sixteen channels on the array), because the edge cycles are a small part of a WS tile and a large part of an OS tile. In every row, OS stays below WS even with its drain hidden. In our model, the remaining cycles of skew per tile are not removed by buffering, since they come from the diagonal wavefront itself.
Common Misconceptions
- “The TOPS figure is the speed.” It is a peak: operations per second only when every PE does useful work every cycle. The example’s layer reached a utilization of at most 0.826004 (WS, eight or sixteen output channels on the array), and only 0.173077 with OS and eight channels on the array; the roofline in edge inference shows how memory bandwidth lowers the achievable rate further for layers with few operations per byte.
- “A bigger array is always faster.” Only for layers large enough to fill it. Output-channel counts, reduction lengths and spatial sizes that are not multiples of the array’s dimensions leave PEs idle, and the fixed cost of filling and draining the array weighs more on short streams.
- “MACs predict time on an accelerator.” Two layers with the same MAC count but different shapes can map very differently onto a fixed array. In our derivation from the utilization formula, a depthwise convolution, which reduces over only the nine taps of one channel and produces one output channel per group, fills a small fraction of a large array.
- “The dataflow is a software detail.” It is fixed by the hardware’s wiring and local memories. Gemmini can offer both WS and OS at run time or harden one when the hardware is generated; Eyeriss was built around RS.
- “The accelerator runs the whole network.” It runs the operators it implements. The rest falls back to another processor, with a transfer at each switch, which is why runtimes partition graphs and why a model’s choice of operators matters.
Where It Is Used
Accelerators of this kind sit in phone and camera systems-on-chip, robots, drones, cars and smart cameras, wherever a vision model must run within a device’s power budget. Examples named in published benchmarks and repositories include Google’s EdgeTPU and Apple’s Neural Engine, on which MobileNetV4 was measured with INT8 and 16-bit floating-point models respectively, as the MobileNet article records; NVIDIA’s Jetson modules, for which the jetson_benchmarks README titles its benchmarks as using “GPU+2DLA”; and accelerators such as Hailo’s and NVDLA, named here only as examples. In the data centre, Jouppi et al. (2017) describe Google’s Tensor Processing Unit (not read for this article). On the research side, Gemmini is a full-system, full-stack platform for exploring DNN hardware, part of the Chipyard ecosystem and written in the Chisel hardware description language, whose generator parameters cover the array’s dimensions, the dataflow, the scratchpad and accumulator capacities and the number formats (README).
The models that run best on such hardware are those whose layers map well onto its array and use only the operators it implements: compact convolutional networks such as MobileNet, quantized to the accelerator’s integer formats and compressed with the methods of model compression. How fast a model then runs end to end, including the layers that fall back to the CPU and the transfers between processors, is measured as described in latency and throughput and edge inference, and choosing a runtime that reaches the accelerator is part of deploying a model.
Related
- Model Compression
Making a trained vision model smaller and cheaper to run by pruning weights or filters, quantizing, distilling it into a smaller network, factorizing its layers or choosing a compact architecture, and how to compare the results fairly in accuracy, size, operations and measured latency.
- Latency and Throughput
The two measures of a vision system's speed, how waiting, service time and batching set them, why percentiles rather than means describe latency, what parameters, multiply-adds and activations do and do not say about runtime, and how to time a component honestly.
- MobileNet
Google's family of convolutional networks for phones and other small devices, from MobileNet v1 (2017), built from depthwise separable convolutions with width and resolution multipliers, through MobileNetV2's inverted residuals with linear bottlenecks and MobileNetV3's latency-driven architecture search, to MobileNetV4's universal inverted bottleneck and mobile attention (2024).
- Depthwise Separable Convolution
A convolutional layer factored into a depthwise convolution, one spatial filter per input channel, and a pointwise 1×1 convolution that mixes channels, cutting parameters and multiply-adds by a factor of about k² at the price of a restricted kernel; the building block of Xception, MobileNet and EfficientNet.
- Convolutional Neural Network
A neural network built from convolutional layers, small learned filters applied at every image position, interleaved with nonlinearities and downsampling, so that features grow from edges to object parts while the parameter count stays independent of image size.
- Deploying Vision Models
Getting a trained vision model into use, from export and the choice of runtime to packaging the model with its preprocessing contract, serving it on a server or a device, keeping pre- and post-processing identical to training, and versioning, rolling back and monitoring it.
References
- Chen, Y.-H., Emer, J. & Sze, V. (2016). Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks. International Symposium on Computer Architecture (ISCA) (read in Joel Emer's copy; page numbers not available to us).
- ucb-bar (2023). Gemmini README, ucb-bar/gemmini v0.7.2 (commit 709bc56, 21 August 2023).
- Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A. et al. (2017). In-Datacenter Performance Analysis of a Tensor Processing Unit. International Symposium on Computer Architecture (ISCA) (not read).
- NVIDIA-AI-IOT. jetson_benchmarks README (commit c029c7d, 1 December 2023).