Vision Systems & Engineering
Inference Optimization
Transformations that make a trained vision network run faster or in less memory without changing the function it computes, from constant folding and operator fusion to layout changes, compilation, kernel selection and memory planning, with a worked example of what fusion stops writing to memory.
advanced
A trained network is a description of a function: convolutions, additions, activations and pooling, arranged in a graph. How fast that function runs, and how much memory it needs, depends on how the description is turned into work for a processor. Inference optimization is the set of transformations a runtime or compiler applies between the exported model and the executed program to make inference cheaper without changing what the model computes. They include folding constants, fusing neighbouring operators so that intermediate tensors are never written to memory, choosing memory layouts and kernels, specializing for known shapes, compiling the graph to generated code, planning memory, and cutting the overhead of launching many small operations.
In our framing, these transformations are separate from model compression, network pruning and quantization, which change the function to make it cheaper: a pruned or 8-bit model computes something different from the original and is judged by its accuracy. A function-preserving transformation is judged by whether its output equals the original’s, to floating-point rounding. Reduced precision and batching are linked here rather than covered: precision belongs to quantization and mixed precision, batching to latency and throughput.
Purpose
An exported model, for example an ONNX file, is written for portability. Its graph has one node per operation as the training framework recorded it, constants that could have been precomputed, and separate nodes for a convolution, its bias, a scale and an activation. Executed node by node, every node writes its full output tensor to memory and the next node reads it back. For a vision network these tensors are feature maps with one value per channel and pixel, and in our reading they can be far larger than the weights: in the example below, the first convolution has 216 weights and writes a feature map of 32,768 values.
The edge inference article’s roofline explains why this matters: a layer whose arithmetic per byte moved is low is limited by memory bandwidth, not by arithmetic, and only less traffic makes it faster. In the article’s per-layer bound, the bytes a layer moves depend on how the runtime schedules it, and fusing layers so that intermediate results stay on chip lowers them. XLA’s GPU documentation gives the same reason for fusion: many GPU workloads are memory-bound, and fusion avoids writing intermediate tensors to the GPU’s main memory (HBM) and reading them back (XLA GPU architecture overview). Inference optimization removes that traffic, the work spent on constants, and the fixed cost of each operation, while keeping the function.
When to Use It
Every deployment uses some of it, usually without asking: ONNX Runtime enables all its graph optimizations by default (ONNX Runtime graph-optimizations documentation), and compilers such as XLA, TensorRT and PyTorch’s TorchInductor apply their own as part of building a program. The decisions a practitioner makes are about the rest:
- Whether to optimize offline. Optimizing at every session start costs start-up time, which ONNX Runtime’s documentation notes can be critical in production; saving the optimized model avoids it, at the price of tying the file to one configuration (see Pitfalls).
- How far to specialize. Fixed input shapes, a fixed device and a fixed library version let a compiler choose kernels and memory plans for exactly that case. A model that must accept varying image sizes, or run on several devices, keeps more generality.
- Whether to build for a specific accelerator. Kernel selection by measurement (autotuning) makes the build slower and the result specific to the hardware it was measured on.
In our reading, the transformations help most when the graph has many small or element-wise operations between the large ones, which is the usual shape of a convolutional network with normalization, bias and activation after each convolution. They help little when the time is already in a few large matrix multiplications or convolutions running near the processor’s peak, and they cannot help a model whose cost is in the operations themselves; there a smaller model, pruning or quantization is the lever.
How It Works
Optimizers work on an intermediate representation of the graph: nodes with inputs, outputs and attributes, and constant tensors (ONNX calls them initializers). Each optimization is a rewrite that matches a pattern, checks conditions under which the rewrite preserves the function, and replaces the matched nodes. ONNX Runtime groups its rewrites into levels, applied in order (graph-optimizations documentation at gh-pages commit e3d000e):
- Basic rewrites are semantics-preserving and run before the graph is partitioned among execution providers, so they apply to all providers: constant folding, elimination of redundant nodes (Identity, Slice, Unsqueeze, Dropout) and fusions that fold one node into another, such as Conv Add, Conv Mul and Conv BatchNorm.
- Extended rewrites are more complex fusions that run after partitioning and only on nodes assigned to the CPU, CUDA or ROCm providers, such as Conv Activation, GEMM Activation, GELU and Layer Normalization fusion.
- Layout optimizations run after partitioning on CPU nodes only; the documented one replaces the NCHW layout with NCHWc.
The documentation lags the code: the v1.32.0 source has a fourth level containing an initializer-fusing transformer (FuseInitializersTransformer, registered for converting float16 initializers to float32), and the gh-pages documentation at e3d000e does not list it (graph_transformer_utils.cc at v1.32.0). The same source shows more Level 1 rewrites than the documentation names, among them common-subexpression elimination, constant sharing and a Div–Mul fusion.
Constant folding and redundant nodes
Constant folding evaluates, once, every part of the graph whose inputs are all constants, and replaces it by its result. An exported model often contains such subgraphs: shape computations, constants scaled by other constants, or reshapes of weights. ONNX Runtime folds them by running the CPU kernels on the constant inputs, and limits each folded node’s output to 1 GB by default, a guard against malicious models causing excessive memory allocation during optimization (constant_folding.cc at v1.32.0). Redundant-node elimination removes nodes that do nothing at inference, such as Identity, or Dropout, which is the identity in evaluation mode. Common-subexpression elimination merges nodes that compute the same value twice.
Algebraic fusion into a convolution
A convolution is linear in its input, so an affine operation applied to its output per output channel can move into its weights and bias. If and the next nodes compute with per-channel constants and , the fused convolution has weights and bias for each output channel . Batch normalization in evaluation mode is such an affine map, and its folding into the preceding convolution is worked through in the quantization article’s example and in the ONNX article’s export example, where the exporter’s optimization pass, which also removes helper nodes, takes the graph from 13 nodes to 8.
The conditions decide whether the rewrite is exact, and implementations check them before fusing. ONNX Runtime 1.32.0’s ConvAddFusion and ConvMulFusion fuse an Add or Mul into the Conv before it only when (our summary of conv_add_fusion.cc and conv_mul_fusion.cc at v1.32.0):
- the Conv is of opset version 1, 11 or 22 and its output has exactly one consumer and is not a graph output;
- the consumer is an Add or Mul (opset 7, 13 or 14) with exactly one input coming from another node, and the constant is its second input;
- the two nodes are assigned to the same execution provider;
- the Conv’s weights, its bias if present, and the Add’s or Mul’s constant are constant initializers of the same floating-point type, and the weights have more than two dimensions;
- the constant has the weights’ rank (for a 2-D convolution, shape ) or one dimension less (), its dimension on that channel axis equals the number of output channels , and every other dimension is 1. ConvMulFusion also accepts a scalar; ConvAddFusion does not;
- an existing Conv bias is one-dimensional with elements.
The shape condition is what keeps the rewrite exact. A constant that broadcasts over the output channels scales or shifts whole channels, which the weights can absorb; a constant that varies over pixel positions, for example a gain that changes across the image’s columns, cannot be absorbed by weights shared across positions, so the pass must refuse it.
Operator fusion and the memory it saves
Epilogue fusion attaches element-wise operations that follow a large operation to its kernel: the activation is applied to each output value as it is produced, before it is written. ONNX Runtime’s ConvActivationFusion replaces a Conv and its lone consumer, if that consumer is Relu, Sigmoid, Tanh, LeakyRelu, Clip with constant bounds, or, on the CPU, JavaScript and WebGPU providers, HardSigmoid (WebGPU accepts further activations), with a FusedConv node carrying the activation as an attribute (conv_activation_fusion.cc at v1.32.0). In 1.32.0 this fusion is registered for the CPU, JavaScript and WebGPU providers (graph_transformer_utils.cc), and its selector returns no match for nodes assigned to the CUDA provider. On CPU the fusion requires float input (or float16 where the build supports it).
More generally, operator fusion groups a chain of operations into one kernel, so that the values passed between them stay in registers or on-chip memory. XLA’s GPU documentation calls fusion “XLA’s single most important optimization” and states two invariants of a fused computation: no intermediate value inside it is materialized in the GPU’s main memory, and it is compiled to exactly one GPU kernel (XLA GPU architecture overview). Its running example fuses a type conversion, a matrix multiplication, a scaling and a negation into one Triton-generated kernel. Fusion saves the bytes of every tensor it keeps internal, which would otherwise be written once and read back once, and, in our reading, the fixed cost of each kernel launch it removes. It does not change the number of multiply-adds.
Fusion changes the order of floating-point operations: scaling the weights before the convolution sums its products is not the same rounding as scaling the sum afterwards. Fused and unfused results can therefore differ in the last bits, unless the implementation happens to keep the order, as the PyTorch article’s compiled model did.
Layouts
A tensor’s logical shape does not fix its order in memory. A batch of images can be stored with channels outermost (NCHW), channels innermost (NHWC), or in blocked forms such as ONNX Runtime’s NCHWc, in which channels are grouped into blocks whose size the platform’s math library sets (graph_transformer_utils.cc). XLA separates logical shape from physical layout in its intermediate representation, chooses a layout for each operation in a layout-assignment pass (its documentation’s example is NHWC for a convolution on NVIDIA’s Ampere GPUs), propagates the choices through the graph, and inserts explicit copy operations where layouts conflict (XLA GPU architecture overview). In our reading, a layout change pays off only if the transposes it adds cost less than the faster kernels it enables; a graph that switches layout around every operation can be slower than one left alone.
Shapes and specialization
A compiler that knows the shapes in advance can choose loop bounds, tile sizes, kernels and memory offsets for them. XLA lists shape specialization as one of its means to speed, specializing known tensor shapes to allow more aggressive constant propagation (XLA architecture). Dynamic shapes, such as a variable batch or image size, limit this, in our reading. TensorRT’s trtexec tool builds engines for a range of shapes given as minimum, optimum and maximum input dimensions (--minShapes, --optShapes, --maxShapes) (trtexec README at v11.4). In our reading, once shapes are known, shape computations in the graph become constant and fold away.
Compilation and code generation
Graph rewriting leaves each node to a library kernel. A compiler goes further and generates code for the fused groups. XLA takes graphs from PyTorch, TensorFlow and JAX in the StableHLO operation set, runs target-independent passes (common-subexpression elimination, fusion and buffer analysis), then target-specific ones, and emits code through LLVM for its CPU and GPU back ends (XLA architecture); for fusions that include a matrix multiplication or softmax, its GPU back end generates code through Triton and selects the tiling (XLA GPU architecture overview). For each operation it chooses between calling a library such as cuBLAS or cuDNN and generating code; its documentation notes that libraries have verified performance but often preclude complex fusions. Apache TVM describes itself as an open machine-learning compilation framework that represents graphs and tensor programs at two levels (Relax and TensorIR) so that both can be optimized jointly with libraries (TVM README at v0.27.0). PyTorch’s torch.compile with TorchInductor generates Triton kernels for GPUs and C++ for CPUs, as the PyTorch article describes, and XLA is the compiler TensorFlow uses through jit_compile=True (TensorFlow).
Kernel selection and autotuning
TensorRT’s sample documentation notes that some layers have several implementations, which TensorRT calls tactics; the sample’s kernels differ in their tile sizes. When TensorRT builds an engine, all the tactics of a layer are profiled, the fastest is chosen, and the decision is written into a timing cache; a later build that reuses the cache skips the profiling and takes the recorded tactics (sampleEditableTimingCache README at v11.4). Because choices are measured, they depend on the device and the run: the trtexec README states that engine builds are not bit-deterministic, since kernel timings vary and the builder breaks ties accordingly, and that the build route its tuner finds is specific to the model (rebuilding with other shapes or precision flags invalidates it), to the GPU model and to the TensorRT version (trtexec README at v11.4). ONNX Runtime’s TensorRT execution provider can cache both the engines and the timing cache to save build time (TensorRT execution provider documentation).
Memory planning
A graph’s intermediate tensors do not all live at once. A planner that knows the execution order computes each tensor’s first and last use and lets a tensor reuse the memory of one that is no longer needed. LiteRT’s arena planner describes exactly this: before execution it determines when each tensor must be allocated and deallocated, preallocates one buffer, and assigns portions of it so that tensors may share memory when one is allocated after another has been released; it orders tensors that live for the whole inference first and the others from largest to smallest, and aligns buffers to 64 bytes by default (arena_planner.h at v2.3.2). ONNX Runtime’s allocation planner records for each value whether it is allocated, reuses another value’s buffer or shares a pre-existing one, using a free list of buffers ordered by when they became free (allocation_planner.cc at v1.32.0). XLA’s buffer assignment, according to its documentation, aims to minimize the intermediate memory consumed, and its scheduler, unlike an eager framework’s allocator, which does not see the graph in advance, can “look into the future” (XLA GPU architecture overview). Fusion and memory planning work together: fusion removes tensors, and planning packs the ones that remain.
Overhead
Each operation an eager framework runs costs a call through the framework, and each kernel on a GPU costs a launch. For small images or small batches these fixed costs can exceed the arithmetic. Capturing the whole graph removes the per-operation interpreter overhead, which is one goal of torch.compile (the PyTorch article describes its "reduce-overhead" mode, which uses CUDA graphs); XLA lists eliminating runtime overhead and the execution time of short-lived operations among its objectives (XLA architecture). Data movement between host and device is a related overhead: ONNX Runtime’s documentation explains that for non-CPU providers without I/O binding, inputs are copied to the device and outputs back to the CPU inside each Run() call, which can dominate the measured time, and that I/O binding arranges inputs on the device and pre-allocates outputs there before Run() is called (I/O Binding documentation).
Online and offline optimization
Optimization can run online, when a session starts, or offline, once, with the optimized model saved. ONNX Runtime supports both: offline mode serializes the optimized graph so that later sessions can load it with optimizations disabled. Its documentation adds two conditions: the offline model must be produced with the same options (execution providers, optimization level) and on hardware of the same kind as the target, and a model with layout optimizations requires compatible hardware, for example a CPU with AVX2 if the layout was optimized for it (graph-optimizations documentation). The same holds more strongly for compiled engines: ONNX Runtime’s documentation of the TensorRT provider states that each engine is built for specific settings (model, precision, workspace, profiles) and specific GPUs, and is not portable (TensorRT execution provider documentation).
How to Apply It
A worked example: what fusion stops writing, and a fold that is wrong at the border
The example applies four kinds of function-preserving rewrite to a small convolutional graph, one after another, and measures two things: how many bytes of intermediate tensors each rewrite removes, and whether each rewritten graph still computes the original function. It also tests a fold that looks exact but is not. It is written in NumPy 2.3.5 with a small graph representation and executor of our own, and renders its input with scikit-image 0.25.2. Passes P2 and P3 re-implement the matching conditions of ONNX Runtime 1.32.0’s conv_add_fusion.cc, conv_mul_fusion.cc and conv_activation_fusion.cc from their description above; no ONNX Runtime code is copied, and the outcomes test our reproduction of the rewrites, not ONNX Runtime itself. The script runs in under a second and printed identical output on two runs.
Data and graph. The input is one rendered RGB frame in 8-bit integers: a disk and a square on a grey background, stored as (NCHW, batch 1). The reference graph G0 has 16 nodes with seeded random weights; no training is needed, since the question is whether rewrites preserve a function, not what the function recognizes:
- a constant subgraph that scales the per-channel mean and standard deviation to raw pixel units (two
Mulnodes by 255), thenCastto float32,Sub(mean)andDiv(std). The constants, mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225), are our choice; the standard deviations are the torchvision values that the deploying vision models article quotes; Convwith padding 1 from 3 to 8 channels without bias,Addof a per-channel bias of shape ,Mulby a per-channel scale of the same shape,Relu,MaxPool;Convfrom 8 to 16 channels,Add,Relu,GlobalAveragePool,FlattenandGemmto three outputs.
Passes, applied cumulatively in ONNX Runtime’s level order:
- P1, constant folding: G1. Every node whose inputs are all constants is evaluated once in float32 and replaced by its result.
- P2, Conv+Add and Conv+Mul fusion: G2, applied until nothing more matches, under the conditions listed in How It Works. The fused node keeps the output name of the node it absorbs, so tensors such as
relu1can be compared across graphs. - P3, Conv+Relu epilogue fusion: G3. It can only fire after P2, because ONNX Runtime’s condition needs the Relu to be the Conv’s only consumer, and in G0 an Add and a Mul sit between them.
- P4, folding the input normalization into the first convolution: not an ONNX Runtime fusion; the algebra below is ours. It is applied to G2, where the first convolution’s output
mul1is still a separate tensor that can be compared, and P3 is applied afterwards to give G4. Three variants: P4z feeds the raw (cast) image to the convolution with zero padding; P4m pads the raw image with the per-channel mean; P4s folds only the division and keeps the subtraction.
A second graph, G0′, replaces the per-channel scale by a gain that varies across the image’s 64 columns, of shape , which P2 must refuse.
The fold and its border, our derivation. Write the normalized input as per channel, and the first convolution as over its 27 taps and channels. Substituting gives with and , so the convolution can read the raw image directly. This is exact wherever all nine taps fall inside the image. At a tap outside it, the reference reads the zero padding of the normalized image, while P4z reads zero in raw units; the bias still subtracts for that tap, but nothing adds it back. The error at output pixel of channel is therefore
which is zero in the interior, non-zero on the one-pixel border ring of the output (252 pixels), and independent of the image content. Padding the raw image with (P4m) makes each outside tap contribute , which cancels the bias term, so everywhere. P4s keeps the subtraction before the convolution, which then sees the same zero-padded input as the reference, scaled by : also exact. In our reading, P4s is the exact variant that needs only zero padding, while P4m needs a convolution or padding operation that takes a separate border value per channel.
What is measured.
- Nodes and intermediate memory: for each graph, the number of nodes and the elements and bytes of every tensor that one node writes and another reads (the input frame, the initializers and the final output are not counted), taken from the executed arrays. These are accounting checks: the savings follow from the graph’s construction, and the code compares each with the arithmetic on the shapes. With , , channels, pixels before pooling and after, P1 should remove the two 3-element constants (6 elements), P2 the outputs of the first Conv and its Add and of the second Conv (), P3 the two pre-activation maps (), P4z and P4m the outputs of Sub and Div () and P4s that of Div alone (12,288). The executor is NumPy, so a fused node still makes temporary arrays internally; the count is of the tensors between graph nodes, which is what a runtime that runs one kernel per node would write.
- Function preservation: the largest absolute difference from G0, both in float32, at
relu1,pool1,relu2and the logits; for the P4 variants also the difference ofmul1from G2’s, split into interior and border ring, with the formula’s prediction beside it. A float64 run of every graph checks the algebra without rounding. - Tolerances: for each tensor , , where is the largest difference between G0 run in float32 and in float64. The rule was fixed before any script ran, but the tolerance values were first printed by a planning script that ran G0 and G0′ without any pass, so each of the five tolerances below (and the four for G0′) is a value set after a planning script. The float64 check uses a bar of , fixed in advance.
Two changes are disclosed here. The batch plan described P1 as “constant folding of the preprocessing constants”, and its planning research as folding them “into one scale per channel”; merging the subtraction and division into one scale per channel is not constant folding, which evaluates nodes whose inputs are all constants, and that merge is P4’s job, so P1 folds the constant subgraph that computes the scaled mean and standard deviation instead. And after the pilot run, the log of the refused graph was changed to print each fusion decision once: the pilot printed the refusal twice, because the pass re-examines the pair after a later fusion. That was a change of print format only; no data, measure, tolerance or pass changed.
import os
os.environ["OMP_NUM_THREADS"] = "1" # one BLAS thread, set before NumPy loads
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["VECLIB_MAXIMUM_THREADS"] = "1"
import numpy as np
from skimage.draw import disk, rectangle
# ---- data: one rendered 64x64 RGB frame (uint8), NCHW, batch 1 -------------
H = W = 64
img = np.full((H, W, 3), 128, np.uint8) # grey background
rd, cd = disk((22, 22), 12, shape=(H, W)); img[rd, cd] = (200, 60, 50)
rs, cs = rectangle((34, 30), end=(55, 53), shape=(H, W)); img[rs, cs] = (40, 90, 180)
frame = img.transpose(2, 0, 1)[None] # (1, 3, 64, 64)
ring = np.zeros((H, W), bool); ring[[0, -1], :] = True; ring[:, [0, -1]] = True
print("frame", frame.dtype, frame.shape, "| disk px", int((img == (200, 60, 50)).all(2).sum()),
"=", len(rd), "| square px", int((img == (40, 90, 180)).all(2).sum()), "=", 22 * 24,
"| border ring all grey:", bool((img[ring] == 128).all()))
# ---- the reference graph G0, seeded random weights --------------------------
def build(dt, column_gain=False):
rng = np.random.default_rng(0)
w = {"mean": np.array([0.485, 0.456, 0.406]).reshape(1, 3, 1, 1),
"std": np.array([0.229, 0.224, 0.225]).reshape(1, 3, 1, 1), "k255": np.array(255.0),
"w1": rng.normal(0, 27 ** -0.5, (8, 3, 3, 3)), "b1": rng.normal(0, 0.1, (1, 8, 1, 1)),
"s1": rng.uniform(0.5, 1.5, (1, 8, 1, 1)),
"w2": rng.normal(0, 72 ** -0.5, (16, 8, 3, 3)), "b2": rng.normal(0, 0.1, (1, 16, 1, 1)),
"wf": rng.normal(0, 0.25, (3, 16)), "bf": rng.normal(0, 0.1, 3)}
if column_gain: # does not broadcast as C x 1 x 1
w["s1"] = np.linspace(0.8, 1.2, W).reshape(1, 1, 1, W)
inits = {k: v.astype(dt) for k, v in w.items()}
N = lambda op, ins, out, **a: dict(op=op, ins=ins, out=out, attr=a)
nodes = [N("Mul", ["mean", "k255"], "mean255"), N("Mul", ["std", "k255"], "std255"),
N("Cast", ["frame"], "x", to=dt), N("Sub", ["x", "mean255"], "sub"),
N("Div", ["sub", "std255"], "div"), N("Conv", ["div", "w1"], "conv1"),
N("Add", ["conv1", "b1"], "add1"), N("Mul", ["add1", "s1"], "mul1"),
N("Relu", ["mul1"], "relu1"), N("MaxPool", ["relu1"], "pool1"),
N("Conv", ["pool1", "w2"], "conv2"), N("Add", ["conv2", "b2"], "add2"),
N("Relu", ["add2"], "relu2"), N("GlobalAveragePool", ["relu2"], "gap"),
N("Flatten", ["gap"], "flat"), N("Gemm", ["flat", "wf", "bf"], "logits")]
return nodes, inits
# ---- executor: runs nodes in order, records every tensor passed between nodes
def conv3x3(x, w, b=None, pad=None, act=None):
n, c, h, wd = x.shape
xp = np.empty((n, c, h + 2, wd + 2), x.dtype)
xp[:] = (0 if pad is None else pad.reshape(1, c, 1, 1)) # border value per channel
xp[:, :, 1:-1, 1:-1] = x
y = np.zeros((n, w.shape[0], h, wd), x.dtype)
for dy in range(3):
for dx in range(3):
y += np.einsum("oc,nchw->nohw", w[:, :, dy, dx], xp[:, :, dy:dy + h, dx:dx + wd])
if b is not None:
y += b.reshape(1, -1, 1, 1)
return np.maximum(y, 0) if act == "Relu" else y
def run(graph):
nodes, inits = graph
v = dict(inits, frame=frame)
for nd in nodes:
a = [v[i] for i in nd["ins"]]; op = nd["op"]
if op == "Cast": r = a[0].astype(nd["attr"]["to"])
elif op == "Add": r = a[0] + a[1]
elif op == "Sub": r = a[0] - a[1]
elif op == "Mul": r = a[0] * a[1]
elif op == "Div": r = a[0] / a[1]
elif op == "Relu": r = np.maximum(a[0], 0)
elif op == "Conv": r = conv3x3(*a, pad=nd["attr"].get("pad"), act=nd["attr"].get("act"))
elif op == "MaxPool": n, c, h, w_ = a[0].shape; r = a[0].reshape(n, c, h // 2, 2, w_ // 2, 2).max((3, 5))
elif op == "GlobalAveragePool": r = a[0].mean((2, 3), keepdims=True)
elif op == "Flatten": r = a[0].reshape(a[0].shape[0], -1)
elif op == "Gemm": r = a[0] @ a[1].T + a[2]
v[nd["out"]] = r
used = {i for nd in nodes for i in nd["ins"]}
inter = [v[nd["out"]] for nd in nodes if nd["out"] in used] # written by one node, read by another
return v, sum(t.size for t in inter), sum(t.nbytes for t in inter)
def consumers(nodes, name):
return [nd for nd in nodes if name in nd["ins"]]
# ---- P1: constant folding (nodes whose inputs are all initializers) -----------
def p1_fold_constants(graph):
nodes, inits = graph[0], dict(graph[1])
keep = []
for nd in nodes:
if all(i in inits for i in nd["ins"]):
inits[nd["out"]] = run(([nd], inits))[0][nd["out"]]
else:
keep.append(nd)
return keep, inits
# ---- P2: Conv+Add and Conv+Mul fusion, with the shape conditions that ONNX
# Runtime 1.32.0's conv_add_fusion.cc and conv_mul_fusion.cc check (re-implemented)
def p2_fuse_conv_add_mul(graph, log):
nodes, inits = list(graph[0]), dict(graph[1])
changed = True
while changed:
changed = False
for conv in nodes:
if conv["op"] != "Conv" or conv["attr"].get("act"):
continue
nxt = consumers(nodes, conv["out"])
if len(nxt) != 1 or nxt[0]["op"] not in ("Add", "Mul"):
continue
nx = nxt[0]
if nx["ins"][0] != conv["out"] or nx["ins"][1] not in inits:
continue
k, wt = inits[nx["ins"][1]], inits[conv["ins"][1]]
cout = wt.shape[0]
if k.ndim == 0 and nx["op"] == "Mul":
ok = True # scalar: ConvMul only
elif k.ndim in (wt.ndim, wt.ndim - 1):
axis = 1 if k.ndim == wt.ndim else 0 # 1xCx1x1 or Cx1x1
ok = k.shape[axis] == cout and all(d == 1 for i, d in enumerate(k.shape) if i != axis)
else:
ok = False
entry = (f"Conv{nx['op']} {conv['out']}+{nx['out']}: "
+ ("fused" if ok else f"refused, initializer shape {k.shape}"))
if entry not in log: # log each decision once
log.append(entry)
if not ok:
continue
b = inits[conv["ins"][2]] if len(conv["ins"]) == 3 else np.zeros(cout, wt.dtype)
if nx["op"] == "Add":
b = b + k.reshape(-1)
else:
s = k.reshape(-1) if k.ndim else k
wt = wt * (s.reshape(-1, 1, 1, 1) if k.ndim else s); b = b * s
new = dict(conv, ins=[conv["ins"][0], conv["out"] + "_W", conv["out"] + "_B"], out=nx["out"])
inits[new["ins"][1]], inits[new["ins"][2]] = wt, b
nodes[nodes.index(conv)] = new; nodes.remove(nx)
changed = True
break
return nodes, inits
# ---- P3: Conv+Relu epilogue fusion --------------------------------------------
def p3_fuse_conv_relu(graph):
nodes = list(graph[0])
for conv in list(nodes):
nxt = consumers(nodes, conv["out"])
if conv["op"] == "Conv" and not conv["attr"].get("act") and len(nxt) == 1 and nxt[0]["op"] == "Relu":
new = dict(conv, out=nxt[0]["out"], attr=dict(conv["attr"], act="Relu"))
nodes[nodes.index(conv)] = new; nodes.remove(nxt[0])
return nodes, graph[1]
# ---- P4: fold the input normalization (Sub mean, Div std) into the first Conv --
# (our derivation; not an ONNX Runtime fusion). mode z: zero-pad the raw input;
# m: pad the raw input with the per-channel mean; s: fold Div only, keep Sub.
def p4_fold_normalization(graph, mode):
nodes, inits = list(graph[0]), dict(graph[1])
sub = next(nd for nd in nodes if nd["op"] == "Sub")
div = consumers(nodes, sub["out"])[0]; conv = consumers(nodes, div["out"])[0]
mean = inits[sub["ins"][1]].reshape(-1); std = inits[div["ins"][1]].reshape(-1)
wt = inits[conv["ins"][1]] / std.reshape(1, -1, 1, 1)
b = inits[conv["ins"][2]]
attr = dict(conv["attr"])
if mode == "s":
src = sub["out"]
else:
src = sub["ins"][0]
b = b - np.einsum("ochw,c->o", wt, mean)
if mode == "m":
attr["pad"] = mean
nodes.remove(sub)
new = dict(conv, ins=[src, conv["out"] + "_W4", conv["out"] + "_B4"], attr=attr)
inits[new["ins"][1]], inits[new["ins"][2]] = wt, b
nodes[nodes.index(conv)] = new; nodes.remove(div)
return nodes, inits
def predicted_border_error(graph, mean):
"""-sum over taps outside the image of W'[o,c,k] * mean_c (zero-padded fold, our derivation)."""
nodes, inits = graph
conv = next(nd for nd in nodes if nd["op"] == "Conv")
wt = inits[conv["ins"][1]]
inside = np.zeros((1, 3, H, W), wt.dtype) + mean.reshape(1, 3, 1, 1)
return conv3x3(inside, wt) - conv3x3(inside, wt, pad=mean)
# ---- build every graph --------------------------------------------------------
def all_graphs(dt, column_gain=False):
log = []
g = {"G0": build(dt, column_gain)}
g["G1"] = p1_fold_constants(g["G0"])
g["G2"] = p2_fuse_conv_add_mul(g["G1"], log)
g["G3"] = p3_fuse_conv_relu(g["G2"])
for m in "zms":
g["P4" + m] = p4_fold_normalization(g["G2"], m) # first conv still materialized
g["G4" + m] = p3_fuse_conv_relu(g["P4" + m])
return g, log
T = ["relu1", "pool1", "relu2", "logits"]
u = 2.0 ** -24
g32, log32 = all_graphs(np.float32)
g64, _ = all_graphs(np.float64)
def tolerances(r32, r64):
return {t: 10 * max(np.abs(r32[t] - r64[t]).max(), u * np.abs(r64[t]).max()) for t in T + ["mul1"]}
ref32, ref64 = run(g32["G0"])[0], run(g64["G0"])[0]
tol = tolerances(ref32, ref64)
print("tolerances: " + ", ".join(f"{t} {tol[t]:.1e}" for t in T + ["mul1"]))
print("P2 on G1:", "; ".join(log32))
# M2, M3 and H2: nodes, intermediate elements and bytes, savings against the arithmetic
C, C1, C2, h2 = 3, 8, 16, H // 2
expect = {"G1": 2 * C, "G2": 2 * C1 * H * W + C2 * h2 * h2, "G3": C1 * H * W + C2 * h2 * h2,
"G4z": 2 * C * H * W, "G4m": 2 * C * H * W, "G4s": C * H * W}
prev = {"G1": "G0", "G2": "G1", "G3": "G2", "G4z": "G3", "G4m": "G3", "G4s": "G3"}
e0 = 2 * C + 3 * C * H * W + 4 * C1 * H * W + C1 * h2 * h2 + 3 * C2 * h2 * h2 + 2 * C2
print(f"\n{'graph':5} {'nodes':>5} {'elements':>9} {'bytes':>9} {'saved':>7} {'expected':>8}")
cnt = {k: run(g32[k])[1:] for k in ["G0", "G1", "G2", "G3", "G4z", "G4m", "G4s"]}
for k, (ne, nb) in cnt.items():
s = f"{cnt[prev[k]][0] - ne:7d} {expect[k]:8d}" if k in prev else f"{'':7} {e0:8d}"
print(f"{k:5} {len(g32[k][0]):5d} {ne:9d} {nb:9d} {s}")
# M1, H1, H3-H5: max |difference| against G0 in float32
print(f"\nmax |diff| vs G0 (float32) " + " ".join(f"{t:>9}" for t in T) + " within tol")
for k in ["G1", "G2", "G3", "G4z", "G4m", "G4s"]:
v = run(g32[k])[0]
d = [np.abs(v[t] - ref32[t]).max() for t in T]
print(f"{k:28} " + " ".join(f"{x:9.2e}" for x in d) + f" {all(x <= tol[t] for x, t in zip(d, T))}")
# M4 and C-pred: first convolution's output, interior pixels against the border ring
ref2 = run(g32["G2"])[0]["mul1"]
print(f"\nfirst conv output vs G2 interior border ring predicted ring |meas-pred| (tol {tol['mul1']:.1e})")
for m in "zms":
e = run(g32["P4" + m])[0]["mul1"] - ref2
mean255 = g32["G2"][1]["mean255"].reshape(-1)
pred = predicted_border_error(g32["P4" + m], mean255) if m == "z" else np.zeros_like(e)
print(f"P4{m}(G2){'':17} {np.abs(e[..., ~ring]).max():9.2e} {np.abs(e[..., ring]).max():11.2e}"
f" {np.abs(pred[..., ring]).max():14.2e} {np.abs(e - pred).max():12.2e}")
# C0: exact arithmetic (float64): every exact graph reproduces G0
r64 = {}
for k in ["G1", "G2", "G3", "P4z", "P4m", "P4s", "G4z", "G4m", "G4s"]:
v = run(g64[k])[0]
r64[k] = max(np.abs(v[t] - ref64[t]).max() for t in T + ["mul1"] if t in v)
print("\nfloat64 max |diff| vs G0:", ", ".join(f"{k} {x:.1e}" for k, x in r64.items()))
# C-refuse: a Mul whose initializer is a column gain (1, 1, 1, 64)
gc, logc = all_graphs(np.float32, column_gain=True)
refc = run(gc["G0"])[0]
tolc = tolerances(refc, run(all_graphs(np.float64, column_gain=True)[0]["G0"])[0])
vc = run(gc["G3"])[0]
dc = [np.abs(vc[t] - refc[t]).max() for t in T]
print("\ncolumn-gain graph, P2:", "; ".join(logc))
print(f"nodes G0 {len(gc['G0'][0])}, after P1-P3 {len(gc['G3'][0])}; ops",
[nd["op"] + ("+Relu" if nd["attr"].get("act") else "") for nd in gc["G3"][0]])
print("max |diff| vs its own G0: " + ", ".join(f"{t} {x:.2e} (tol {tolc[t]:.1e})" for t, x in zip(T, dc))
+ f"; within tol {all(x <= tolc[t] for x, t in zip(dc, T))}")
Output:
frame uint8 (1, 3, 64, 64) | disk px 437 = 437 | square px 528 = 528 | border ring all grey: True
tolerances: relu1 3.0e-06, pool1 3.0e-06, relu2 4.5e-06, logits 2.6e-07, mul1 3.0e-06
P2 on G1: ConvAdd conv1+add1: fused; ConvMul add1+mul1: fused; ConvAdd conv2+add2: fused
graph nodes elements bytes saved expected
G0 16 225318 901272 225318
G1 14 225312 901248 6 6
G2 11 143392 573568 81920 81920
G3 9 94240 376960 49152 49152
G4z 7 69664 278656 24576 24576
G4m 7 69664 278656 24576 24576
G4s 8 81952 327808 12288 12288
max |diff| vs G0 (float32) relu1 pool1 relu2 logits within tol
G1 0.00e+00 0.00e+00 0.00e+00 0.00e+00 True
G2 2.38e-07 2.38e-07 4.17e-07 4.47e-08 True
G3 2.38e-07 2.38e-07 4.17e-07 4.47e-08 True
G4z 2.00e+00 2.00e+00 1.43e+00 3.57e-02 False
G4m 8.94e-07 7.15e-07 5.96e-07 1.94e-07 True
G4s 2.38e-07 2.38e-07 3.58e-07 2.98e-08 True
first conv output vs G2 interior border ring predicted ring |meas-pred| (tol 3.0e-06)
P4z(G2) 8.34e-07 3.54e+00 3.54e+00 8.34e-07
P4m(G2) 8.34e-07 8.05e-07 0.00e+00 8.34e-07
P4s(G2) 3.58e-07 5.96e-08 0.00e+00 3.58e-07
float64 max |diff| vs G0: G1 0.0e+00, G2 8.9e-16, G3 8.9e-16, P4z 3.5e+00, P4m 1.3e-15, P4s 8.9e-16, G4z 2.0e+00, G4m 1.2e-15, G4s 8.9e-16
column-gain graph, P2: ConvAdd conv1+add1: fused; ConvMul add1+mul1: refused, initializer shape (1, 1, 1, 64); ConvAdd conv2+add2: fused
nodes G0 16, after P1-P3 11; ops ['Cast', 'Sub', 'Div', 'Conv', 'Mul', 'Relu', 'MaxPool', 'Conv+Relu', 'GlobalAveragePool', 'Flatten', 'Gemm']
max |diff| vs its own G0: relu1 0.00e+00 (tol 3.5e-06), pool1 0.00e+00 (tol 3.5e-06), relu2 0.00e+00 (tol 4.5e-06), logits 0.00e+00 (tol 5.2e-07); within tol True
The data check held: the frame has the planned type and shape, the disk and square have the pixel counts of their construction, and the border ring is all background, so nothing in the image content sits under the padding.
Memory. The accounting check held for every pass: each printed saving equals the arithmetic on the shapes, G0’s total equals the construction, and the node counts fall from 16 to 14, 11 and 9, and to 7 or 8 with the normalization folded. Since the savings follow from construction, the table is a check that the passes did what they claim rather than a finding. What it shows is where the memory goes. Constant folding removes two three-element tensors, 24 bytes: its benefit is the work of recomputing constants on every run, not memory. The Conv+Add, Conv+Mul and Conv+Relu fusions remove the feature maps between each convolution and its activation, which by our arithmetic on the construction () are 131,072 of G0’s 225,318 intermediate elements, because each is as large as the convolution’s output. Folding the normalization removes the two float copies of the input. In our reading, each removed tensor would otherwise be written once and read once, so the traffic saved is twice the printed bytes; whether that saves time depends on whether the operations were memory-bound, as the edge inference article’s roofline sets out.
Function preservation. H1 held: G1, G2 and G3 agree with G0 within the tolerance at every compared tensor. Constant folding gives exactly zero difference, because the folded constants are the same float32 products computed once instead of on every run. The fused graphs differ in the last bits, and G3 differs from G0 exactly as G2 does, so the Relu epilogue changed nothing in our executor. The column-gain graph, in which Conv+Mul fusion was refused but Conv+Add and Conv+Relu were applied, shows zero difference; in our reading, the differences in G2 therefore come from Conv+Mul fusion, which scales the weights before the products are summed instead of scaling the sum.
The border. H3 held: with zero padding of the raw image, the folded convolution matches G2 to rounding in the interior (within its tolerance) and is wrong by up to 3.54 on the border ring. The check of the formula (C-pred) held: the measured error equals the predicted one to within the tolerance over the whole map, for P4m and P4s (predicted zero) as well. The error does not stay at the border of the first layer. It passes through the Relu and pooling to the second convolution and into the global average, and the logits of G4z differ from G0’s by , far above their tolerance, so a fold that is exact on every interior pixel still changes the model’s output. H4 held: padding with the per-channel mean makes the fold agree with G2 to rounding on the interior and the ring, and G4m agrees with G0 at every tensor. H5 held: folding only the division is also exact everywhere. The float64 check held: every exact graph matches G0 to about , while P4z and G4z differ by 3.5 and 2.0, so the border error is algebra, not rounding.
The refusal. The refusal check held: on G0′, Conv+Add fused, Conv+Mul was refused for the gain, and Conv+Add fused for the second convolution. Because the Mul stays between the first Conv and its Relu, the Conv+Relu fusion could not apply there either, and G0′ ends with 11 nodes, two more than G3, with a separate Mul and Relu. One refused rewrite blocks the next, which in our reading is why a single unusual operation in an exported model can cost more than its own run time.
Applying it in practice
- Start from the runtime’s defaults. ONNX Runtime enables all levels by default; TensorRT and XLA fuse and select kernels as part of building. Most of the gain needs no configuration, only an export that does not hide the patterns (see Pitfalls).
- Verify against the unoptimized graph. Run the optimized and unoptimized model, or the training framework, on the same inputs, and compare within a tolerance, as the ONNX article’s parity check does; then measure task accuracy when the optimization lowers precision.
- Fix shapes where the application allows. A camera with a fixed resolution and a batch of one can be compiled for that shape; for variable shapes, give the compiler the range (TensorRT’s minimum, optimum and maximum shapes).
- Save and pin the optimized artefact. Offline optimization or a cached engine saves start-up time, and must be rebuilt when the hardware, the runtime version or the options change.
- Measure, then look at the graph. Time the model as the latency and throughput article describes, and inspect the optimized graph (ONNX Runtime can save it; XLA and TorchInductor can print it) to see which fusions happened.
Variants
- Graph-rewriting runtimes. ONNX Runtime applies rewrites at its levels and runs the result with library kernels on each execution provider; the ONNX article covers its providers in outline.
- Compilers that generate code. XLA (through StableHLO, with LLVM and Triton back ends), Apache TVM, and PyTorch’s TorchInductor generate kernels for fused groups; the PyTorch article describes
torch.compile, and the TensorFlow article XLA’s history in TensorFlow. - Vendor inference builders. NVIDIA TensorRT builds a serialized engine for its target GPU from an ONNX file, Torch-TensorRT or its network-definition API, choosing tactics by measurement; its 11.x releases removed weakly typed networks in favour of strongly typed ones and implicit quantization in favour of explicit quantization (TensorRT README at v11.4). The deploying vision models article places it among the runtimes; Intel’s OpenVINO, which that article describes, can also fold preprocessing into the model.
- On-device runtimes. LiteRT plans its tensors into a preallocated arena, as described under Memory planning; the edge inference article describes the on-device runtimes.
- Hand-fused kernels. XLA’s documentation describes custom operations that were originally fused by hand, and lists among its objectives making automatically fused operations fast enough to remove the need for many of them (XLA architecture).
Best Practices
- Keep normalization and preprocessing visible to the optimizer, but fold them deliberately. Folding the input normalization into the first convolution is exact only if the border is handled, as the example shows; folding the scale alone, or padding with the mean, keeps it exact.
- Export in evaluation mode with constants as constants. Batch normalization folds into a convolution as exact algebra only with its statistics fixed, as in evaluation mode (ONNX), and ONNX Runtime’s basic level removes Dropout nodes, which do nothing at inference.
- Keep the fusable pairs adjacent. A conversion, reshape or custom operation between a convolution and its bias, scale or activation blocks the fusion, as the refused gain blocked both Conv+Mul and Conv+Relu in the example.
- Compare with tolerances, not equality. Fused and compiled graphs change summation order; a bitwise match, as in some of the example’s rows and the PyTorch article’s compiled model, is not to be expected in general.
- Treat compiled artefacts as build products. Record the runtime version, options and device they were built for, next to the portable model they were built from (deploying vision models).
Pitfalls
- An exact-looking fold that is not exact. Moving an affine map across a padded convolution, or in our reading any operation that treats borders or constants specially, changes results at those places, as the zero-padded normalization fold did on the border ring and then in every logit.
- Silent non-application. Rewrites check conditions and refuse quietly when they fail: a constant of the wrong shape, a second consumer of the convolution’s output, an output that is also a graph output, or nodes assigned to different providers. In ONNX Runtime 1.32.0, Conv+Activation fusion is not applied on the CUDA provider by that transformer at all. The way to know what happened is to inspect the optimized graph.
- Optimized models tied to their environment. ONNX Runtime’s offline models need the same options and hardware as the target, and layout-optimized ones a compatible CPU; TensorRT engines are specific to their settings and GPU and, per ONNX Runtime’s provider documentation, not portable: they must be rebuilt when those settings change, and a timing cache is used only on a matching GPU unless forced.
- Non-deterministic builds. TensorRT engine builds are not bit-deterministic, because kernel timings vary between runs (trtexec README); two builds of the same model can differ, and the engine to ship is the saved one.
- Benchmarks that include start-up or copies. Online optimization, compilation on the first call and host–device copies inside each run can dominate measured time; time steady-state runs and move inputs to the device beforehand where the runtime allows.
Evidence and Impact
Sources describe these transformations mostly as mechanisms rather than with measured speedups. XLA’s documentation calls fusion its most important optimization and gives the reason, memory-bound GPU workloads, without a figure in the pages read here. ONNX Runtime’s documentation presents its levels as performance improvements and offline mode as a saving of start-up time, also without figures for vision models. The one aggregate figure in a neighbouring article is for a whole compiler stack: as the PyTorch article reports, Ansel et al. (2024) give geometric-mean speedups of TorchInductor over eager PyTorch across more than 180 models, under a specific set-up described there; it combines fusion, code generation and overhead reduction and does not separate them.
What the example shows is narrower and exact: on one small graph, fusions that keep the function remove most of the bytes passed between nodes, and an algebraic fold that is correct on every interior pixel is wrong at the image border unless the padding is changed with it. The memory saved is a property of the graph; the time saved depends on the hardware, which is why runtimes measure kernels on the target rather than predict them.
Related
- PyTorch
An open-source Python library for building and training neural networks with tensors, automatic differentiation and GPU acceleration, run eagerly as ordinary Python code and optionally compiled or exported for deployment.
- Edge Inference
Running a vision model on the device that captures the images, such as a phone, camera, robot or microcontroller, instead of in a data centre, covering why it is done, the limits of compute, memory, power and heat, the classes of hardware and runtimes, the roofline model for finding memory-bound layers, split and offloaded computation, and how latency and energy are measured.
- Model Quantization
Storing and computing a trained network's weights and activations in low-precision formats such as INT8, INT4 or FP8, with a scale and zero point per tensor, channel or group; the integer arithmetic behind it, post-training calibration, quantization-aware training with the straight-through estimator, and what it costs in accuracy.
- Deploying Vision Models
Getting a trained vision model into use, from export and the choice of runtime to packaging the model with its preprocessing contract, serving it on a server or a device, keeping pre- and post-processing identical to training, and versioning, rolling back and monitoring it.
- Model Compression
Making a trained vision model smaller and cheaper to run by pruning weights or filters, quantizing, distilling it into a smaller network, factorizing its layers or choosing a compact architecture, and how to compare the results fairly in accuracy, size, operations and measured latency.
- TensorFlow
Google's open-source machine learning framework, which runs computations as dataflow graphs, executes eagerly by default since version 2.0, and pairs Keras for model building with tf.data for input pipelines, SavedModel for export, TensorFlow Serving for servers and LiteRT for devices.
References
- ONNX Runtime documentation. Graph Optimizations in ONNX Runtime, docs/performance/model-optimizations/graph-optimizations.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
- ONNX Runtime documentation. TensorRT Execution Provider, docs/execution-providers/TensorRT-ExecutionProvider.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
- ONNX Runtime documentation. I/O Binding, docs/performance/tune-performance/iobinding.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
- ONNX Runtime source, onnxruntime/core/optimizer/conv_add_fusion.cc, microsoft/onnxruntime at tag v1.32.0 (MIT License).
- ONNX Runtime source, onnxruntime/core/optimizer/conv_mul_fusion.cc, microsoft/onnxruntime at tag v1.32.0 (MIT License).
- ONNX Runtime source, onnxruntime/core/optimizer/conv_activation_fusion.cc, microsoft/onnxruntime at tag v1.32.0 (MIT License).
- ONNX Runtime source, onnxruntime/core/optimizer/graph_transformer_utils.cc, microsoft/onnxruntime at tag v1.32.0 (MIT License).
- ONNX Runtime source, onnxruntime/core/optimizer/constant_folding.cc, microsoft/onnxruntime at tag v1.32.0 (MIT License).
- ONNX Runtime source, onnxruntime/core/framework/allocation_planner.cc, microsoft/onnxruntime at tag v1.32.0 (MIT License).
- ONNX Runtime release v1.32.0 (GitHub release of microsoft/onnxruntime, 13 November 2026).
- OpenXLA. XLA architecture, docs/architecture.md, openxla/xla at commit 45c7aaa (16 November 2026; Apache License 2.0).
- OpenXLA. XLA:GPU Architecture Overview, docs/gpu_architecture.md, openxla/xla at commit 45c7aaa (16 November 2026).
- Apache TVM README, apache/tvm at tag v0.27.0 (Apache License 2.0).
- NVIDIA TensorRT open-source components. README, NVIDIA/TensorRT at tag v11.4 (9 October 2026; Apache License 2.0).
- NVIDIA TensorRT open-source components. sampleEditableTimingCache README, samples/sampleEditableTimingCache/README.md, NVIDIA/TensorRT at tag v11.4.
- NVIDIA TensorRT open-source components. trtexec README, samples/trtexec/README.md, NVIDIA/TensorRT at tag v11.4.
- LiteRT source, tflite/arena_planner.h, google-ai-edge/LiteRT at tag v2.3.2 (26 October 2026; Apache License 2.0).