Vision Systems & Engineering

ONNX Runtime

Microsoft's open-source inference engine for ONNX models, which optimizes a model's graph, partitions it among execution providers for the hardware present, inserts the copies between devices and runs it, with a worked example of partitioning and copy insertion.

intermediate

ONNX Runtime is Microsoft’s open-source inference engine for ONNX models, released under the MIT License. It loads an .onnx file, rewrites its graph, divides the graph among execution providers, the interfaces through which it reaches CPUs, GPUs and neural processing units, and runs it. The same program and the same API serve a server with an NVIDIA GPU, a laptop CPU, a phone with a neural processing unit and a web page. A vision team meets it, for example, when a model exported from PyTorch has to run without PyTorch: in a C++ or C# service, on a mobile device, or behind OpenCV’s dnn module, which can be compiled with an ONNX Runtime back end.

The ONNX article covers the format, the exporters and opsets, and gives ONNX Runtime one paragraph; its examples run an ONNX Runtime session checked against PyTorch and a model whose opset ONNX Runtime 1.31.1 refused. The Inference Optimization article covers the graph rewrites ONNX Runtime applies, its fusions and their conditions. This article covers the runtime itself: what happens between loading a model and running it, how execution providers are chosen and how the graph is split among them, where copies between devices come from, threading, I/O binding, the reduced builds for mobile and web, the precision tools, and the versions as of November 2026. Statements about the code come from the source at tag v1.32.0 (13 November 2026). Statements about the documentation come from its source on the repository’s gh-pages branch at commit e3d000e (14 October 2026), which predates that release; each is dated by its source.

Purpose

An exported model says what to compute, not where or how. The ONNX Runtime documentation describes the execution-provider framework as the means by which the runtime “works with different hardware acceleration libraries” to run a model on whatever platform it is deployed to, with one API across language bindings. In practice that means three jobs:

  • Running ONNX without the training framework. The runtime has its own kernels for the standard operators on the CPU, plus “contrib” operators of its own. A model that loads runs on a CPU even when no accelerator is present.
  • Using the accelerator that is there. Hardware vendors’ libraries (CUDA, TensorRT, OpenVINO, DirectML, Core ML, NNAPI, QNN and others) are wrapped as providers. The application lists the providers it wants in priority order, and the runtime hands each one the parts of the graph it can run.
  • Making the graph cheaper first. Before and after partitioning, the runtime applies function-preserving rewrites: constant folding, fusions and layout changes. Inference Optimization describes them.

Core Capabilities

The life of a session

Everything happens when an InferenceSession is created. Run() then only executes the prepared plan. Pieced together from the documentation and the v1.32.0 source, a session is built in these steps:

  1. Load. The model is read from an ONNX file, or from an ORT-format file in reduced builds (below).
  2. Basic graph optimizations. The documentation describes these as semantics-preserving rewrites that “run before graph partitioning and thus apply to all the execution providers”: constant folding, removal of redundant nodes (Identity, Slice, Unsqueeze, Dropout) and fusions such as Conv+Add, Conv+Mul and Conv+BatchNorm.
  3. Partitioning. The graph is divided among the registered providers (next section).
  4. Extended and layout optimizations. The documentation says extended fusions run after partitioning and only on nodes assigned to the CPU, CUDA or ROCm providers, with a table that says which provider each fusion applies to. Layout optimizations, such as the NCHWc layout, run only on nodes assigned to the CPU provider.
  5. Copy insertion. Where a tensor passes between nodes on different devices, copy nodes are added (transformer_memcpy.cc).
  6. Allocation planning. Before execution, a planner decides which tensors can reuse the memory of tensors that are no longer needed. In allocation_planner.cc a newly produced value can take over a freed buffer whose shape and type are compatible, or reuse one of the node’s inputs in place. The v1.32.0 release notes report that replacing linear free-buffer scans with an indexed lookup, together with other fixes, cut session creation for constant-heavy graphs of 100,000 nodes “from over 10 minutes to ~5 seconds”.
  7. Execution. The default execution mode is sequential, one node after another (threading documentation).

All three optimization levels are enabled by default. GraphOptimizationLevel selects ORT_DISABLE_ALL, ORT_ENABLE_BASIC, ORT_ENABLE_EXTENDED or ORT_ENABLE_ALL, and each level includes the ones before it. Setting optimized_model_filepath saves the optimized graph, so later sessions can skip the work (“offline mode”). The documentation warns that the saved model must be produced with the same options (execution providers, optimization level) and hardware as the target machine. A model saved with layout optimizations for AVX2, for example, needs a CPU that supports AVX2.

Execution providers and how the graph is split

A provider tells the runtime which nodes it can run through a GetCapability() call, and the providers’ order is the user’s priority. The documentation’s own example, ['CUDAExecutionProvider', 'CPUExecutionProvider'], means “execute a node using CUDAExecutionProvider if capable, otherwise execute using CPUExecutionProvider”. In Python the list is passed as providers= when the session is created. get_providers() returns the registered providers, and set_providers() registers a new list and re-creates the underlying session.

The partitioner in graph_partitioner.cc (v1.32.0) is, in its own words, a “greedy partitioning algorithm per provider preferences”:

  • Providers are processed one at a time, in order. Each one’s GetCapability() returns groups of nodes it can take: single nodes, or subgraphs with a description (a “MetaDef”) of the fused node they should become.
  • A group is assigned to that provider only if none of its nodes has been assigned already. A multi-node subgraph that contains a node already taken by an earlier provider is skipped as a whole, because, as a source comment says, the provider is assumed to run the subgraph “as a whole unit”. A later provider therefore cannot take a node from an earlier one, since assignment follows priority.
  • A subgraph that is taken is replaced by one fused node, named after the provider. If no kernel is registered for it, the provider compiles it. In our reading, this is the path by which a compiling provider such as TensorRT turns each claimed subgraph into an engine.
  • The CPU provider “is expected to be able to run any node” and comes last, so it takes whatever is left.

The real partitioner does more than this core. Nested control-flow subgraphs are partitioned first, bottom up. Providers that prefer the NHWC layout get a two-pass flow with a layout transformation in between. There is an optional resource accountant that limits what a provider may claim, and a path for precompiled “EP context” nodes. Functions without a kernel are inlined and partitioning is repeated. Separately, fallback_cpu_capability.cc provides a CPU-preference pass, GetCpuPreferredNodes. Given the nodes a provider has tentatively claimed, it finds those that only consume tensors already on the CPU (small initializers of at most 100 elements, graph inputs or CPU outputs), usually a shape computation, and forces them onto the CPU to avoid the copy. It never does so for a node with a float16, bfloat16, float8 or float4 input, types that, the comment says, have little to no CPU kernel coverage.

As of the documentation at e3d000e (14 October 2026), the provider table lists the default CPU provider, Intel DNNL (its page is the oneDNN provider’s), TVM (preview), Intel OpenVINO, XNNPACK and AMD ROCm (deprecated); NVIDIA CUDA and TensorRT, DirectML, AMD MIGraphX, Qualcomm QNN and WebGPU; Android NNAPI, Arm NN (preview), the Arm Compute Library (preview) and Core ML (preview); and Rockchip NPU, Xilinx Vitis AI, Huawei CANN and Azure (all preview). The table lags the code: the v1.32.0 release removed the Arm Compute Library provider, which had been deprecated in 1.31.0. The documentation also notes that a package built with several providers needs all their dependent libraries in the execution environment.

Choosing providers is therefore a matter of listing the accelerator first and the CPU last, then checking what the accelerator actually took, which depends on its operator coverage. The Edge AI Accelerators article explains why an accelerator runs only the operators it implements and why each switch to another processor costs a transfer.

Copies between devices

Once each node has a provider, transformer_memcpy.cc makes the data movement explicit. Its overview comment describes three cases, and ModifyGraph implements them:

  • A value computed by a provider node and read by a non-provider node gets a new value and one copy node, MemcpyToHost, from the device to the host.
  • A value computed on the host and read by a provider node gets one MemcpyFromHost node.
  • An initializer read by both kinds of node is duplicated, so each device has its own copy.

There is one copy per crossing value, not per reader: every provider node that reads the value is rewired to the same copy. Graph inputs get a copy node only when both provider and non-provider nodes read them; otherwise, a source comment says, utils::CopyInputsAcrossDevices() copies them when the model runs. The transformer acts for the first provider in the list that is not CPU-based, and for the CUDA provider it logs a warning when it adds copies, saying they “might have negative impact on performance (including unable to run CUDA graph)”. The copy nodes themselves are assigned to the provider.

Threading

For the CPU provider the documentation lists the following settings (threading documentation, e3d000e):

  • intra_op_num_threads is the total number of threads that parallelize work inside each operator. The default, 0, means one thread per physical core, with threads after the first pinned to their cores. An explicit count turns the pinning off.
  • execution_mode is ORT_SEQUENTIAL by default. ORT_PARALLEL runs independent nodes concurrently on an inter-op pool sized by inter_op_num_threads. The documentation says this usually helps models with many branches and can hurt models without them.
  • Spinning. By default, idle pool threads spin while waiting for work, which the documentation says gives faster inference at the cost of CPU cycles and power. session.intra_op.allow_spinning turns it off, spin_duration_us bounds it in time, and spin_backoff_max adds exponential back-off.
  • Affinity and NUMA. session.intra_op_thread_affinities pins threads to chosen cores. The documentation recommends testing settings on NUMA machines.

Several sessions in one process each start their own pools unless a global thread pool is created. The memory documentation adds that an arena allocator can be shared between sessions to reduce memory.

I/O binding

When the providers are not CPU-based, Run() copies inputs from the host to the device and outputs back to the host. The I/O-binding documentation warns that these copies can take most of the time, “misleading users into thinking ORT is slow”. IoBinding lets the application place inputs on the device and bind outputs to device memory before calling Run(). An output can be bound to a device without a fixed shape, so the session allocates it at the size it turns out to need, or bound to a pre-allocated tensor that is reused. For a camera pipeline that preprocesses on the GPU, this keeps frames on the device from decoding to postprocessing (our reading of the documentation; Latency and Throughput covers timing inference, including on devices that queue work asynchronously).

TensorRT and other compiling providers

The TensorRT provider’s options show what partitioning means in practice. trt_min_subgraph_size is the “minimum node size in a subgraph after partitioning”, and smaller subgraphs “will fall back to other execution providers” (default 1). trt_max_partition_iterations (default 1000) bounds the provider’s partitioning, after which the whole model falls back to other providers such as CUDA or the CPU. trt_op_types_to_exclude keeps chosen operator types off TensorRT. Its example provider list puts TensorRT first and CUDA second, and the documentation recommends registering CUDA so that nodes TensorRT does not support can go to CUDA rather than the CPU.

Building a TensorRT engine can take long, so the provider can cache engines (trt_engine_cache_enable, trt_engine_cache_path) and a timing cache. The documentation says each engine is built for specific settings and GPUs and “it’s not portable”, so a changed setting means rebuilding the engine, and it warns that cached engine and profile files must be cleaned up when the model, the ONNX Runtime version or the TensorRT version changes. The requirements table at e3d000e pairs ONNX Runtime 1.22, and main, with TensorRT 10.9 and has no rows for later releases.

Reduced builds and the ORT format

For phones and the web, ONNX Runtime has reduced-size builds that read the ORT format, an internal format that “is supported by reduced size ONNX Runtime builds” (ORT format documentation). A full build reads both formats. The script python -m onnxruntime.tools.convert_onnx_models_to_ort loads, optimizes and saves each model as an .ort file. It also writes a configuration file listing the operators, and optionally the types, the models need, from which a reduced-operator build can be compiled. The documentation’s example of type reduction is Softmax: a model that uses it only on float data does not need the double implementation. The optimization style is “Fixed”, which bakes in platform-specific optimizations, or “Runtime”, recommended with compiling providers such as NNAPI or Core ML, which saves certain optimizations to be applied at runtime. Format versions changed: the ORT format was introduced in 1.5, version 5 (from 1.13) removed kernel definition hashes, and 1.14 and later read v5 and v4 with limited support. On a device, session.use_ort_model_bytes_directly lets the session use the model buffer without copying it, which may reduce peak memory. The v1.32.0 release notes state that it is the last release for CocoaPods and that iOS packaging moves to the Swift Package Manager.

Precision tools

Two tool sets prepare reduced-precision models. Their output still runs through the same session.

  • Float16. The documentation converts a model with convert_float_to_float16 from the separate onnxconverter-common package. Options keep inputs and outputs in float32 (keep_io_types) and leave listed operators or nodes in float32, with casts inserted around them. Because, in the documentation’s words, “the CPU version of ONNX Runtime doesn’t support float16 ops”, the automatic mixed-precision tool, which converts clusters of operators and splits a cluster in half when it fails a tolerance, must run on a machine with a GPU. Mixed precision training covers the float16 format.
  • Quantization. The onnxruntime.quantization tools produce 8-bit linear quantization in two representations: QOperator (quantized operators such as QLinearConv) or QDQ (QuantizeLinear and DequantizeLinear pairs around the original operators). They support dynamic quantization and static quantization calibrated with MinMax, Entropy or Percentile, plus a pre-processing step and a debugging API that matches tensors between the float and the quantized model. The theory is in Quantization.

Plugin execution providers

Recent releases are moving providers out of the main package. The v1.32.0 notes describe a CUDA plugin EP v0.4.0 and a WebGPU plugin EP as “matched” with the release, and warn that core and plugin must use the same contrib-operator schemas: a mismatch “can cause incorrect execution or crashes”, and building both from the same revision is recommended. The in-tree NvTensorRTRTXExecutionProvider was removed in favour of a standalone NVIDIA TensorRT RTX plugin registered with OrtApi::RegisterExecutionProviderLibrary. The repository’s release list shows the plugins with their own tags, for example plugin-ep-cuda/v0.4.0 on 14 November 2026.

Minimal Example

ONNX Runtime is not installed on the machine used for this article, and nothing could be installed, so the example does not call it. Instead it reproduces the core of partitioning and copy insertion in NumPy, following graph_partitioner.cc and transformer_memcpy.cc at v1.32.0. It is our analogue, written from the source and not ported. The outcomes test the analogue, not ONNX Runtime.

The analogue has these parts:

  • Graph. An 11-node segmentation-style graph over one rendered 64×64 grey image: Cast (uint8 to float32), Div by 255, Conv, Relu, MaxPool, Conv, Relu, Resize (nearest, ×2), Conv, Softmax and ArgMax over three channels. The graph has two outputs, the probabilities (float32) and the label map (int64).
  • Weights. The weights are untrained and drawn from a seeded generator. The output is a test pattern, not a segmentation.
  • Devices and kernels. There are two “devices”, a hypothetical accelerator (ACC) and the CPU. Both run the same NumPy kernel for each operator, so a correct partition must reproduce the all-CPU result bit for bit. The convolution is written as shift-and-accumulate in a fixed order rather than as a BLAS call, so that, in our reasoning, its result does not depend on where its buffers sit in memory.
  • Partitioner. Providers are processed in priority order, a group is assigned only if none of its nodes is taken, and the CPU claims each remaining node. How the accelerator groups its nodes is our choice: it forms maximal runs of consecutive supported nodes (the graph is a chain) and drops runs smaller than a min_size modelled on trt_min_subgraph_size. In ONNX Runtime, each provider’s GetCapability() forms its own subgraphs, and graph_partitioner.cc contains no such rule.
  • Copies. Copy insertion adds one MemcpyToHost or MemcpyFromHost node per value that crosses devices inside the graph. Graph inputs and outputs are copied by Run() and counted separately.
  • Executor. The executor tags every value with its device and raises an error when a node reads a value that is not on its device.

Five scenarios run:

  • A: the accelerator supports every operator.
  • B: the accelerator lacks Resize, a node in the middle of the graph.
  • C3 and C4: as B, with minimum subgraph sizes 3 and 4.
  • D: the accelerator supports everything but is listed after the CPU.

A negative control removes one copy from B and must be caught by the device check.

The expected assignments, copies and byte counts below were derived by hand from the graph’s shapes before any script ran, then confirmed by a planning script that ran the graph on the CPU only and printed each tensor’s size. No bar was set after a script, since every check compares exact integers. The script uses NumPy 2.3.5, runs in about 0.1 seconds, and printed identical output on two runs.

import os
for v in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ[v] = "1"                      # one BLAS thread, set before NumPy loads
import numpy as np
print("numpy", np.__version__)

# ---- data: a rendered 64x64 grey image (uint8), NCHW, batch 1 -----------------
yy, xx = np.mgrid[0:64, 0:64] + 0.5
img = np.full((64, 64), 40, np.uint8)
img[(xx - 20) ** 2 + (yy - 22) ** 2 <= 12 ** 2] = 200      # disk
img[36:52, 38:54] = 120                                      # square
img = img[None, None]

# ---- untrained, seeded weights; initializers ---------------------------------
rng = np.random.default_rng(0)
init = {"w1": rng.normal(0, 0.5, (8, 1, 3, 3)), "b1": rng.normal(0, 0.1, 8),
        "w2": rng.normal(0, 0.2, (8, 8, 3, 3)), "b2": rng.normal(0, 0.1, 8),
        "w3": rng.normal(0, 0.2, (3, 8, 3, 3)), "b3": rng.normal(0, 0.1, 3)}
init = {k: v.astype(np.float32) for k, v in init.items()}
init["k255"] = np.float32(255)

# ---- one NumPy kernel per operator, shared by every "device" ------------------
def conv(x, w, b):                           # 3x3, pad 1, fixed accumulation order
    n, c, h, wd = x.shape
    xp = np.zeros((n, c, h + 2, wd + 2), np.float32); xp[:, :, 1:-1, 1:-1] = x
    out = np.zeros((n, w.shape[0], h, wd), np.float32)
    for o in range(w.shape[0]):
        acc = np.full((n, h, wd), b[o], np.float32)
        for i in range(c):
            for dy in range(3):
                for dx in range(3):
                    acc = acc + w[o, i, dy, dx] * xp[:, i, dy:dy + h, dx:dx + wd]
        out[:, o] = acc
    return out

def softmax(x):                              # over channels, sum written out
    e = np.exp(x - x.max(axis=1, keepdims=True))
    return e / (e[:, 0:1] + e[:, 1:2] + e[:, 2:3])

KERNELS = {
    "Cast": lambda x: x.astype(np.float32),
    "Div": lambda x, k: x / k,
    "Conv": conv,
    "Relu": lambda x: np.maximum(x, np.float32(0)),
    "MaxPool": lambda x: x.reshape(x.shape[0], x.shape[1], x.shape[2] // 2, 2,
                                   x.shape[3] // 2, 2).max(axis=(3, 5)),
    "Resize": lambda x: x.repeat(2, axis=2).repeat(2, axis=3),   # nearest, x2
    "Softmax": softmax,
    "ArgMax": lambda x: x.argmax(axis=1, keepdims=True).astype(np.int64),
}

# ---- the graph: (name, op, inputs, output), in topological order --------------
GRAPH = [("n0", "Cast", ["image"], "x0"), ("n1", "Div", ["x0", "k255"], "x1"),
         ("n2", "Conv", ["x1", "w1", "b1"], "x2"), ("n3", "Relu", ["x2"], "x3"),
         ("n4", "MaxPool", ["x3"], "x4"), ("n5", "Conv", ["x4", "w2", "b2"], "x5"),
         ("n6", "Relu", ["x5"], "x6"), ("n7", "Resize", ["x6"], "x7"),
         ("n8", "Conv", ["x7", "w3", "b3"], "x8"), ("n9", "Softmax", ["x8"], "probs"),
         ("n10", "ArgMax", ["probs"], "labels")]
INPUTS, OUTPUTS = ["image"], ["probs", "labels"]
ALL_OPS = set(KERNELS)

# ---- partitioning: providers in priority order, unassigned nodes only ---------
def capability(provider, ops, assigned, min_size):
    """Groups of unassigned nodes the provider can run. CPU: one node each.
    ACC: maximal runs of consecutive supported nodes (the graph is a chain),
    runs smaller than min_size dropped, as trt_min_subgraph_size does."""
    free = [i for i, (_, op, _, _) in enumerate(GRAPH) if assigned[i] is None and op in ops]
    if provider == "CPU":
        return [[i] for i in free]
    runs, run = [], []
    for i in free:
        if run and i != run[-1] + 1:
            runs.append(run); run = []
        run.append(i)
    if run:
        runs.append(run)
    return [r for r in runs if len(r) >= min_size]

def partition(providers, acc_ops, min_size=1):
    assigned, subgraphs = [None] * len(GRAPH), []
    for p in providers:
        ops = ALL_OPS if p == "CPU" else acc_ops
        for group in capability(p, ops, assigned, min_size):
            if all(assigned[i] is None for i in group):   # never steal a node
                for i in group:
                    assigned[i] = p
                if p != "CPU":
                    subgraphs.append(group)
    return assigned, subgraphs

# ---- copy insertion: one copy per value that crosses devices inside the graph -
def insert_copies(assigned):
    producer = {out: assigned[i] for i, (_, _, _, out) in enumerate(GRAPH)}
    nodes, renamed, copies = [], {}, []
    for i, (name, op, ins, out) in enumerate(GRAPH):
        dev = assigned[i]
        new_ins = []
        for v in ins:
            src = producer.get(v)              # None: graph input or initializer
            if src is not None and src != dev:
                if (v, dev) not in renamed:
                    kind = "MemcpyToHost" if dev == "CPU" else "MemcpyFromHost"
                    renamed[(v, dev)] = f"{v}_{dev}"
                    nodes.append((f"copy_{v}", kind, [v], f"{v}_{dev}", dev))
                    copies.append((kind, v))
                v = renamed[(v, dev)]
            new_ins.append(v)
        nodes.append((name, op, new_ins, out, dev))
    return nodes, copies

# ---- executor with a device check ----------------------------------------------
def run(nodes, image):
    vals = {}                                  # value name -> (array, device)
    for name, op, ins, out, dev in nodes:      # initializers live where they are read
        for v in ins:
            if v in init:
                vals[v] = (init[v], dev)
    bytes_in = bytes_out = 0
    first_dev = {v: dev for _, _, ins, _, dev in reversed(nodes) for v in ins}
    vals["image"] = (image, "CPU")
    if first_dev["image"] != "CPU":            # Run() copies the input to the device
        vals["image"] = (image.copy(), first_dev["image"]); bytes_in += image.nbytes
    same_dev_copies = 0
    for name, op, ins, out, dev in nodes:
        if op.startswith("Memcpy"):            # a copy reads from the other device
            same_dev_copies += vals[ins[0]][1] == dev
            vals[out] = (np.array(vals[ins[0]][0], copy=True), dev)
            continue
        for v in ins:
            if vals[v][1] != dev:
                raise RuntimeError(f"device check: node {name} ({op}) on {dev} "
                                   f"reads {v} held on {vals[v][1]}")
        vals[out] = (KERNELS[op](*[vals[v][0] for v in ins]), dev)
    for v in OUTPUTS:                          # Run() copies device outputs to the host
        if vals[v][1] != "CPU":
            bytes_out += vals[v][0].nbytes
    return {v: vals[v][0] for v in OUTPUTS}, vals, bytes_in, bytes_out, same_dev_copies

# ---- reference: everything on the CPU provider ---------------------------------
ref_nodes, _ = insert_copies(partition(["CPU"], set())[0])
ref, ref_vals, _, _, _ = run(ref_nodes, img)
ref2, _, _, _, _ = run(ref_nodes, img.copy())
print(f"D1 image {img.dtype} {img.shape}, grey levels {np.unique(img).tolist()}")
print(f"D2 reference labels take values {np.unique(ref['labels']).tolist()}")
print(f"D3 two reference runs, differing elements: probs "
      f"{int((ref['probs'] != ref2['probs']).sum())}, labels {int((ref['labels'] != ref2['labels']).sum())}")

SCENARIOS = [("A ", ["ACC", "CPU"], ALL_OPS, 1),
             ("B ", ["ACC", "CPU"], ALL_OPS - {"Resize"}, 1),
             ("C3", ["ACC", "CPU"], ALL_OPS - {"Resize"}, 3),
             ("C4", ["ACC", "CPU"], ALL_OPS - {"Resize"}, 4),
             ("D ", ["CPU", "ACC"], ALL_OPS, 1)]
for tag, providers, acc_ops, min_size in SCENARIOS:
    assigned, subgraphs = partition(providers, acc_ops, min_size)
    nodes, copies = insert_copies(assigned)
    out, vals, b_in, b_out, same = run(nodes, img)
    copy_bytes = [ref_vals[v][0].nbytes for _, v in copies]
    print(f"\n{tag} providers {providers}, min_subgraph_size {min_size}")
    print("   assignment", "".join("A" if a == "ACC" else "c" for a in assigned),
          "| ACC subgraphs", [f"n{g[0]}-n{g[-1]} ({len(g)})" for g in subgraphs])
    print("   copy nodes", len(copies), [f"{k}({v}) {b}" for (k, v), b in zip(copies, copy_bytes)],
          "| in-graph bytes", sum(copy_bytes), "| Run() bytes in", b_in, "out", b_out)
    print("   copies reading from their own device (must be 0):", same)
    print("   differing elements vs CPU: probs", int((out["probs"] != ref["probs"]).sum()),
          "labels", int((out["labels"] != ref["labels"]).sum()))

# ---- negative control: scenario B with the copy before n8 left out --------------
assigned, _ = partition(["ACC", "CPU"], ALL_OPS - {"Resize"})
nodes, _ = insert_copies(assigned)
nodes = [nd for nd in nodes if nd[0] != "copy_x7"]
nodes = [(n, op, [v if v != "x7_ACC" else "x7" for v in ins], out, dev)
         for n, op, ins, out, dev in nodes]
try:
    run(nodes, img)
    print("\nC-neg: no error raised")
except RuntimeError as e:
    print("\nC-neg:", e)

Output:

numpy 2.3.5
D1 image uint8 (1, 1, 64, 64), grey levels [40, 120, 200]
D2 reference labels take values [0, 1, 2]
D3 two reference runs, differing elements: probs 0, labels 0

A  providers ['ACC', 'CPU'], min_subgraph_size 1
   assignment AAAAAAAAAAA | ACC subgraphs ['n0-n10 (11)']
   copy nodes 0 [] | in-graph bytes 0 | Run() bytes in 4096 out 81920
   copies reading from their own device (must be 0): 0
   differing elements vs CPU: probs 0 labels 0

B  providers ['ACC', 'CPU'], min_subgraph_size 1
   assignment AAAAAAAcAAA | ACC subgraphs ['n0-n6 (7)', 'n8-n10 (3)']
   copy nodes 2 ['MemcpyToHost(x6) 32768', 'MemcpyFromHost(x7) 131072'] | in-graph bytes 163840 | Run() bytes in 4096 out 81920
   copies reading from their own device (must be 0): 0
   differing elements vs CPU: probs 0 labels 0

C3 providers ['ACC', 'CPU'], min_subgraph_size 3
   assignment AAAAAAAcAAA | ACC subgraphs ['n0-n6 (7)', 'n8-n10 (3)']
   copy nodes 2 ['MemcpyToHost(x6) 32768', 'MemcpyFromHost(x7) 131072'] | in-graph bytes 163840 | Run() bytes in 4096 out 81920
   copies reading from their own device (must be 0): 0
   differing elements vs CPU: probs 0 labels 0

C4 providers ['ACC', 'CPU'], min_subgraph_size 4
   assignment AAAAAAAcccc | ACC subgraphs ['n0-n6 (7)']
   copy nodes 1 ['MemcpyToHost(x6) 32768'] | in-graph bytes 32768 | Run() bytes in 4096 out 0
   copies reading from their own device (must be 0): 0
   differing elements vs CPU: probs 0 labels 0

D  providers ['CPU', 'ACC'], min_subgraph_size 1
   assignment ccccccccccc | ACC subgraphs []
   copy nodes 0 [] | in-graph bytes 0 | Run() bytes in 0 out 0
   copies reading from their own device (must be 0): 0
   differing elements vs CPU: probs 0 labels 0

C-neg: device check: node n8 (Conv) on ACC reads x7 held on CPU

The data checks held. The image is uint8 with three grey levels. The reference label map takes all three values, so the label output is not constant. Two reference runs on copies of the image agree in every element.

Bitwise agreement (H1) held in all five scenarios. In A, B, C3, C4 and D, 0 elements of the float32 probabilities and 0 of the labels differ from the all-CPU run. Because both “devices” run the same kernels, this is a test of the wiring: every value reaches the right node, and every copy carries the value it should. On real hardware, in our reading, different kernels on different devices would in general agree only within rounding, as the parity checks in the ONNX article show for one runtime against PyTorch.

One accelerator subgraph when everything is supported (A) held. All eleven nodes went to the accelerator as one subgraph, with no copy nodes. The only transfers are at the Run() boundary: 4,096 bytes of uint8 image in, and 81,920 bytes of probabilities and labels out.

Removing one operator from the middle splits the graph (H2, scenario B) held. Without Resize, the accelerator’s claim breaks into two subgraphs, n0–n6 and n8–n10, with Resize alone on the CPU between them. Two copy nodes appear: MemcpyToHost of the 8×32×32 feature map before Resize (32,768 bytes) and MemcpyFromHost of the upsampled 8×64×64 map after it (131,072 bytes), 163,840 bytes in all. These counts and byte sizes equal the values derived by hand from the tensor shapes before any script ran (8 × 32 × 32 and 8 × 64 × 64 float32 values of 4 bytes each). That comparison, which does not depend on the copy code, is the independent check of the repair described below. The Run() boundary copies are the same as in A. One unsupported node in the middle of a graph costs two copies, and in our reading the second can be larger than the first when the unsupported operator enlarges its tensor, as Resize does here by a factor of four.

The minimum subgraph size (H3 and H3b) held.

  • C3: with a minimum of 3, nothing changed from B, because the three-node tail is not smaller than 3, and the documented rule sends back only subgraphs of “smaller size”.
  • C4: with a minimum of 4, the tail of Conv, Softmax and ArgMax fell back to the CPU. That removed the MemcpyFromHost copy of the upsampled map (131,072 bytes in B) and the 81,920-byte output transfer, since the outputs are now produced on the host.

Whether that is faster depends on how fast the CPU runs those three nodes compared with the transfers saved, which the example does not measure. The option is there to make that trade.

Priority decides (H4, scenario D) held. With the CPU listed first, it claims every node, and the accelerator, though it supports everything, gets nothing. The partitioner never moves a node that an earlier provider has taken.

Both checks that the copies are right held. No copy read a value already on its own device, in any scenario. That check nearly holds by construction, since copies are inserted only where producer and reader devices differ; it could fail only if the executor’s device tags drifted from the assignment. The independent check of the repair is H2’s comparison with the counts and bytes derived by hand before any script ran. In the negative control, the run stopped at n8, the convolution that reads the uncopied Resize output. In our analogue, NumPy would have computed the right numbers even without the copy, because both “devices” share the host’s memory. Only the device check reveals the missing copy, which is why the executor must check placement rather than trust the arithmetic.

Process note. The first pilot of this script stopped with an error after scenario A. The executor applied its device check to copy nodes too, although a copy node reads, by construction, a value held on the other device. The plan’s description of the executor had the same flaw. With the editor’s approval, copy nodes now skip the check, and every scenario prints a new check that counts copies reading from their own device, which must be 0. Nothing else changed, and the repaired pilot was fixed in advance as final; it is the run shown. Scenario A’s outcomes in the repaired run are identical to those of the first pilot. Among the departures from the batch plan that the plan states itself, two change the scenarios: the minimum subgraph size is run at 4 as well as at the batch plan’s 3, because with this graph a minimum of 3 cannot move the three-node tail; and scenario D was added to test priority order.

Ecosystem and Integrations

  • Exporters. The ONNX article describes how PyTorch’s torch.onnx.export produces the files ONNX Runtime reads, and how the exporter’s verify=True option runs the exported model in ONNX Runtime as a check. TensorFlow, Keras and scikit-learn models reach it through the converters that article lists.
  • Serving. The model deployment article serves a vision model through ONNX Runtime with its preprocessing contract and version stored in the file’s metadata.
  • OpenCV. The OpenCV article reports that an ONNX Runtime back end can be compiled into the dnn module, and that GPU inference in OpenCV 5.0 needs either its classic engine or that back end.
  • Devices. The edge inference article places ONNX Runtime among on-device runtimes, with the mobile providers Core ML and NNAPI. The ORT format and reduced builds described above are the mobile and web packaging. The ORT format documentation lists C, C++, Java and Kotlin on Android, C, C++ and Objective-C on iOS, and JavaScript on the web.
  • Hardware vendors. Each provider wraps a vendor library: CUDA and TensorRT, OpenVINO, DirectML, QNN, MIGraphX and others. A provider’s documentation, as for TensorRT, lists its own options and version requirements.

Alternatives

These are other programs that execute a trained vision model, described as their own articles or documentation describe them. Alongside each, in our framing, is how it relates to ONNX Runtime.

  • NVIDIA TensorRT (TensorRT) imports ONNX through its own parser and builds engines for NVIDIA GPUs. ONNX Runtime can call it as a provider, which leaves the rest of the graph to CUDA or the CPU. Using TensorRT directly gives up that fallback.
  • Intel OpenVINO reads ONNX files or converts them to its own IR, as the ONNX article describes. It is also available inside ONNX Runtime as a provider.
  • LiteRT (formerly TensorFlow Lite) and ExecuTorch are the TensorFlow and PyTorch on-device runtimes, with their own file formats. Edge inference describes both, and ExecuTorch also partitions a graph among back ends, with CPU fallback.
  • OpenCV’s dnn module runs ONNX models within OpenCV, without a separate runtime (OpenCV).
  • Compilers such as XLA, TVM and TorchInductor generate code for a whole graph. The Inference Optimization article covers them. ONNX Runtime, by contrast, mainly runs library kernels on each provider and delegates compilation to providers such as TensorRT.

Versions and Compatibility

As of November 2026, the release list of microsoft/onnxruntime shows:

Release Date Notes
v1.31.0 9 Oct 2026 Arm Compute Library provider deprecated (1.32.0 notes)
v1.31.1 26 Oct 2026 Refused opsets 27 and 28 in the ONNX article’s test
v1.30.1 29 Oct 2026 Published three days after v1.31.1
v1.32.0 13 Nov 2026 ONNX 1.23.2 with opset 28; ACL provider and in-tree TensorRT RTX provider removed
plugin-ep-cuda/v0.4.0 14 Nov 2026 CUDA plugin provider matched with 1.32.0
  • Opsets. The compatibility page says ONNX Runtime supports the opsets of the latest released ONNX version and, in every version, opsets from 7 on. Its version table at e3d000e ends at 1.20 (ONNX 1.16.1, opset 21). The 1.32.0 release notes state that ONNX 1.23.2 was reintegrated, “raising supported opset ceilings to 28”. The ONNX article’s test, run with 1.31.1, refused opsets 27 and 28; that statement is dated and stays correct for 1.31.1.
  • Backward compatibility. The compatibility page states that newer versions support all models that worked with prior versions. For the ORT format, the format-version table above applies.
  • Plugins. With providers released separately, the core and each plugin have their own versions, and the release notes ask for matched pairs.
  • Licence. The repository and its documentation branch are under the MIT License, and every source file read for this article carries an MIT header.
  • Documentation lag. The documentation source at e3d000e predates 1.32.0. It still lists the Arm Compute Library provider, and its TensorRT requirements and opset tables stop at older releases. Facts in this article are dated by the source they come from.

Limitations

  • A provider takes only what it supports. Coverage gaps split a graph into several subgraphs, and every split adds copies, as the example shows. Which nodes landed where has to be checked in the session itself; for the CUDA provider, the copy transformer logs a warning with the number of copy nodes it added and points to a more verbose log level for the details.
  • Not every optimization applies to every provider. Extended fusions apply only to CPU, CUDA and ROCm nodes, and layout optimizations only to CPU nodes (graph-optimizations documentation). Nodes taken by a compiling provider are left to that provider’s compiler.
  • Saved artefacts are tied to their environment. Offline-optimized models need the same options and comparable hardware, and TensorRT engine caches must be rebuilt when the model, ONNX Runtime, TensorRT or the GPU changes.
  • Reduced precision has its own constraints. The CPU build has no float16 kernels for the mixed-precision tool, according to its documentation, and the CPU-preference pass never moves nodes with reduced-precision inputs to the CPU.
  • Session creation costs time. Optimization, partitioning, provider compilation and allocation planning all happen before the first run. The 1.32.0 notes report large improvements on very large graphs, and offline optimization and engine caches exist to avoid repeating the work.
  • Fast-moving releases. Four core releases and separate plugin releases appeared between 9 October and 14 November 2026, providers were removed, and the documentation trails the code. Pinning the ONNX Runtime version with the exporter’s and the providers’ versions keeps a deployment reproducible. The 1.32.0 notes themselves record fixes to fusions that had changed results, for example a pad fusion that changed MaxPool results on negative inputs, so a version upgrade deserves the same output comparison as a new export.

Related

  • Deploying Vision Models

    Getting a trained vision model into use, from export and the choice of runtime to packaging the model with its preprocessing contract, serving it on a server or a device, keeping pre- and post-processing identical to training, and versioning, rolling back and monitoring it.

  • Model Quantization

    Storing and computing a trained network's weights and activations in low-precision formats such as INT8, INT4 or FP8, with a scale and zero point per tensor, channel or group; the integer arithmetic behind it, post-training calibration, quantization-aware training with the straight-through estimator, and what it costs in accuracy.

  • Edge Inference

    Running a vision model on the device that captures the images, such as a phone, camera, robot or microcontroller, instead of in a data centre, covering why it is done, the limits of compute, memory, power and heat, the classes of hardware and runtimes, the roofline model for finding memory-bound layers, split and offloaded computation, and how latency and energy are measured.

  • Edge AI Accelerators

    The inside of the neural processing units and other fixed-function accelerators that run vision models on devices, covering arrays of multiply-accumulate units, systolic arrays, weight- and output-stationary dataflows, the on-chip memory hierarchy, tiling a convolution onto a fixed array, utilization lost to shape mismatch and fill and drain, operator coverage, and why a peak TOPS figure is not a throughput.

  • PyTorch

    An open-source Python library for building and training neural networks with tensors, automatic differentiation and GPU acceleration, run eagerly as ordinary Python code and optionally compiled or exported for deployment.

  • OpenCV

    The open-source C++ library of computer vision and image processing functions, with Python bindings that work on NumPy arrays; its modules, conventions, versions from 1.0 to 5.0, contrib modules and licence.

References

  1. ONNX Runtime source, onnxruntime/core/framework/graph_partitioner.cc, microsoft/onnxruntime at tag v1.32.0 (MIT License).
  2. ONNX Runtime source, onnxruntime/core/optimizer/transformer_memcpy.cc, microsoft/onnxruntime at tag v1.32.0 (MIT License).
  3. ONNX Runtime source, onnxruntime/core/framework/fallback_cpu_capability.cc, microsoft/onnxruntime at tag v1.32.0 (MIT License).
  4. ONNX Runtime source, onnxruntime/core/framework/allocation_planner.cc, microsoft/onnxruntime at tag v1.32.0 (MIT License).
  5. ONNX Runtime release v1.32.0 (GitHub release of microsoft/onnxruntime, 13 November 2026).
  6. ONNX Runtime releases (GitHub releases of microsoft/onnxruntime, v1.31.0 to v1.32.0 and plugin execution provider releases, October–November 2026).
  7. ONNX Runtime licence file (MIT License), microsoft/onnxruntime.
  8. ONNX Runtime documentation. ONNX Runtime Execution Providers, docs/execution-providers/index.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
  9. ONNX Runtime documentation. TensorRT Execution Provider, docs/execution-providers/TensorRT-ExecutionProvider.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
  10. ONNX Runtime documentation. Graph Optimizations in ONNX Runtime, docs/performance/model-optimizations/graph-optimizations.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
  11. ONNX Runtime documentation. Thread management, docs/performance/tune-performance/threading.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
  12. ONNX Runtime documentation. I/O Binding, docs/performance/tune-performance/iobinding.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
  13. ONNX Runtime documentation. Reduce memory consumption, docs/performance/tune-performance/memory.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
  14. ONNX Runtime documentation. ORT model format, docs/performance/model-optimizations/ort-format-models.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
  15. ONNX Runtime documentation. Create Float16 and Mixed Precision Models, docs/performance/model-optimizations/float16.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
  16. ONNX Runtime documentation. Quantize ONNX Models, docs/performance/model-optimizations/quantization.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
  17. ONNX Runtime documentation. ONNX Runtime compatibility, docs/reference/compatibility.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).