Vision Systems & Engineering
TensorRT
NVIDIA's inference builder and runtime for NVIDIA GPUs, which turns an ONNX file, a PyTorch program or a network built through its API into a serialized engine, with the 11.x move to strongly typed networks, mixed precision prepared by Model Optimizer's AutoCast, tactics and timing caches, optimization profiles, plugins, trtexec and Polygraphy, and a worked example of AutoCast's precision rule.
intermediate
TensorRT is NVIDIA’s library for running trained neural networks on NVIDIA GPUs. It works in two phases. A builder takes a network definition, from an ONNX file, from PyTorch through Torch-TensorRT, or written directly with its network-definition API, and produces a serialized plan, also called an engine, for the GPU it was built on. A runtime then loads that plan and executes it. The repository’s import guide puts the division plainly: the TensorRT runtime “deserializes plans, it does not parse ONNX” (import_workflows.md at v11.4). NVIDIA develops it. The GitHub repository NVIDIA/TensorRT holds the open-source parts under the Apache License 2.0: the ONNX parser, the plugins, the samples and tools such as Polygraphy. Its README calls these “a subset of the TensorRT General Availability (GA) release with some extensions and bug-fixes”. The GA library itself is distributed by NVIDIA under its own licence, which this article did not read.
This article describes TensorRT 11.4 from that repository (tag v11.4, 9 October 2026) and the Model Optimizer 0.48.0 sources that prepare models for it, as of November 2026. NVIDIA’s documentation site was not used for this article. Everything here therefore rests on the GitHub sources, ONNX Runtime’s documentation and published articles. The inference optimization article covers the transformations a builder applies in general, including TensorRT’s tactics and timing cache. This article covers the tool: how a model gets in, how precision is fixed in 11.x, what the build products are and what they depend on, and the tools around it.
Purpose
An exported model is written to be portable. An engine is written for one target. The deploying vision models article places TensorRT among the runtimes that compile a network in a build phase. The inference optimization article counts TensorRT among the compilers that apply their own graph optimizations as part of building a program. TensorRT’s builder measures candidate kernels on the GPU at hand and records the choices in the plan. The output is therefore tied to its settings and its hardware (see Versions and Compatibility), and it is a build product made from the portable model, not a replacement for it.
In vision, TensorRT is the runtime behind some of the speeds that papers report on NVIDIA hardware. Published articles quote such figures with their set-ups: PointPillars (kernels built with TensorRT), several YOLO versions and RT-DETR in DETR on T4 GPUs, and a gate detector on a Jetson Xavier in drone vision. Those figures belong to their papers and set-ups, and none is restated here.
Core Capabilities
Import paths
The import guide at v11.4 describes four paths into an engine:
- ONNX. The guide calls this the most portable path. The model is exported to ONNX (the guide prefers PyTorch’s dynamo exporter for TensorRT 11 and later, and uses
tf2onnxfor TensorFlow) and is optionally simplified and sanitized, for example withpolygraphy surgeon sanitize --fold-constants. The ONNX parser then fills a TensorRT network that the builder compiles. The guide’s compatibility note says that TensorRT supports neither every ONNX operator nor every opset, and that coverage depends on the release. The parser’s operator matrix for 11.4 (onnx-tensorrt, branch 11.4-GA) covers opsets 9 to 24; the ONNX article summarizes its restrictions. - Torch-TensorRT. Torch-TensorRT compiles inside PyTorch. The guide calls the Dynamo front end (
torch.compile(backend="tensorrt")) “the active, preferred path”, and says the older JIT and tracing-basedtorch_tensorrt.compilereceives “minimal new investment”. Ahead-of-time compilation goes throughtorch.exportandtorch_tensorrt.dynamo.compileand saves a standalone artefact. Graph breaks fall back to eager PyTorch, as with thetorch.compilethat the PyTorch article describes. - Hugging Face models. These are exported to ONNX through
optimum-onnxand then follow the ONNX path. For language-model generation the guide points to TensorRT-LLM. - Network Definition API. The network is built layer by layer from C++ or Python. The guide reserves this for architectures that neither ONNX nor PyTorch can express.
Building from ONNX has three front ends. They are the trtexec command-line tool, the Python tensorrt.Builder with OnnxParser, and polygraphy convert. The guide says all three call the same IBuilder and nvonnxparser::IParser underneath. In Python, the guide’s sequence is: create a builder, create a network with the STRONGLY_TYPED creation flag, parse the ONNX bytes, set a workspace memory-pool limit on a builder configuration, and call build_serialized_network, whose result is written to a .plan or .engine file. The C++ sample sampleOnnxMNIST follows the same steps with createParser, parseFromFile and buildSerializedNetwork. It notes that the model expects its input scaled to , so preprocessing is the application’s job.
Strong typing in TensorRT 11
TensorRT 11.0 (2 June 2026) removed weakly typed networks and the APIs related to them, replacing them with strongly typed networks. It also removed implicit quantization in favour of explicit quantization (README at v11.4). The strongly_type_autocast sample describes the difference. Under weak typing, TensorRT could substitute different precisions for tensors if that increased performance. Under strong typing, it follows the types in the ONNX model strictly. The sample says weak typing was deprecated in 10.12. In trtexec, --stronglyTyped creates the network with the kSTRONGLY_TYPED flag, so that tensor types are inferred from the network’s input types and the operators’ type specifications. The README says that builder precision flags such as --int8 and --best are not allowed with it (trtexec README at v11.4).
The practical consequence is that the precision of every tensor is decided before the build, in the model file. An FP32 ONNX model builds an FP32 engine. To run parts of it in FP16 or BF16, the model must first be converted so that the conversion is written into the graph as typed tensors and Cast nodes. That is AutoCast’s job (below). For integer formats, the scales travel in explicit QuantizeLinear and DequantizeLinear nodes. The ONNX article notes that the parser accepts INT8, INT4, FP8 and FP4 only through such quantization, and the quantization article covers calibration and placement. That article’s report that entropy calibration was TensorRT’s default calibration in 2020 is history: the README at v11.4 says that implicit quantization and its APIs were removed in 11.x, replaced by explicit quantization.
The repository’s own documents have not all caught up. At v11.4, the import guide’s trtexec example still adds --fp16 (with the comment “or –bf16, –int8, –fp8 (platform-dependent)”), and its Polygraphy example passes --fp16 too. The trtexec README’s examples for NVIDIA’s Deep Learning Accelerator (DLA) also use --fp16 and --int8. These are reported as printed. Which of these flags a strongly typed 11.4 build accepts would have to be settled from documentation that this article did not read.
AutoCast: mixed precision before the build
AutoCast, part of NVIDIA’s Model Optimizer, converts an FP32 ONNX model to mixed FP32–FP16 or FP32–BF16 precision. It keeps selected nodes in FP32 and inserts the casts around them (AutoCast guide at 0.48.0). The strongly_type_autocast sample recommends it as the step before a strongly typed build. The sample converts an MNIST model with convert_to_mixed_precision, checks the converted model with ONNX Runtime against FP32 reference outputs, then builds with STRONGLY_TYPED and checks the engine against the same references.
The source at 0.48.0 shows how the nodes are chosen:
- Reference data.
ReferenceRunnerruns the FP32 model in ONNX Runtime with every tensor marked as an output, and records each tensor’s values. The graph inputs are recorded too. If calibration data are given, each batch is run and the statistics are combined: the absolute maximum (absmax) of a tensor is the maximum over batches. Without calibration data, random inputs are generated, and the guide advises providing representative data. - Classification.
NodeClassifierapplies rules in turn, and a node stays in high precision if any rule fires. By default there are two rules. The I/O-range rule fires when the absmax of any input or output of the node in the reference data exceedsdata_max; the comparison is strict, Constant nodes are skipped, and tensors absent from the reference data, such as initializers, pass. The initializer-range rule fires when a float32 initializer of the node has a value whose magnitude exceedsinit_max. The guide gives the reason for the first threshold: “The unit in last place (ULP) for 512 is 0.5, for 1024 it is 1.0”. The reason for the second is that values aboveinit_maxoverflow. Further rules are disabled by default: node-name and operator-type patterns, include lists that force nodes into low precision, and a depth-of-reduction limit for large matrix products and convolutions. - Conversion.
PrecisionConvertercasts the inputs of low-precision nodes down and their outputs up where an FP32 node consumes them. It converts initializers to their consumer’s precision, clamping values beyond the format’s range with a warning. Unlesskeep_io_typesis set, it also turns the graph’s FP32 inputs and outputs into the low-precision type. The guide adds that the opset is raised to at least 13 for FP16 and 22 for BF16, and that TensorRT “might not be able to support all BF16 converted models”.
The defaults have two values in the source. The entry point convert.py sets DEFAULT_DATA_MAX = 512 and DEFAULT_INIT_MAX to float16’s maximum, 65,504, and passes them to the classifier; the guide gives the same defaults. NodeClassifier.__init__ itself defaults data_max to 1000.0, which applies only to code that constructs the classifier directly (both at 0.48.0). Users of convert_to_mixed_precision or the command line get 512.
Checking reduced precision: Polygraphy
Polygraphy, in the same repository, runs a model in several back ends and compares the results. It also converts models to engines, inspects them, edits ONNX graphs (extracting subgraphs, folding constants) and isolates faulty tactics (Polygraphy README at v11.4). The import guide’s verification step compares an engine with ONNX Runtime directly, as polygraphy run model.onnx --trt --onnxrt with absolute and relative tolerances. Its how-to on reduced precision says that models trained in FP32 can often run in TF32 and FP16 with little or no extra effort, but that INT8 generally needs extra steps. If an FP16 engine is not accurate enough, it suggests running the same model in FP16 under ONNX Runtime to see whether the limit lies in the format rather than in TensorRT. The next step is to isolate the problem layers and override their precision to FP32. If that is not enough, the model itself should keep its intermediate activations in a representable range, for example by normalizing inputs to and using batch normalization during training. The example below shows what unnormalized input does to AutoCast’s decisions.
The repository also ships ONNX GraphSurgeon, which the import guide lists for “programmatic ONNX graph edits”, next to the third-party onnxsim for folding constants.
Tactics and the timing cache
Many layers have several implementations, which TensorRT calls tactics. The builder profiles them and writes the fastest into a timing cache, which a later build can reuse instead of profiling again. The inference optimization article describes this mechanism. The sampleEditableTimingCache sample adds a way to override the choice. It builds a three-node network (MatMul, Softmax, MatMul) with the editable timing cache enabled, so that the profiling table of each layer appears in the log. It then writes a different tactic for the first MatMul into the cache and builds again. The second build skips profiling and uses the recorded tactics, so only the first MatMul’s kernel changes. The sample’s title calls this a deterministic build; in our reading, once a layer’s tactic is recorded in the cache, its choice no longer depends on timing noise. For trtexec’s build-route tuner, the README states that engine builds are not bit-deterministic.
Dynamic shapes and optimization profiles
An engine is built for a range of input shapes. On the command line, the range is three shapes given with --minShapes, --optShapes and --maxShapes, and --shapes chooses the shape of a benchmark run (trtexec README). In the network API, an optimization profile is created with builder.create_optimization_profile(), gets set_shape(name, min, opt, max) for each dynamic input, and is added to the builder configuration (import guide, Path 4). ONNX Runtime’s TensorRT provider caches the profile with each engine. If later input shapes fall outside it, the provider widens the profile and rebuilds the engine (TensorRT execution provider documentation). The guide notes that some data-dependent shapes, such as a where whose output shape is dynamic, need shape layers and profile-shaped tensors.
Plugins
Operators that TensorRT does not support can be supplied as plugins. In 11.0, the IPluginV2 interfaces were removed in favour of IPluginV3 (README). 11.2 added a sample that shows the migration and an FFT plugin to support ONNX’s DFT operator (CHANGELOG). The import guide’s steps are: check IPluginRegistry for an existing plugin, implement IPluginV3, register it with REGISTER_TENSORRT_PLUGIN in C++ or through the plugin registry in Python, and export the ONNX node under a custom domain with a matching name. Plugins shipped in libnvinfer_plugin are pre-registered. trtexec loads one’s own plugin libraries with --dynamicPlugins, and --setPluginsToSerialize stores a library inside the engine so that it need not be named again at load time (trtexec README). 11.0 also removed a dozen deprecated plugins, among them nms, proposal, cropAndResize and leakyRelu (CHANGELOG).
trtexec
trtexec has two purposes, according to its README: benchmarking a network on random or user-supplied inputs, and generating serialized engines. It builds from --onnx, saves with --saveEngine, and runs a saved plan with --loadEngine. Inputs are random unless --loadInputs names binary files, which the README suggests writing with NumPy’s tofile. Its other features include:
- exporting timing traces as JSON (
--exportTimes); - running several concurrent streams (
--streams); - since 11.1, a global performance tuner (CHANGELOG). It sweeps “build routes”, combinations of internal builder knobs, under a search strategy (
fast,fullormixed). It can reject routes whose outputs drift from reference outputs beyond an accuracy threshold, and it records its results in a resumable cache.
The README’s caveats for the tuner include that a pinned knob can be overridden during compilation, that results are specific to the model, the GPU model and the TensorRT version, and that the saved engine, not the route string, should be shipped.
Minimal Example
TensorRT needs an NVIDIA GPU, and the machine this article was written on has none. The example therefore reproduces the step before a strongly typed FP16 build: AutoCast’s node classification, ported to NumPy from nodeclassifier.py at Model Optimizer 0.48.0, applied to a small network with float16 arithmetic emulated in NumPy. TensorRT does not run here. The outcomes test our port of AutoCast’s default rules and our emulation of FP16 arithmetic. They say nothing about what TensorRT’s kernels compute.
Data. The images are synthetic 8-bit grayscale images, one shape each: a disk, a square or a triangle at level 160–255 on a background at level 60–100, with Gaussian noise (), rounded and clipped. They are drawn with scikit-image 0.25.2’s skimage.draw. Eight calibration images (seed 1) provide AutoCast’s reference data, and 32 test images (seed 2) are used to evaluate. As printed below, the calibration set contains five disks, no square and three triangles, and the test set ten, ten and twelve. The input is fed unnormalized, as values 0–255, without the input normalization that the Polygraphy how-to lists as a remedy.
Network. The network has 11 nodes with ONNX operator names and seeded, untrained He-normal weights (seed 0): Conv (, ), Relu, MaxPool (), Conv (), Relu, ReduceSum over the map, Mul by , Flatten, Gemm (), Relu, Gemm (). No training is needed: the question is how the precision of each node changes the computed function, not what it recognizes. The ReduceSum followed by a Mul is a mean written as a sum and a scale. It is the constructed overflow case, a sum of 1,024 non-negative activations whose magnitudes the raw input inflates.
Variants. Five variants are run:
- V0 is the FP32 reference.
- V1 puts every node in FP16.
- V2 is AutoCast’s mixed precision: nodes the rules exclude stay in FP32, the rest are FP16.
The reference data for V2 are the per-tensor absmax over the eight calibration images, combined as referencerunner.py combines batches. As with keep_io_types=False, the graph input becomes float16 in V1 and V2, which is exact for integers 0–255. Each node casts its inputs to its own format, which stands in for AutoCast’s Cast nodes; a float16 cast of a value beyond the format’s range gives infinity.
Arithmetic. Every reduction (the convolutions as a sum over input channels and taps, the ReduceSum, the Gemm) accumulates sequentially in a fixed order. How the FP16 kernels of a TensorRT engine accumulate is not stated in any source read. That is a modelling choice the sources leave open, so FP16 nodes run under two choices, and both are printed:
- fp16 accumulation (V1a, V2a): every multiply-add is rounded once to float16, as a fused multiply-add.
- fp32 accumulation (V1b, V2b): float16 inputs and weights are multiplied and summed in float32, and the result is rounded to float16 once.
The mixed precision training article covers the formats themselves, their rounding and why sums are accumulated in FP32.
Checks and hypotheses. The script prints the following, each with the value it is decided on:
- D1, a data check on both image sets;
- C1, the port’s excluded set against an independently written set comprehension;
- C2, the FP32 loops against the same loops in float64, with a bar of ;
- C3, eight float16 rounding facts, including the spacing of 0.5 at 512 and 1 at 1024 that AutoCast’s guide states;
- P1, the design premise that some node exceeds 512 and that the ReduceSum exceeds float16’s range on every calibration image.
The hypotheses were written before any code ran:
- H1. The nodes excluded on the calibration images are the ones the same rule would exclude on the test images. This is a check of the calibration data that can fail.
- H2. All-FP16 overflows at the ReduceSum on every test image, and the mixed graph has no non-finite value anywhere.
- H3. The mixed graph’s maximum relative logit error against FP32 is below all-FP16’s under each accumulation.
- H3b. The same holds for the error at the last tensor before the overflow, Relu2’s output.
The relative error of an image is against the FP32 tensor , taken as infinite if any value is non-finite, and the script prints the maximum over the 32 test images.
A planning script, run once before the example’s first run on the calibration images only, printed the calibration absmax of every node and hence the excluded set. The classification below is therefore an illustration rather than a prediction. The C2 bar of was set after that script had printed C2’s reachable value on the calibration images, so it is a bar set after a planning script. The example’s script uses NumPy 2.3.5 and scikit-image 0.25.2, runs in under a second, and printed identical output on two runs. The code is the script as first run; nothing in it was changed afterwards.
import os
os.environ["OMP_NUM_THREADS"] = "1"; os.environ["OPENBLAS_NUM_THREADS"] = "1"
import numpy as np
from skimage.draw import disk, rectangle, polygon
# ---- data: rendered 64x64 uint8 shape images, fed unnormalized (0..255) -------------
def render(rng):
b, v = rng.integers(60, 101), rng.integers(160, 256)
k = rng.integers(0, 3) # 0 disk, 1 square, 2 triangle
cy, cx = rng.uniform(16, 48, size=2); r = rng.uniform(8, 14)
img = np.full((64, 64), float(b)); mask = np.zeros((64, 64), bool)
if k == 0:
rr, cc = disk((cy, cx), r, shape=(64, 64))
elif k == 1:
rr, cc = rectangle((cy - r, cx - r), (cy + r, cx + r), shape=(64, 64))
rr, cc = np.round(rr).astype(int), np.round(cc).astype(int)
else:
a = np.deg2rad([90, 210, 330])
rr, cc = polygon(cy - r * np.sin(a), cx + r * np.cos(a), shape=(64, 64))
mask[rr, cc] = True; img[mask] = v
img = np.clip(np.round(img + rng.normal(0, 3, img.shape)), 0, 255).astype(np.uint8)
return img, mask, k
def image_set(seed, n):
rng = np.random.default_rng(seed)
return [render(rng) for _ in range(n)]
calib, test = image_set(1, 8), image_set(2, 32)
# D1: data check
def d1(name, s):
ims = np.stack([i for i, _, _ in s])
ok_type = ims.dtype == np.uint8
area = min(int(m.sum()) for _, m, _ in s)
inside = min(float(i[m].mean()) for i, m, _ in s)
outside = max(float(i[~m].mean()) for i, m, _ in s)
counts = np.bincount([k for _, _, k in s], minlength=3).tolist()
ok = ok_type and ims.min() >= 0 and ims.max() <= 255 and area > 0 and inside >= 150 and outside <= 110
print(f"D1 {name}: uint8={ok_type} min={ims.min()} max={ims.max()} min mask area={area} "
f"min mean inside={inside:.2f} max mean outside={outside:.2f} classes={counts} -> {'held' if ok else 'failed'}")
d1("calibration", calib); d1("test", test)
# ---- the graph: 11 nodes, seeded untrained weights -----------------------------------
rng = np.random.default_rng(0)
INIT = {"W1": rng.normal(0, np.sqrt(2 / 9), (8, 1, 3, 3)).astype(np.float32),
"W2": rng.normal(0, np.sqrt(2 / 72), (16, 8, 3, 3)).astype(np.float32),
"s": np.array(1 / 1024, np.float32),
"G1": rng.normal(0, np.sqrt(2 / 16), (32, 16)).astype(np.float32),
"b1": np.zeros(32, np.float32),
"G2": rng.normal(0, np.sqrt(1 / 32), (4, 32)).astype(np.float32),
"b2": np.zeros(4, np.float32)}
NODES = [ # (name, op_type, inputs, outputs)
("conv1", "Conv", ["x", "W1"], ["t1"]), ("relu1", "Relu", ["t1"], ["t2"]),
("pool1", "MaxPool", ["t2"], ["t3"]), ("conv2", "Conv", ["t3", "W2"], ["t4"]),
("relu2", "Relu", ["t4"], ["t5"]), ("rsum", "ReduceSum", ["t5"], ["t6"]),
("scale", "Mul", ["t6", "s"], ["t7"]), ("flat", "Flatten", ["t7"], ["t8"]),
("gemm1", "Gemm", ["t8", "G1", "b1"], ["t9"]), ("relu3", "Relu", ["t9"], ["t10"]),
("gemm2", "Gemm", ["t10", "G2", "b2"], ["t11"])]
NODE_TO_INIT = {n: [t for t in ins if t in INIT] for n, _, ins, _ in NODES}
# ---- arithmetic: sequential accumulation in a fixed order ----------------------------
# fmt "f32": float32 products and sums; "f16a": float16 inputs, one float16 rounding per
# multiply-add (fused); "f16b": float16 inputs, float32 accumulation, output to float16;
# "f64": float64 everywhere (emulator check only).
def accumulate(terms_w, terms_x, fmt, out_shape):
"""sum_k w[k] * x[k] in order k = 0..K-1; w[k] broadcasts against x[k]."""
if fmt == "f16a":
acc = np.zeros(out_shape, np.float16)
for w, x in zip(terms_w, terms_x):
with np.errstate(over="ignore", invalid="ignore"):
acc = (acc.astype(np.float64) + w.astype(np.float64) * x.astype(np.float64)).astype(np.float16)
return acc
dt = np.float64 if fmt == "f64" else np.float32
acc = np.zeros(out_shape, dt)
with np.errstate(over="ignore", invalid="ignore"):
for w, x in zip(terms_w, terms_x):
acc = acc + w.astype(dt) * x.astype(dt)
return acc.astype(np.float16) if fmt == "f16b" else acc
def store(a, fmt): # the node's tensor format (inputs are cast to it)
with np.errstate(over="ignore"):
return a.astype({"f32": np.float32, "f64": np.float64}.get(fmt, np.float16))
def conv3x3(x, W, fmt): # x: (Cin, H, W), pad 1, stride 1
x = np.pad(store(x, fmt), ((0, 0), (1, 1), (1, 1)))
W = store(W, fmt); cout, cin = W.shape[:2]; H, Wd = x.shape[1] - 2, x.shape[2] - 2
tw, tx = [], []
for c in range(cin):
for i in range(3):
for j in range(3):
tw.append(W[:, c, i, j][:, None, None]); tx.append(x[c, i:i + H, j:j + Wd][None])
return accumulate(tw, tx, fmt, (cout, H, Wd))
def run_node(op, ins, fmt):
if op == "Conv":
return conv3x3(ins[0], ins[1], fmt)
if op == "Relu":
return np.maximum(store(ins[0], fmt), 0)
if op == "MaxPool":
x = store(ins[0], fmt); c, h, w = x.shape
return x.reshape(c, h // 2, 2, w // 2, 2).max(axis=(2, 4))
if op == "ReduceSum":
x = store(ins[0], fmt).reshape(ins[0].shape[0], -1)
one = np.ones(x.shape[0], x.dtype)
return accumulate([one] * x.shape[1], list(x.T), fmt, (x.shape[0],)).reshape(-1, 1, 1)
if op == "Mul":
with np.errstate(over="ignore", invalid="ignore"):
return store(store(ins[0], fmt).astype(np.float64) * store(ins[1], fmt).astype(np.float64), fmt)
if op == "Flatten":
return store(ins[0], fmt).reshape(-1)
if op == "Gemm":
x, G, b = store(ins[0], fmt), store(ins[1], fmt), store(ins[2], fmt)
y = accumulate(list(G.T), list(x), fmt, (G.shape[0],))
with np.errstate(over="ignore", invalid="ignore"):
if fmt == "f16a":
return (y.astype(np.float64) + b.astype(np.float64)).astype(np.float16)
dt = np.float64 if fmt == "f64" else np.float32
y = y.astype(dt) + b.astype(dt)
return y.astype(np.float16) if fmt == "f16b" else y
raise ValueError(op)
def run_graph(img, fmt_of_node, io_fp16):
x = img.astype(np.float32)[None] # 1 channel, unnormalized 0..255
T = {"x": x.astype(np.float16) if io_fp16 else x} # keep_io_types=False: input becomes FP16
T.update(INIT)
for name, op, ins, outs in NODES:
T[outs[0]] = run_node(op, [T[t] for t in ins], fmt_of_node[name])
return T
# ---- AutoCast's node classification (NumPy port) -------------------------------------
# Ported from modelopt/onnx/autocast/nodeclassifier.py, NVIDIA/Model-Optimizer at tag 0.48.0.
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# Modified (ported): ONNX protos replaced by a minimal node list; logging removed; only the
# default rules (initializer range, I/O range) kept. Licence:
# https://github.com/NVIDIA/Model-Optimizer/blob/0.48.0/LICENSE
class NodeRuleBase:
def check(self, node):
return bool(self._check_inner(node))
class InitializerRangeRule(NodeRuleBase):
def __init__(self, init_max, node_to_init_map):
self.init_max, self.node_to_init_map = init_max, node_to_init_map
def _check_inner(self, node):
for name in self.node_to_init_map[node[0]]:
a = INIT[name]
if a.dtype == np.float32 and np.any(np.abs(a) > self.init_max):
return True
return False
class IORangeRule(NodeRuleBase):
def __init__(self, data_max, reference_data):
self.data_max, self.reference_data = data_max, reference_data
def _check_inner(self, node):
def is_io_out_of_range(tensor_name):
if tensor_name not in self.reference_data: # initializers are not reference data
return False
return self.reference_data[tensor_name] > self.data_max # stored as absmax
name, op_type, inputs, outputs = node
if op_type == "Constant":
return False
if self.reference_data:
for t in inputs:
if is_io_out_of_range(t):
return True
for t in outputs:
if is_io_out_of_range(t):
return True
return False
def classify(reference_data, data_max=512.0, init_max=float(np.finfo(np.float16).max)):
rules = [InitializerRangeRule(init_max, NODE_TO_INIT), IORangeRule(data_max, reference_data)]
low, high = [], []
for node in NODES:
(high if any(r.check(node) for r in rules) else low).append(node[0])
return low, high
# ---- end of port ----------------------------------------------------------------------
ALL32 = {n[0]: "f32" for n in NODES}
def absmax_stats(images): # FP32 reference run; absmax per tensor, maximum over images
st = {}
for img, _, _ in images:
T = run_graph(img, ALL32, io_fp16=False)
for t in ["x"] + [n[3][0] for n in NODES]:
st[t] = max(st.get(t, 0.0), float(np.max(np.abs(T[t]))))
return st
ref_cal, ref_test = absmax_stats(calib), absmax_stats(test)
low, high = classify(ref_cal)
_, high_test = classify(ref_test)
# C1: the port against an independent set comprehension
io_max = lambda st, n: max(st[t] for t in n[2] + n[3] if t in st)
indep = {n[0] for n in NODES if io_max(ref_cal, n) > 512} | \
{n[0] for n in NODES if any(np.abs(INIT[t]).max() > 65504 for t in NODE_TO_INIT[n[0]])}
print(f"C1 port = independent set: {set(high) == indep} -> {'held' if set(high) == indep else 'failed'}")
# C3: float16 rounding (nearest even, overflow) and the spacing AutoCast's guide states
f = lambda v: float(np.float16(v)) if np.isfinite(np.float16(v)) else np.inf
with np.errstate(over="ignore"):
c3 = [("1+2^-11", f(1 + 2**-11), 1.0), ("1+3*2^-11", f(1 + 3 * 2**-11), 1 + 2**-9),
("512.25", f(512.25), 512.0), ("512.75", f(512.75), 513.0),
("65519", f(65519), 65504.0), ("65520", f(65520), np.inf),
("spacing(512)", float(np.spacing(np.float16(512))), 0.5),
("spacing(1024)", float(np.spacing(np.float16(1024))), 1.0)]
print("C3 " + " ".join(f"{k}->{v:g}" for k, v, _ in c3) +
f" -> {'held' if all(v == e for _, v, e in c3) else 'failed'}")
print("\nM1 node I/O absmax (calibration max of 8 | test max of 32), data_max = 512")
for n in NODES:
a, b = io_max(ref_cal, n), io_max(ref_test, n)
flag = " near 512" if min(abs(a - 512), abs(b - 512)) < 1.0 else ""
print(f" {n[0]:<6} {n[1]:<9} {a:12.2f} {b:12.2f} {'FP32' if n[0] in high else 'FP16'}{flag}")
print(f"P1 some node > 512: {max(io_max(ref_cal, n) for n in NODES) > 512}; "
f"ReduceSum output absmax min over calibration images: "
f"{min(float(np.abs(run_graph(i, ALL32, False)['t6']).max()) for i, _, _ in calib):.2f} (> 65520 needed)")
print(f"M2 excluded (calibration): {high}")
print(f"M2 excluded (test-image absmax): {high_test}")
print(f"H1 calibration set == test set: {high == high_test} -> {'held' if high == high_test else 'failed'}")
print(f"M3 nodes in FP16: {len(low)}/{len(NODES)} = {len(low) / len(NODES):.3f}")
# ---- variants on the test images -------------------------------------------------------
VARIANTS = {"V0 FP32": (ALL32, False),
"V1a all FP16, fp16 acc": ({n[0]: "f16a" for n in NODES}, True),
"V1b all FP16, fp32 acc": ({n[0]: "f16b" for n in NODES}, True),
"V2a AutoCast, fp16 acc": ({n[0]: ("f32" if n[0] in high else "f16a") for n in NODES}, True),
"V2b AutoCast, fp32 acc": ({n[0]: ("f32" if n[0] in high else "f16b") for n in NODES}, True)}
def rel_err(y, r):
y = y.astype(np.float64); r = r.astype(np.float64)
return np.inf if not np.all(np.isfinite(y)) else float(np.max(np.abs(y - r)) / np.max(np.abs(r)))
refs = [run_graph(i, ALL32, False) for i, _, _ in test]
r64 = [run_graph(i, {n[0]: "f64" for n in NODES}, False) for i, _, _ in test]
c2 = max(rel_err(a["t11"], b["t11"]) for a, b in zip(refs, r64))
print(f"\nC2 FP32 emulator against float64, max relative logit difference: {c2:.3e} (bar 1e-05) "
f"-> {'held' if c2 <= 1e-5 else 'failed'}")
res = {}
print(f"\n{'variant':<24} {'rsum inf':>8} {'any nonfin':>10} {'logit nonfin':>12} {'M4 logit err':>13} {'M6 relu2 err':>13}")
for name, (fmts, io16) in VARIANTS.items():
Ts = [run_graph(i, fmts, io16) for i, _, _ in test]
n_inf = sum(bool(np.isinf(T["t6"]).any()) for T in Ts)
n_any = sum(any(not np.all(np.isfinite(T[n[3][0]])) for n in NODES) for T in Ts)
n_log = sum(not np.all(np.isfinite(T["t11"])) for T in Ts)
m4 = max(rel_err(T["t11"], R["t11"]) for T, R in zip(Ts, refs))
m6 = max(rel_err(T["t5"], R["t5"]) for T, R in zip(Ts, refs))
res[name] = (n_inf, n_any, m4, m6)
print(f"{name:<24} {n_inf:>8} {n_any:>10} {n_log:>12} {m4:>13.3e} {m6:>13.3e}")
v = list(res.values()); N = len(test)
h2 = v[1][0] == N and v[2][0] == N and v[3][1] == 0 and v[4][1] == 0
print(f"\nH2 rsum inf on {v[1][0]}/{N} (V1a), {v[2][0]}/{N} (V1b); non-finite images {v[3][1]} (V2a), "
f"{v[4][1]} (V2b) -> {'held' if h2 else 'failed'}")
for acc, a, b in (("fp16", 1, 3), ("fp32", 2, 4)):
print(f"H3 {acc} acc: M4 mixed {v[b][2]:.3e} < all-FP16 {v[a][2]:.3e} -> {'held' if v[b][2] < v[a][2] else 'failed'}")
print(f"H3b {acc} acc: M6 mixed {v[b][3]:.3e} < all-FP16 {v[a][3]:.3e} -> {'held' if v[b][3] < v[a][3] else 'failed'}")
Output:
D1 calibration: uint8=True min=55 max=255 min mask area=84 min mean inside=170.00 max mean outside=97.97 classes=[5, 0, 3] -> held
D1 test: uint8=True min=47 max=255 min mask area=81 min mean inside=161.95 max mean outside=100.02 classes=[10, 10, 12] -> held
C1 port = independent set: True -> held
C3 1+2^-11->1 1+3*2^-11->1.00195 512.25->512 512.75->513 65519->65504 65520->inf spacing(512)->0.5 spacing(1024)->1 -> held
M1 node I/O absmax (calibration max of 8 | test max of 32), data_max = 512
conv1 Conv 863.13 869.15 FP32
relu1 Relu 863.13 869.15 FP32
pool1 MaxPool 817.74 815.43 FP32
conv2 Conv 1160.53 1169.57 FP32
relu2 Relu 1160.53 1169.57 FP32
rsum ReduceSum 305450.56 328760.19 FP32
scale Mul 305450.56 328760.19 FP32
flat Flatten 298.29 321.05 FP16
gemm1 Gemm 298.29 321.05 FP16
relu3 Relu 279.53 301.14 FP16
gemm2 Gemm 279.53 301.14 FP16
P1 some node > 512: True; ReduceSum output absmax min over calibration images: 214780.31 (> 65520 needed)
M2 excluded (calibration): ['conv1', 'relu1', 'pool1', 'conv2', 'relu2', 'rsum', 'scale']
M2 excluded (test-image absmax): ['conv1', 'relu1', 'pool1', 'conv2', 'relu2', 'rsum', 'scale']
H1 calibration set == test set: True -> held
M3 nodes in FP16: 4/11 = 0.364
C2 FP32 emulator against float64, max relative logit difference: 9.822e-07 (bar 1e-05) -> held
variant rsum inf any nonfin logit nonfin M4 logit err M6 relu2 err
V0 FP32 0 0 0 0.000e+00 0.000e+00
V1a all FP16, fp16 acc 32 32 32 inf 2.727e-03
V1b all FP16, fp32 acc 32 32 32 inf 7.724e-04
V2a AutoCast, fp16 acc 0 0 0 1.428e-03 0.000e+00
V2b AutoCast, fp32 acc 0 0 0 8.170e-04 0.000e+00
H2 rsum inf on 32/32 (V1a), 32/32 (V1b); non-finite images 0 (V2a), 0 (V2b) -> held
H3 fp16 acc: M4 mixed 1.428e-03 < all-FP16 inf -> held
H3b fp16 acc: M6 mixed 0.000e+00 < all-FP16 2.727e-03 -> held
H3 fp32 acc: M4 mixed 8.170e-04 < all-FP16 inf -> held
H3b fp32 acc: M6 mixed 0.000e+00 < all-FP16 7.724e-04 -> held
Checks. The data, the port, the FP32 emulator and the float16 facts all behave as intended: D1, C1, C2 ( against the bar of ) and C3 held, and so did the design premise P1. The rounding lines confirm the guide’s reason for its threshold. At 512 the spacing of float16 numbers is 0.5, so 512.25 rounds to 512 and 512.75 to 513. 65,520 becomes infinity, while 65,519 still rounds to 65,504.
Classification. Seven of the eleven nodes stay in FP32. They are the two convolutions with their Relu and MaxPool nodes, the ReduceSum and the Mul. Every one of them sees reference values above 512, from about 818 at the pooling node to about 305,000 at the sum. The four nodes after the Mul, Flatten and the two-layer head, see reference values below 300 and run in FP16. In our reading, the pattern follows from the input: with 0–255 at the input and weights scaled for unit-sized input, the early activations are in the hundreds, and the I/O-range rule keeps that whole front end in FP32. H1 held: the test images’ absmax gives the same excluded set. The closest margins in the printed table are the pooling node’s test absmax of 815.43 and the Flatten and first Gemm nodes’ 321.05, both far from 512. The rule is decided per node by a single maximum over the reference data. In our reading, that makes it sensitive to how representative the calibration images are, which is why the guide asks for real data, even though here a calibration set without a single square gave the same decision as the test set.
Overflow. H2 held. With every node in FP16, the ReduceSum output is infinite on all 32 test images under both accumulations, and the logits are non-finite on all 32. In the mixed graph, no tensor on any test image is non-finite. The difference is the classification. The mixed graph computes the sum and the Mul in FP32 and hands the FP16 head a mean, whose values lie inside float16’s range.
Error. H3 held under both accumulations: the mixed graph’s maximum relative logit error is with fp16 accumulation and with fp32 accumulation, against infinity for all-FP16. Because the all-FP16 logits are non-finite, H3 follows from H2 and adds nothing beyond it. H3b held as well, but by construction: in the mixed graph every node up to Relu2 is FP32, as the planning script had already printed, and the float16 input cast is exact, so the mixed Relu2 output equals the FP32 one and its error is exactly zero. What H3b’s printed values do show is the error all-FP16 had already accumulated before the overflow: with fp16 accumulation and with fp32 accumulation. In both comparisons the fp32-accumulation row has the smaller error. In our reading, the reason is that the fp16 rows round every partial sum to float16 (72 times per output in all-FP16’s second convolution, 16 and 32 times in the two Gemms), while the fp32 rows round to float16 once per node output. Which of these rows is closer to a given TensorRT FP16 kernel is not something the sources read here state, so the example claims neither.
The example shows the arrangement a strongly typed build receives. AutoCast fixes every node’s type from reference magnitudes before the build. With unnormalized input, most of this small network stays in FP32, and only the head runs in FP16. In our reading, making more of it FP16 under the default rules would need what the Polygraphy how-to describes: inputs normalized at training time, or a model whose activations stay small.
Ecosystem and Integrations
- ONNX Runtime’s TensorRT execution provider. ONNX Runtime can hand supported subgraphs to TensorRT. The import guide explains that the claim “TensorRT executes ONNX directly” refers to this provider, which builds a TensorRT engine internally on the first call. The provider has its own options:
trt_fp16_enable,trt_min_subgraph_size(subgraphs below it fall back to other providers; the default is 1), an engine cache, a timing cache, and an “EPContext” model, which wraps a serialized engine in an ONNX file so that a session can load it without building (TensorRT execution provider documentation). Those options belong to ONNX Runtime and are separate from TensorRT’s own API. - Model Optimizer. NVIDIA’s Model Optimizer contains AutoCast, and TensorRT’s CHANGELOG for 10.15 records parser support for “more specialized quantization ops for models quantized through TensorRT ModelOptimizer”. The quantization article covers the methods.
- Torch-TensorRT and TensorRT-LLM. Torch-TensorRT is the PyTorch front end. TensorRT-LLM is what the import guide recommends for language-model generation.
- Serving. NVIDIA’s Triton Inference Server serves TensorRT engines among other back ends, as the deploying vision models article describes.
- Profiling and inspection. The README says the TREX tool was removed in 11.0, replaced by Nsight Deep Learning Designer. The import guide lists
nsysandncufrom the CUDA toolkit for profiling. - Installation. The README offers a prebuilt Python package (
pip install tensorrt). For 11.x, the import guide says to use the-cu13packages and not to mix in-cu12wheels.
Alternatives
- ONNX Runtime runs the same ONNX files on many back ends, including CPUs, and partitions graphs among execution providers. TensorRT is one of them.
- Compilers that generate code, such as XLA, Apache TVM and PyTorch’s TorchInductor through
torch.compile, are covered in the inference optimization and PyTorch articles. - Other vendors’ runtimes, such as Intel’s OpenVINO, which the deploying vision models article describes, and LiteRT and the other on-device runtimes in the edge inference article, play the same role on other hardware.
In our reading, the choice is about the target. TensorRT runs only on NVIDIA GPUs and its engines are tied to one of them, while the alternatives trade some of that specialization for portability.
Versions and Compatibility
- Releases. The 11.x series on GitHub runs from v11.0 (2 June 2026) through v11.1 (24 June), v11.2 (4 August) and v11.3 (22 September) to v11.4 (9 October 2026). Building the v11.4 open-source components needs the GA build TensorRT 11.4.0.106. The README lists CUDA 13.4.0 and 12.9.0 as recommended versions, cuDNN 8.9 as optional, and Python 3.10 to 3.14. Its build instructions cover Linux on x86-64 and aarch64, Windows, cross-compilation for Jetson, and NVIDIA’s DriveOS with Linux and QNX.
- What 11.0 removed (README at v11.4): weakly typed networks, implicit quantization,
IPluginV2, the TREX tool, and Python bindings for Python 3.9 and older. - Engines are not portable. The import guide’s troubleshooting section says that plans are not portable across compute capabilities, and that the remedy is to rebuild on the deployment GPU or to “target multiple SMs” at build time. ONNX Runtime’s provider documentation says that each engine is created for specific settings (model, precision, workspace, profiles) and specific GPUs and is not portable. It lists changes of the model, of ONNX Runtime’s version, of TensorRT’s version and of the hardware as reasons to rebuild cached engines. The README for trtexec’s tuner adds that a tuned build route depends on the TensorRT version and should be re-tuned after an upgrade. NVIDIA’s documentation, which this article did not read, may give wider compatibility guarantees; none is stated here.
- Licensing. The repository is under the Apache License 2.0, and the README says its components are a subset of the GA release. The sample READMEs end by referring to the TensorRT Software License Agreement on NVIDIA’s documentation site, which was not read; nothing is copied from them here. The example’s port of AutoCast is from Model Optimizer’s Apache-2.0 source and carries its notice.
Limitations
- NVIDIA GPUs only. The import guide’s first prerequisite is “a supported NVIDIA GPU”. Nothing in this article was run with TensorRT.
- Coverage depends on the release. Not every ONNX operator or opset is supported. The import guide’s remedies are updating the exporter or opset, rewriting the subgraph, or writing a plugin.
- Precision is the user’s decision. Under strong typing, an FP32 model stays FP32 unless it is converted. AutoCast’s rule depends on reference data, and the example shows how unnormalized input keeps much of a network in FP32. The guide’s random-input default gives reference magnitudes that, in our reading, need not resemble real images.
- Build cost and first-run cost. ONNX Runtime’s provider documentation says that TensorRT may take a long time to optimize and build an engine, and offers engine and timing caches to save that time. The import guide attributes slow first inference to one-time kernel compilation and plan deserialization, and advises at least three warm-up iterations before timing; the latency and throughput article covers measurement.
- Documentation lag and gaps. At v11.4 the repository’s examples still show builder precision flags such as
--fp16. The sources read do not state the arithmetic of FP16 kernels, the layer-fusion catalogue or the compatibility guarantees; NVIDIA’s documentation site, which may state them, was not used.
Related
- Deploying Vision Models
Getting a trained vision model into use, from export and the choice of runtime to packaging the model with its preprocessing contract, serving it on a server or a device, keeping pre- and post-processing identical to training, and versioning, rolling back and monitoring it.
- ONNX Runtime
Microsoft's open-source inference engine for ONNX models, which optimizes a model's graph, partitions it among execution providers for the hardware present, inserts the copies between devices and runs it, with a worked example of partitioning and copy insertion.
- Model Quantization
Storing and computing a trained network's weights and activations in low-precision formats such as INT8, INT4 or FP8, with a scale and zero point per tensor, channel or group; the integer arithmetic behind it, post-training calibration, quantization-aware training with the straight-through estimator, and what it costs in accuracy.
- Mixed-Precision Training
Training a network with most of its arithmetic and storage in 16-bit (FP16 or BF16) or 8-bit floating point while keeping the numerically sensitive parts in FP32, using an FP32 master copy of the weights, loss scaling, FP32 accumulation and per-tensor scales for FP8.
- Edge Inference
Running a vision model on the device that captures the images, such as a phone, camera, robot or microcontroller, instead of in a data centre, covering why it is done, the limits of compute, memory, power and heat, the classes of hardware and runtimes, the roofline model for finding memory-bound layers, split and offloaded computation, and how latency and energy are measured.
- GPUs for Computer Vision
The graphics processing unit as a machine for vision workloads, covering SIMT execution in warps, latency hiding by many resident threads, the memory hierarchy from registers to host memory, data reuse by tiling, why convolutions suit GPUs and element-wise operations are limited by bandwidth, TF32 and reduced precision, memory capacity, host–device transfer, GPUs from other vendors and how to choose one.
References
- NVIDIA TensorRT open-source components. README, NVIDIA/TensorRT at tag v11.4 (9 October 2026; Apache License 2.0).
- NVIDIA TensorRT open-source components. TensorRT OSS Release Changelog, CHANGELOG.md, NVIDIA/TensorRT at tag v11.4.
- NVIDIA TensorRT open-source components. TensorRT Import Workflows: Step-by-Step Guide, documents/import_workflows.md, NVIDIA/TensorRT at tag v11.4.
- NVIDIA TensorRT open-source components. trtexec README, samples/trtexec/README.md, NVIDIA/TensorRT at tag v11.4 (its License section refers to the TensorRT Software License Agreement, not read).
- NVIDIA TensorRT open-source components. sampleEditableTimingCache README, samples/sampleEditableTimingCache/README.md, NVIDIA/TensorRT at tag v11.4 (License section refers to the TensorRT Software License Agreement, not read).
- NVIDIA TensorRT open-source components. sampleOnnxMNIST README, samples/sampleOnnxMNIST/README.md, NVIDIA/TensorRT at tag v11.4 (License section refers to the TensorRT Software License Agreement, not read).
- NVIDIA TensorRT open-source components. strongly_type_autocast sample README, samples/python/strongly_type_autocast/README.md, NVIDIA/TensorRT at tag v11.4 (License section refers to the TensorRT Software License Agreement, not read).
- NVIDIA Polygraphy. README and Working With Reduced Precision, tools/Polygraphy/README.md and tools/Polygraphy/how-to/work_with_reduced_precision.md, NVIDIA/TensorRT at tag v11.4.
- NVIDIA TensorRT releases v11.0 (2 June 2026) to v11.4 (9 October 2026), GitHub releases of NVIDIA/TensorRT.
- NVIDIA Model Optimizer. AutoCast (ONNX) guide, docs/source/guides/8_autocast.rst, NVIDIA/Model-Optimizer at tag 0.48.0 (2 November 2026; Apache License 2.0).
- NVIDIA Model Optimizer source, modelopt/onnx/autocast/nodeclassifier.py, NVIDIA/Model-Optimizer at tag 0.48.0 (Apache License 2.0).
- NVIDIA Model Optimizer source, modelopt/onnx/autocast/convert.py, NVIDIA/Model-Optimizer at tag 0.48.0 (Apache License 2.0).
- NVIDIA Model Optimizer source, modelopt/onnx/autocast/referencerunner.py, NVIDIA/Model-Optimizer at tag 0.48.0 (Apache License 2.0).
- NVIDIA Model Optimizer source, modelopt/onnx/autocast/precisionconverter.py, NVIDIA/Model-Optimizer at tag 0.48.0 (Apache License 2.0).
- ONNX Runtime documentation. TensorRT Execution Provider, docs/execution-providers/TensorRT-ExecutionProvider.md, microsoft/onnxruntime gh-pages branch at commit e3d000e (14 October 2026).
- TensorRT Backend for ONNX: Supported ONNX Operators, docs/operators.md, onnx/onnx-tensorrt branch 11.4-GA at commit 8f60063 (Apache License 2.0).