Deploying Vision Models

Deploying a vision model means turning the network that came out of training into a component that other software can call: a file that a runtime loads on a server, a phone, a camera or a robot, wrapped in the code that turns raw frames into the tensor the network expects and turns its outputs back into labels, boxes or masks. The trained weights are only part of what has to be reproduced. The same model gives different answers if the image is resized with another filter, its colour channels are in another order or its pixel values are scaled differently, and none of these mistakes raises an error. This article covers the steps from a trained model to a monitored service: export, the choice of runtime, packaging, serving on a server or on the device, pre- and post-processing parity, versioning and rollback, and monitoring in outline. The file format used for most exports is described in ONNX; the timing of a live camera pipeline in real-time vision pipelines; making the network itself smaller in model compression and quantization; and the constraints of running on the device itself in edge inference. Products named here are dated October 2026 and described from their makers’ documentation.

Purpose

In training, a model runs inside its framework, fed by its own data loader. In use, it runs in another process, often in another language, on hardware chosen for cost, power or latency, and receives images from a camera or a client. Deployment is the work of making that new environment compute the same function the evaluation measured, within its latency and resource limits, and of noticing when it stops doing so.

Sculley et al. (2015) described the code around a model in production systems as mostly “glue”: a mature system, they suggest, might end up as at most 5% machine-learning code and at least 95% code that moves data into and out of general-purpose machine-learning packages. Breck et al. (2017) turned the risks into a rubric of 28 tests, drawn from their experience with production systems, and found from structured interviews with 36 teams that the test for training/serving skew, whether training and serving compute the same input values, was among the least often implemented, although they found such skew responsible for production issues across a wide range of teams. For vision models, the input values in question are the pixels after decoding, resizing, colour conversion and normalization.

When to Use It

Every model used outside the training notebook needs some of these steps; how many depends on the form of deployment (see Variants). Offline inference on stored images needs little more than a pinned environment, the right preprocessing and a record of the model version; an online service adds a server, batching and gradual roll-out; a device adds conversion, tight budgets and slow updates. Even a model that only ever runs inside its training framework needs versioning and preprocessing parity.

How It Works

Export

Export turns the model from a program in the training framework into a self-contained description of its computation. For PyTorch models the usual paths are torch.export, which captures a whole-graph program that PyTorch’s own deployment tools consume, and torch.onnx.export, which writes an ONNX file for other runtimes; TensorFlow models are saved as SavedModels (see the PyTorch and TensorFlow articles). The export step is where data-dependent control flow, fixed shapes, unsupported operators and opset versions cause trouble; the ONNX article works through these pitfalls and shows a numerical parity check between the framework and the exported file. That check covers the network alone. The rest of this article is about everything around it.

Choosing a runtime

A runtime loads the exported model and executes it on a particular kind of hardware. The choice follows from the target rather than from a ranking of products:

• Hardware. A data-centre GPU, a server CPU, a phone’s neural processing unit and a microcontroller each have their own runtimes.
• Operator support at the opset or version the export produced, which differs per runtime and per operator.
• Portability. Some runtimes execute a portable graph; others compile it into a program for one device, which ties the artefact to that device.
• Numerics. Reduced precision (FP16, INT8) and fused kernels change outputs slightly, which has to be measured on task accuracy.
• Updates. A server can load a new file without any change on the client; an app may need a release through an app store.

The main families, as their makers describe them (October 2026):

• Framework-native. Running the model in PyTorch or TensorFlow itself avoids conversion, at the cost of shipping the framework.
• ONNX Runtime executes ONNX files and hands subgraphs to hardware-specific execution providers through a GetCapability() interface, with the CPU provider as the default (ONNX Runtime documentation); its providers include CUDA, TensorRT, OpenVINO, DirectML, Qualcomm QNN, NNAPI and Core ML.
• NVIDIA TensorRT compiles a network in a build phase that selects the fastest available kernel for each layer on the target GPU and writes a serialized engine. By default, an engine is guaranteed to work only with the same operating system, CPU architecture, GPU model and TensorRT version it was built with, and the documentation warns that deserializing an untrusted engine is equivalent to running untrusted native code (TensorRT 11.4.1 documentation). An engine is therefore a build product for one target, made from the portable model rather than replacing it.
• Intel OpenVINO reads ONNX files or converts them to its own representation, and can fold preprocessing steps into the model, as described below.
• On-device runtimes, such as LiteRT (formerly TensorFlow Lite), ExecuTorch and Apple’s Core ML, run converted models on phones and embedded devices; the edge inference article describes them.

Packaging

The deployable unit is more than the weights. Our reading of what a vision model needs, gathered from the failure modes below, is:

• The model file in the runtime’s format, with its exact input and output names, types and shapes, including which dimensions may vary.
• The preprocessing contract: how images are decoded, the colour order (RGB or BGR), the target size, whether the aspect ratio is kept with padding (letterboxing) or the image is stretched or cropped, the interpolation filter and its library, the scaling to $[0, 1]$, the per-channel mean and standard deviation, and the tensor layout (NCHW or NHWC).
• The postprocessing contract: class names in output order, score thresholds, the box format and coordinate frame, and the non-maximum suppression settings.
• Identity and provenance: a model version, a checksum of the file, and the training run, data and code versions it came from.
• A golden set: a few dozen inputs with the expected preprocessed tensors and outputs, so that any new serving environment can be tested against the reference.

ONNX files have a metadata_props field of free-form key–value pairs that can carry the contract inside the model file, as the example below does; Triton’s model repository keeps each version in a numbered subdirectory beside the model’s configuration file, and TensorFlow Serving exports each version to a numbered subdirectory of its own. Keeping the contract in the same artefact as the weights means that a new model version cannot be deployed with the old preprocessing by accident.

Some runtimes can take part of the contract into the model. OpenVINO’s preprocessing API declares the input as it arrives, for example an 8-bit $480 \times 640$ BGR image in NHWC layout, and adds colour conversion, resizing and mean and scale normalization to the execution graph, so the application passes the raw image (OpenVINO documentation). Core ML Tools stores a global scale and per-channel biases with an image input. Its documentation notes that Core ML has one scale for all channels, while torchvision models normalize each channel by its own standard deviation, and recommends their average, 0.226, in place of 0.229, 0.224 and 0.225. By our arithmetic this scales the red channel by $0.229/0.226 \approx 1.013$, a 1.3% deviation from the training preprocessing that the documentation describes as working reasonably well; it is a deliberate mismatch that the accuracy test of the converted model should measure.

Serving on a server or on the device

A model server loads models from a repository, exposes them over HTTP or gRPC, and schedules requests onto CPUs or GPUs. As of October 2026:

• NVIDIA Triton Inference Server (release 2.73.0, 30 September 2026) serves TensorRT, ONNX, PyTorch, OpenVINO and other back ends from a model repository in which each model has numbered version subdirectories. A version policy selects which versions are loaded, by default only the numerically greatest; models can be loaded and unloaded explicitly through an API, and the documentation advises against its polling mode, which picks up repository changes on its own, in production. Ensembles chain several models into one pipeline, such as a preprocessing model followed by a classifier, and the Python and DALI back ends run Python code or DALI pipelines as models, so preprocessing can run on the server. Because TensorRT engines are tied to a GPU’s compute capability, a configuration can map each compute capability to its own engine file (Triton documentation at v2.73.0).
• TensorFlow Serving serves SavedModels as versioned “servables”, can load several versions concurrently over the lifetime of one server for gradual roll-out, and includes a request batcher (TensorFlow Serving architecture document; see TensorFlow).
• TorchServe, the model server in the PyTorch GitHub organization, carries a notice that it is no longer actively maintained and has no planned updates, bug fixes or security patches; its repository is archived, and its last release, 0.12.0, dates from September 2024.

Batching is the server’s main lever for throughput: combining concurrent requests amortizes per-call overheads, such as copying inputs to the GPU, and lets the hardware process many inputs in parallel (Crankshaw et al., 2017). Their Clipper system adapted the maximum batch size of each model by additive increase and multiplicative decrease, growing it until the batch latency exceeded the latency objective and then cutting it by 10%; on their benchmarks, batching raised the throughput of a scikit-learn linear SVM up to 26 times while meeting a 20 ms objective, and briefly delaying dispatch so that more requests could join a batch helped some models (3.3 times for one model with a 2 ms delay) and not others. Triton’s dynamic batcher combines requests on the server: by default it forms batches as large as possible up to the model’s maximum and does not wait, and a max_queue_delay_microseconds setting lets a request wait a bounded time for others to join. The trade-off is direct: every microsecond a request waits for company is added to its latency. In our reading, a model exported with a fixed batch size cannot take part in dynamic batching, one more reason to export with a symbolic batch dimension.

On the device, there is one user and usually one frame at a time, so batching gives way to the latency of a single inference, memory, power, thermal limits and the size of the app. The model is converted to the device’s runtime, often quantized, and shipped inside an application or downloaded by it. Updates are slower and cannot be rolled back as quickly as on a server, and the team cannot inspect inputs unless the device sends them back, which raises privacy questions. The edge inference article covers these constraints.

Pre- and post-processing parity

The network computes a function of a tensor, and it was trained on tensors produced by one particular preprocessing pipeline. Any difference between that pipeline and the one used in deployment changes the input distribution, usually without an error. The common sources in vision are these:

• Colour order. OpenCV decodes colour images in B, G, R order, while Pillow, torchvision and most training code use R, G, B (see OpenCV). OpenCV’s blobFromImage leaves the order unchanged unless swapRB=True is passed (OpenCV 5.0.0 header). A swapped model sees red objects as blue.
• Resizing. Libraries interpolate differently, and their defaults differ. In Pillow 13.0.0, Image.resize uses bicubic filtering unless told otherwise, and its bilinear and bicubic filters always widen with the downscaling factor (antialiasing); torchvision 0.29’s Resize uses bilinear interpolation with antialiasing on by default; OpenCV 5.0.0’s cv2.resize defaults to INTER_LINEAR, with no prefilter, and its documentation recommends INTER_AREA for shrinking; the ONNX Resize operator (opset 19) defaults to nearest-neighbour mode with antialiasing off. Parmar et al. (2022) showed how far apart the results are. Downsampling the same 1024-pixel face images to 299 pixels with TensorFlow’s and PyTorch’s fixed-width bilinear filters instead of Pillow’s bicubic filter changed the images enough to give a Fréchet Inception Distance of 4.34 and 4.36 between two copies of the same set, against 0.64 for Pillow’s own bilinear filter (their Table 1); they note that recent generative models typically differ by less than 1 on this benchmark. Downsampling without a prefilter is aliasing in the sense of sampling and aliasing.
• Geometry. Stretching an image to a square, center-cropping it and letterboxing it (scaling it to fit and padding the rest) give different pictures of the same scene. A detector trained on letterboxed images and served stretched images sees objects with the wrong aspect ratio.
• Value scaling. Pixel values may be expected in $[0, 255]$, $[0, 1]$ or $[-1, 1]$, normalized with per-channel means and standard deviations from the training set or from ImageNet. Forgetting one step, or applying it twice, shifts every input.
• Decoding. JPEG decoders can differ slightly, and recompressing images at the server changes them. Parmar et al. found that saving $256 \times 256$ face images as JPEG at Pillow’s default quality of 75 gave an FID of 20.96 against the lossless PNG originals (their Figure 6; the text rounds it to 21), although the copies were perceptually indistinguishable, and noted that even quality 100 is not lossless. Libraries also treat metadata differently: OpenCV 5.0.0 rotates a decoded image according to its EXIF orientation tag unless IMREAD_IGNORE_ORIENTATION or IMREAD_UNCHANGED is set, while Pillow 13.0.0 returns the stored pixels and leaves rotation to ImageOps.exif_transpose (we confirmed both on a test JPEG), so a phone photo can reach the model rotated in one pipeline and upright in the other.
• Layout and type. NCHW against NHWC, and float32 against float16 inputs.

Postprocessing has its own conventions. Boxes predicted in the coordinates of the letterboxed input must be mapped back to the original frame. With a model input of $S \times S$ pixels and a frame of $W \times H$, letterboxing scales by $s = S / \max(W, H)$ and pads by $p_x = (S - sW)/2$ and $p_y = (S - sH)/2$, so a predicted coordinate $x'$ maps back to $x = (x' - p_x)/s$ (our derivation; rounding of the resized size changes it by a pixel). Using the stretch mapping on a letterboxed model, or forgetting the padding offset, shifts and scales every box. Score thresholds, IoU thresholds and whether suppression is done per class or across classes change which detections survive, and implementations disagree on details: the ONNX NonMaxSuppression operator expects corner boxes as $[y_1, x_1, y_2, x_2]$ by default, or centre and size with center_point_box=1, while torchvision’s nms expects $(x_1, y_1, x_2, y_2)$ (ONNX 1.23.2 operator documentation; TorchVision 0.29). Since IoU does not change when the $x$ and $y$ axes are exchanged, the two corner orders give the same suppressions, but boxes read back in the wrong order are transposed; passing centre-and-size boxes where corners are expected changes the overlaps and so the suppressions. Neither raises an error.

The remedy is to treat preprocessing as part of the model and test it. A parity test runs the serving pipeline on the golden inputs and compares the resulting tensors and outputs with the reference, with tolerances set from the data type. Breck et al. recommend the stronger form in production: log a sample of serving inputs with identifiers and check that the values match what the training code computes for the same examples.

Versioning, canaries and rollback

Each deployed model has a version that identifies the exact file, and every prediction can be traced to it. Versions change for several reasons that are worth keeping apart: new training data or weights, a new architecture, a new preprocessing contract, a new export or runtime version. The last two change outputs without any change to the weights.

Breck et al. (2017) recommend testing that a new model loads in the production serving binaries and runs on production inputs before it serves traffic, because model code changes faster than serving code and an older server may not support an operator a newer model uses. They describe turning a new model up gradually, with old and new running concurrently and the new one receiving a small and growing fraction of traffic (a canary), and being able to roll back quickly to the previous known-good version, a procedure operators should practise. A further option, which the cited sources do not describe, is to run a new model in shadow, on copies of live requests without acting on its outputs, which compares the two on real inputs before any user sees the new one. Model servers support these patterns by keeping several versions loaded at once, as in Triton’s version policies and TensorFlow Serving’s servables.

Monitoring

Once deployed, a model can fail because its inputs change (new cameras, lighting, seasons, firmware that alters colour processing), because an upstream component changes, or because of a bug in a new release. Ground-truth labels usually arrive late or never, so monitoring watches what can be seen:

• Inputs. Statistics of the images and of the preprocessed tensors: brightness, contrast, colour channel means, image size, decode failures. Breck et al. recommend checking serving inputs against a schema of expected feature values, which can be inferred from the training data, with tuned alert thresholds. For high-dimensional images, Rabanser et al. (2019) compared shift detectors on MNIST and CIFAR-10 with simulated shifts and found that two-sample tests on the softmax outputs of a classifier trained on the source data (black-box shift detection) performed best of the dimensionality reductions they tried for univariate tests. In their Table 2a, this detector found large shifts in 54% of cases from 20 samples, medium shifts in 57% only from 500 samples, and small shifts in 54% only from 10,000; the text summarizes this as medium and small shifts needing orders of magnitude more samples.
• Predictions. The distribution of predicted classes, scores and box counts. Sculley et al. propose comparing the distribution of predicted labels with that of observed labels as a simple but useful diagnostic.
• System health. Latency, throughput and memory, sliced by model and data version as well as by code version (Breck et al., Monitor 6), and error rates.
• Quality. Accuracy on a periodically labelled sample of logged inputs (Breck et al., Monitor 7).

Aggregate statistics have blind spots: an error that preserves them is invisible to them, as the example below shows.

How to Apply It

In outline:

1. Export the model with the batch dimension (and, for fully convolutional models, the image size) declared dynamic, and check numerical parity of the network alone against the framework.
2. Choose the runtime for the target and check its operator support and numerical behaviour on the task metric, especially for reduced precision.
3. Package the model with its preprocessing and postprocessing contract, labels, version, checksum and a golden set.
4. Implement serving preprocessing from the contract, preferably the training code itself or a graph-embedded version, and run the parity test against the golden set.
5. Roll out behind a canary or shadow deployment, with the previous version ready for rollback.
6. Monitor inputs, predictions, latency and labelled samples.

Example: one model, six serving pipelines

The script trains a small CNN to sort rendered parts into six classes, a disk or a square in a warm, green or cold colour, on $96 \times 96$ frames downsampled to $32 \times 32$ with Pillow’s bilinear filter. It exports the model to ONNX with a dynamic batch dimension, stores the preprocessing contract, the labels and a version in the file’s metadata, and then serves the same 600 test frames, stored as PNG files, through six preprocessing pipelines in ONNX Runtime. One reproduces the training code with Pillow; one decodes with OpenCV but converts to RGB and resizes with Pillow; the others use OpenCV with INTER_AREA resizing, OpenCV’s default resizing, OpenCV without the BGR-to-RGB conversion, and Pillow without the mean and standard-deviation normalization. Each pipeline is compared with the reference on accuracy and agreement and put through a parity test on 60 golden frames. Training is repeated for seeds 0, 1 and 2 to show the spread. It uses PyTorch 2.14.1, ONNX 1.23.2, ONNX Runtime 1.31.1, OpenCV 5.0.0 and Pillow 13.0.0 on one CPU thread, takes about 20 seconds (17.9 to 24.0 s in five timed runs on an Apple M5 CPU), and printed identical output in four runs of the final version.

import hashlib
import io
import json
import cv2
import numpy as np
import onnx
import onnxruntime as ort
import PIL
import torch
import torch.nn as nn
import torch.nn.functional as F
from PIL import Image

torch.set_num_threads(1)
print(f"torch {torch.__version__}, onnx {onnx.__version__}, onnxruntime {ort.__version__}, "
      f"opencv {cv2.__version__}, pillow {PIL.__version__}")

FRAME, SIZE = 96, 32                      # camera frame and model input (pixels)
LABELS = [f"{c} {s}" for c in ("warm", "green", "cold") for s in ("disk", "square")]
COLOURS = {"warm": ((0.75, 1.0), (0.1, 0.45), (0.0, 0.2)),    # RGB ranges per family
           "green": ((0.0, 0.3), (0.6, 1.0), (0.0, 0.3)),
           "cold": ((0.0, 0.2), (0.3, 0.65), (0.75, 1.0))}

def render(k, rng):
    """A 96x96 RGB uint8 frame: one coloured disk or square on a grey, noisy background."""
    family, shape = LABELS[k].split()
    yy, xx = np.mgrid[0:FRAME, 0:FRAME] + 0.5
    r = rng.uniform(12, 24)
    cx, cy = rng.uniform(r + 2, FRAME - r - 2, 2)
    if shape == "disk":
        mask = (xx - cx) ** 2 + (yy - cy) ** 2 <= r ** 2
    else:
        mask = (abs(xx - cx) <= r) & (abs(yy - cy) <= r)
    colour = np.array([rng.uniform(*c) for c in COLOURS[family]])
    img = np.full((FRAME, FRAME, 3), rng.uniform(0.3, 0.6))
    img[mask] = colour
    img = img * rng.uniform(0.7, 1.1) + rng.normal(0, 0.06, img.shape)
    return (np.clip(img, 0, 1) * 255).round().astype(np.uint8)

def png_bytes(rgb):                       # what the camera service stores and sends
    buf = io.BytesIO()
    Image.fromarray(rgb).save(buf, format="PNG")
    return buf.getvalue()

def frames(n, seed):
    rng = np.random.default_rng(seed)
    y = np.arange(n) % len(LABELS)
    return [png_bytes(render(k, rng)) for k in y], y

# ---- Reference preprocessing: the code used in training (Pillow, RGB, bilinear) ----
def resize_pil(rgb):
    return np.asarray(Image.fromarray(rgb).resize((SIZE, SIZE), Image.Resampling.BILINEAR))

def to_tensor(rgb_small, mean, std):
    x = rgb_small.astype(np.float32) / 255.0
    return ((x - mean) / std).transpose(2, 0, 1)          # HWC -> CHW

def reference(png, cfg):
    rgb = np.asarray(Image.open(io.BytesIO(png)).convert("RGB"))
    return to_tensor(resize_pil(rgb), cfg["mean"], cfg["std"])

# ---- Training ----
train_png, y_train = frames(1800, seed=1)
test_png, y_test = frames(600, seed=2)
small = np.stack([resize_pil(np.asarray(Image.open(io.BytesIO(p)))) for p in train_png])
cfg = {"mean": (small / 255.0).mean(axis=(0, 1, 2)).astype(np.float32),
       "std": (small / 255.0).std(axis=(0, 1, 2)).astype(np.float32)}
x_train = torch.from_numpy(np.stack([reference(p, cfg) for p in train_png]))

class PartsCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.c1, self.b1 = nn.Conv2d(3, 16, 3, padding=1), nn.BatchNorm2d(16)
        self.c2, self.b2 = nn.Conv2d(16, 32, 3, padding=1), nn.BatchNorm2d(32)
        self.c3, self.b3 = nn.Conv2d(32, 32, 3, padding=1), nn.BatchNorm2d(32)
        self.fc = nn.Linear(32, len(LABELS))
    def forward(self, x):
        x = F.max_pool2d(F.relu(self.b1(self.c1(x))), 2)
        x = F.max_pool2d(F.relu(self.b2(self.c2(x))), 2)
        x = F.relu(self.b3(self.c3(x)))
        return self.fc(x.mean(dim=(2, 3)))

def train(seed):
    torch.manual_seed(seed)
    model, yt = PartsCNN(), torch.from_numpy(y_train)
    opt = torch.optim.Adam(model.parameters(), lr=1e-2)
    for epoch in range(8):
        perm = torch.randperm(len(yt), generator=torch.Generator().manual_seed(seed * 10 + epoch))
        for i in range(0, len(yt), 50):
            idx = perm[i:i + 50]
            loss = F.cross_entropy(model(x_train[idx]), yt[idx])
            opt.zero_grad(); loss.backward(); opt.step()
    return model.eval()

# ---- Packaging: model, preprocessing contract, labels and version in one file ----
def package(model, path, version):
    prog = torch.onnx.export(model, (x_train[:2],), input_names=["image"], output_names=["logits"],
                             dynamic_shapes={"x": {0: torch.export.Dim("batch")}}, verbose=False)
    prog.save(path, external_data=False)
    m = onnx.load(path)
    for node in m.graph.node:                 # drop exporter debug info (source paths)
        del node.metadata_props[:]
    contract = {"color_order": "RGB", "layout": "NCHW", "size": [SIZE, SIZE],
                "resize": "Pillow 13 BILINEAR (antialiased)", "scale": 1 / 255,
                "mean": [round(float(v), 4) for v in cfg["mean"]],
                "std": [round(float(v), 4) for v in cfg["std"]]}
    for k, v in {"preprocessing": json.dumps(contract), "labels": json.dumps(LABELS),
                 "model_version": version}.items():
        m.metadata_props.add(key=k, value=v)
    onnx.save(m, path)
    return hashlib.sha256(open(path, "rb").read()).hexdigest()

# ---- Serving side: reads everything it needs from the package ----
def load(path):
    m = onnx.load(path)
    meta = {p.key: p.value for p in m.metadata_props}
    pre = json.loads(meta["preprocessing"])
    so = ort.SessionOptions(); so.intra_op_num_threads = 1
    sess = ort.InferenceSession(path, so, providers=["CPUExecutionProvider"])
    return sess, {"mean": np.float32(pre["mean"]), "std": np.float32(pre["std"])}, meta

def decode_cv(png):                       # OpenCV decodes colour images as B, G, R
    return cv2.imdecode(np.frombuffer(png, np.uint8), cv2.IMREAD_COLOR)

VARIANTS = {
    "reference (Pillow)": reference,
    "OpenCV decode + Pillow resize": lambda p, c: to_tensor(
        resize_pil(cv2.cvtColor(decode_cv(p), cv2.COLOR_BGR2RGB)), c["mean"], c["std"]),
    "OpenCV, INTER_AREA resize": lambda p, c: to_tensor(cv2.resize(
        cv2.cvtColor(decode_cv(p), cv2.COLOR_BGR2RGB), (SIZE, SIZE),
        interpolation=cv2.INTER_AREA), c["mean"], c["std"]),
    "OpenCV, default resize": lambda p, c: to_tensor(cv2.resize(
        cv2.cvtColor(decode_cv(p), cv2.COLOR_BGR2RGB), (SIZE, SIZE)), c["mean"], c["std"]),
    "OpenCV, BGR not converted": lambda p, c: to_tensor(cv2.resize(
        decode_cv(p), (SIZE, SIZE), interpolation=cv2.INTER_AREA), c["mean"], c["std"]),
    "Pillow, mean/std skipped": lambda p, c: to_tensor(resize_pil(
        np.asarray(Image.open(io.BytesIO(p)))), 0.0, 1.0),
}

def parity_test(variant, cfg, sess, golden_png, golden_x, golden_logits):
    """Compare a serving pipeline with the stored golden tensors and outputs."""
    x = np.stack([variant(p, cfg) for p in golden_png])
    logits = sess.run(["logits"], {"image": x})[0]
    tol_x = 1.0 / (255 * cfg["std"].min())        # one grey level after normalization
    dx = np.abs(x - golden_x).max()
    ok = (dx <= tol_x and np.allclose(logits, golden_logits, rtol=1e-4, atol=1e-5)
          and (logits.argmax(1) == golden_logits.argmax(1)).all())
    return ok, dx

results = {name: [] for name in VARIANTS}
for seed in (0, 1, 2):
    model = train(seed)
    digest = package(model, f"parts_v{seed + 1}.onnx", version=f"1.{seed}.0")
    sess, pcfg, meta = load(f"parts_v{seed + 1}.onnx")
    if seed == 0:
        print("package metadata:", sorted(meta), "| sha256", digest[:12], "...")
        print("contract:", meta["preprocessing"])
    golden_png = test_png[:60]                    # stored with the package
    golden_x = np.stack([reference(p, pcfg) for p in golden_png])
    golden_logits = sess.run(["logits"], {"image": golden_x})[0]
    ref_pred = None
    for name, variant in VARIANTS.items():
        x = np.stack([variant(p, pcfg) for p in test_png])
        pred = sess.run(["logits"], {"image": x})[0].argmax(1)
        ref_pred = pred if ref_pred is None else ref_pred
        ok, dx = parity_test(variant, pcfg, sess, golden_png, golden_x, golden_logits)
        results[name].append(((pred == y_test).mean(), (pred == ref_pred).mean(), dx, ok, pred,
                             x.mean(axis=(0, 2, 3))))

print(f"\n{'serving pipeline':31s} {'accuracy':>13s} {'agrees w/ ref':>13s} "
      f"{'max |dx|':>8s}  parity test")
for name, r in results.items():
    acc, agr, dx, ok = (np.array([v[i] for v in r]) for i in range(4))
    print(f"{name:31s} {acc.min():.3f}-{acc.max():.3f}   {agr.min():.3f}-{agr.max():.3f}   "
          f"{dx.max():7.3f}   {'pass' if ok.all() else 'FAIL'} {ok.sum()}/3")

print("\nseed 0: what aggregate monitors would see on the 600 test frames")
for name, r in results.items():
    counts = np.bincount(r[0][4], minlength=len(LABELS))
    print(f"{name:31s} predictions per class {counts}  input channel means {r[0][5].round(2)}")

bgr_pred = results["OpenCV, BGR not converted"][0][4]
per_class = [(bgr_pred[y_test == k] == k).mean() for k in range(len(LABELS))]
print("\nBGR not converted, seed 0, accuracy per class:",
      ", ".join(f"{LABELS[k]} {a:.2f}" for k, a in enumerate(per_class)))

Output:

torch 2.14.1, onnx 1.23.2, onnxruntime 1.31.1, opencv 5.0.0, pillow 13.0.0
package metadata: ['labels', 'model_version', 'preprocessing'] | sha256 994390dc51e3 ...
contract: {"color_order": "RGB", "layout": "NCHW", "size": [32, 32], "resize": "Pillow 13 BILINEAR (antialiased)", "scale": 0.00392156862745098, "mean": [0.3963, 0.412, 0.3966], "std": [0.1472, 0.1193, 0.1468]}

serving pipeline                     accuracy agrees w/ ref max |dx|  parity test
reference (Pillow)              0.992-0.998   1.000-1.000     0.000   pass 3/3
OpenCV decode + Pillow resize   0.992-0.998   1.000-1.000     0.000   pass 3/3
OpenCV, INTER_AREA resize       0.995-0.997   0.995-0.998     0.748   FAIL 0/3
OpenCV, default resize          0.785-0.883   0.787-0.880     2.958   FAIL 0/3
OpenCV, BGR not converted       0.327-0.333   0.327-0.333     6.767   FAIL 0/3
Pillow, mean/std skipped        0.170-0.400   0.172-0.402     3.929   FAIL 0/3

seed 0: what aggregate monitors would see on the 600 test frames
reference (Pillow)              predictions per class [100 100 100 100  99 101]  input channel means [0.02 0.03 0.02]
OpenCV decode + Pillow resize   predictions per class [100 100 100 100  99 101]  input channel means [0.02 0.03 0.02]
OpenCV, INTER_AREA resize       predictions per class [100 100  99 101  99 101]  input channel means [0.02 0.03 0.02]
OpenCV, default resize          predictions per class [ 88 112  69 131  70 130]  input channel means [0.02 0.03 0.02]
OpenCV, BGR not converted       predictions per class [100 100 100 100 100 100]  input channel means [0.02 0.03 0.02]
Pillow, mean/std skipped        predictions per class [ 94   4  63  95   0 344]  input channel means [0.4  0.42 0.4 ]

BGR not converted, seed 0, accuracy per class: warm disk 0.00, warm square 0.00, green disk 1.00, green square 1.00, cold disk 0.00, cold square 0.00

The serving code reads the mean and standard deviation from the model file rather than from its own constants, so each model version brings its own contract. The exporter records, in each node’s metadata, the stack trace of the Python code it came from, including the script’s absolute path; the script deletes it, so the file’s checksum identifies the model rather than the directory it was built in. Decoding with OpenCV is harmless once the channels are converted to RGB and the resize is done as in training: that pipeline produces bit-identical tensors and passes the parity test for all three models. The other four fail it, with very different consequences:

• INTER_AREA is a faithful port on the task: accuracy and agreement stay within a few frames of the reference. Its tensors still differ by up to 0.75 in normalized units, about three quarters of a channel’s standard deviation, because the filters average different neighbourhoods: at this exact factor of 3, OpenCV’s area filter takes the mean of each $3 \times 3$ block, while Pillow’s bilinear filter, stretched by the factor, weights five pixels along each axis by $1, 2, 3, 2, 1$ (we checked both by resizing test arrays). The parity test is right to flag the change; whether to accept it is a decision for the task test, after which the contract records the new resize.
• The default cv2.resize loses 11 to 21 points of accuracy. At a factor of 3 its bilinear interpolation lands exactly on the centre pixel of each block, so it reads one pixel in nine and averages nothing (also checked). By our arithmetic the pixel noise then reaches the network with about twice the standard deviation per axis that Pillow’s filter leaves ($1$ against $\sqrt{19}/9 \approx 0.48$), and edges become jagged; presumably this is why many disks are read as squares.
• Skipping the BGR-to-RGB conversion turns warm parts into cold ones and back while leaving green parts unchanged, so accuracy is zero on four classes and perfect on two. A test set dominated by green parts would have hidden it.
• Skipping the normalization feeds raw values in $[0, 1]$ to a network trained on standardized inputs, which by our arithmetic shrinks their spread 6.8 to 8.4 times (one over the stored standard deviations) and moves their mean, and the predictions collapse onto a few classes.

The aggregate monitors see only some of this. Missing normalization moves the input channel means from 0.02 to 0.40 and skews the prediction histogram, and the default resize shifts the histogram towards squares. The BGR swap changes neither: every class is still predicted 100 times, because warm and cold parts swap one for one on this balanced stream, and the channel means stay put, because the red and blue channels of this synthetic data have nearly equal means (0.3963 and 0.3966). Real images rarely have such symmetric statistics, but the lesson holds: an error that preserves the monitored statistics is invisible to them, while a parity test against reference outputs catches it whatever the data look like. The tolerance of one grey level is our choice for 8-bit inputs and float32 arithmetic; a reduced-precision runtime needs tolerances set from its own precision and is better judged on task accuracy, as in the quantization article.

Variants

• Offline batch inference, such as labelling an archive of images overnight: latency hardly matters, and each result records the model version that produced it.
• An online service answering client requests through a model server, with batching, latency targets, canaries and rollback.
• Live video, where frame budgets, queues and timestamps dominate, as described in the real-time vision pipelines article.
• On-device deployment, with conversion to a mobile or embedded runtime, usually quantization, and updates shipped to devices the team does not control.
• Cascades and hybrids, in which a small model filters frames on the device or at the camera and a larger model on a server handles the frames that pass.
• Multi-model pipelines, such as detection followed by classification or tracking, where each stage has its own model, version and contract, and the interfaces between stages need parity tests of their own.

Best Practices

• One preprocessing implementation, shared by training and serving or embedded in the graph; a graph version still has to match training’s resize mode and antialiasing, which each library defaults differently.
• A golden set of real or rendered images, with expected tensors and outputs, turns every new runtime, device or library upgrade into a test; random noise exercises value ranges poorly.
• Two kinds of test: numerical tolerances catch accidental changes, and accuracy and agreement on a labelled set decide whether an intended change, such as FP16 or a new resize, is acceptable.
• Pinned versions of the exporter, runtime and preprocessing libraries, which are part of the deployed function.
• Trusted artefacts only. TensorRT’s documentation equates loading an untrusted engine with running untrusted native code, and the PyTorch and TensorFlow articles describe similar warnings for their formats.

Pitfalls

• Porting preprocessing with another library’s defaults: Pillow resizes bicubically, torchvision antialiases, OpenCV’s bilinear resize has no prefilter, ONNX Resize is nearest-neighbour, and blobFromImage does not swap channels.
• Normalizing twice, for example when a runtime’s image input already scales pixels and the application scales them again.
• Postprocessing drift: thresholds or class order hard-coded differently in the server, or boxes mapped back without the letterbox offset.
• A fixed batch size in the export, which prevents dynamic batching and fails on other batch sizes.
• Engines on the wrong hardware, built for one GPU model or runtime version and deployed on another.
• Benchmarking the network instead of the system, omitting decoding, preprocessing, transfer, queueing and postprocessing.
• Unmaintained serving software: as of October 2026, TorchServe receives no security patches.

Evidence and Impact

The evidence on deployment practice comes mostly from engineering reports rather than controlled studies. Besides the skew finding above, Breck et al. (2017) report that many of the teams they interviewed canaried models, usually because their release framework made it easy, though a canary catches problems long after the decisions that caused them. Teams working purely with images or audio told them that many of the rubric’s data tests did not apply; the authors pointed to manual inspection of raw data as a substitute, which can reveal skewed distributions or backgrounds correlated with the label.

Parmar et al. (2022) showed that preprocessing changes what a model learns, not only what it is evaluated on: StyleGAN2 models trained on FFHQ downsampled with different resize functions reached FIDs from 4.82 to 6.21 under otherwise identical training (their Table 3). For monitoring, Rabanser et al. (2019) also found that shifts affecting only 10% of inputs were hard to detect, which suggests why aggregate monitoring complements parity tests and labelled samples rather than replacing them.