Vision Systems & Engineering
MLOps for Computer Vision
The engineering practices that keep a vision model reproducible and maintainable after its first release, from versioning images, annotations and models by their content through pipelines that re-run only the stages a change affects, experiment tracking and the model registry to continuous training and lineage, with an example that caches a shape classifier's pipeline by content hash.
intermediate
MLOps (machine-learning operations) is the set of engineering practices that keep a learned model reproducible, retrainable and traceable once it leaves the experiment in which it was first trained. For a vision system the practices are about four kinds of artefact and the links between them: the images, the annotations, the code and environment that turn them into a model, and the model files that are deployed. Each is versioned, the steps that connect them are recorded as a pipeline, every training run is logged with its parameters and results, and the models that may be deployed are kept in a registry that knows where each came from. This article covers versioning by content, pipelines that re-run only the stages a change affects, experiment tracking, the model registry, continuous integration, delivery and training, lineage and reproducibility. The neighbouring steps have their own articles: packaging, canaries and rollback in deploying vision models, drift detection and re-validation in model monitoring, regression tests between versions in testing and validating vision systems, and the design of the whole system, including the retraining loop in outline, in designing a computer vision system. Tools are described as their own documentation describes them, as of November 2026: DVC 3.67.1 (31 March 2026) and MLflow 3.18.0 (4 November 2026).
Purpose
A model is a function of its training data, its labels, its preprocessing, its training code and settings, and the library versions that ran them. Change any of these and the model changes, often without an error anywhere. Once a model is in use, someone will need to answer questions that only records can answer: which images and which label version trained the model now serving, whether last month’s model can be rebuilt, what else must be retrained when a resize routine or an annotation guideline changes, and whether a new candidate differs from the old one only in the change that was intended.
Sculley et al. (2015) framed the cost of not having such answers as technical debt. The vision system design article reports their entanglement principle, their warnings about thresholds and monitoring, and their point that data dependencies are harder to detect than code dependencies. Several of their other risk factors are what MLOps tooling addresses directly:
- Unstable data dependencies. Some inputs produced by other systems are unstable, changing behaviour over time, and even an “improvement” to an input can harm a model fitted to the old one. Their mitigation is a versioned copy of the signal, frozen until an updated version has been vetted, at the cost of staleness and of maintaining several versions. In a vision system, the analogues in our reading are an upstream detector whose crops feed a classifier, a camera’s image processing, and a preprocessing library.
- Pipeline jungles. Data preparation that grows by accretion into “scrapes, joins, and sampling steps”, often with intermediate files, which is hard to manage, test and recover; they argue it can be avoided only by thinking about data collection and feature extraction as a whole.
- Dead experimental codepaths. Experiments implemented as conditional branches in production code, which accumulate and make testing all interactions between branches difficult or impossible; they suggest periodically re-examining each experimental branch to see what can be removed.
- Configuration debt. The configuration of a mature system can exceed its code in lines, and each line can be wrong. Their principles: a configuration should be easy to specify as a small change from a previous one, differences between two models’ configurations should be easy to see, basic facts such as the transitive closure of data dependencies should be checkable automatically, and configurations should be code-reviewed and checked into a repository.
- Reproducibility debt. Strict reproducibility is made difficult by randomized algorithms, non-determinism in parallel learning, dependence on initial conditions and interaction with the external world.
- Undeclared consumers of a model’s outputs, which couple it silently to other systems, and process management debt: many models, each with configurations and resources, and processes with many manual steps, which they name as a system-level smell.
Among the questions they suggest for measuring debt are how precisely the impact of a change can be measured and what the transitive closure of all data dependencies is. Versioning, pipelines and lineage are the means of answering both.
When to Use It
Any model that will be retrained, compared with a successor or audited needs some of these practices. A one-off experiment needs little more than a record of the data version, code version and seed. A model that is deployed and retrained as data arrive, as in an active learning loop or after monitoring finds a shift, needs versioned data and labels, a pipeline that can be re-run, and a registry from which deployment takes its model. Several teams or many models sharing data and preprocessing add the need for lineage across them, so that the consequences of a change can be found. The practices cost storage, discipline and some tooling; for a single model trained once on a fixed public benchmark, a pinned environment and a recorded seed may be all that pays.
How It Works
Versioning data, annotations and models by content
A content hash, such as SHA-256 or MD5 of a file’s bytes, identifies the file’s content: equal bytes give equal hashes, and in practice any change gives a different one. A dataset version can then be a manifest, a list of file names with their hashes, and the hash of that manifest identifies the whole set. Storing the files under their hashes (content addressing) means that a file shared by many versions is stored once, and that a version recorded in a manifest can always be checked against the files.
DVC applies this to data and models alongside Git. Its README describes storing data and model files in a cache outside Git, with remote storage on cloud or on-premise systems for sharing, while small metadata files that stand in for the data are committed to the Git repository, so that a commit identifies both the code and the data version. Which hash DVC records depends on where the file lives: a comment in dvc/output.py at 3.67.1 lists md5 as the checksum name for local and SSH outputs (etag for S3, GS, OSS, Azure and HTTP), and outputs written by DVC 2.x without a hash name are read as md5-dos2unix and updated to md5 when their checksum changes.
Three properties of vision data matter here:
- Images are large and binary, so they are kept out of the code repository and referenced by hash; the code repository stays small enough to clone, and the data store is shared.
- A byte hash identifies an encoding, not pixels. Re-saving a PNG with another compression level changes its bytes but not its pixels, since PNG has no lossy mode (image file formats); decoding and re-encoding a JPEG quantizes again, so errors can accumulate in the pixels as well (image compression). A version system that hashes files therefore sees a change wherever the encoding changes, and a pipeline that hashes the decoded arrays can tell whether the change reached the data the model sees. The example below shows both.
- Annotations are a separate artefact with their own versions. Labels are corrected, guidelines change, and an annotation quality review can flip thousands of labels while not a single image changes. Keeping annotations in their own versioned files, in one of the annotation formats, lets a model record which label version trained it, and lets a pipeline know that a label change needs retraining but no new preprocessing.
The step of building and maintaining a dataset, including its documentation and the rule that every change makes a new version, belongs to dataset curation. Content addressing is one way of making that rule mechanical.
Pipelines that re-run only what changed
A pipeline describes how artefacts are produced from other artefacts: decode and resize the images, train on the result, evaluate the model. Written as a directed acyclic graph of stages, each with declared inputs (data, code, parameters) and outputs, it can be re-run by a tool rather than by hand. DVC’s README calls its pipelines “Makefiles for ML”: a pipeline specifies the steps required to produce a model, with “input dependencies including code, data, commands to run” and the outputs to be saved, and when something changes, DVC runs “only the steps impacted by those changes”. Its quick-start declares the training stage with dvc stage add -n train -d features/ -d train.py -o model.p -M metrics.json python train.py, so the training script itself is listed as a dependency of that stage.
The mechanism, as the example reproduces it, is a cache keyed by the hashes of each stage’s inputs and parameters. Before a stage runs, the key is computed; if the cache already holds an output for that key, the stage is skipped and the stored output used. A change to raw data changes the key of the first stage that reads it; whether later stages re-run depends on whether that stage’s output changed. Two consequences follow. A change that does not reach a stage’s output stops there, so a cosmetic re-encoding of images costs one re-run of decoding, not a retraining. And anything a stage depends on but does not declare is invisible to the cache: an undeclared library upgrade, a configuration file read inside the training code, or a seed taken from the clock gives stale outputs that look current.
Experiment tracking
Experiment tracking records each training run with what went into it and what came out. In MLflow 3.18.0’s terms, a run is one execution of some code, such as one python train.py, and records parameters, metrics, start and end times and artifacts such as model weights and images; an experiment groups runs and models for one task; and the API logs parameters and metrics with functions such as mlflow.log_param and mlflow.log_metric, or automatically for supported libraries with mlflow.autolog(). By default the records go to a local mlruns directory, and a tracking server with a database backend and an artifact store serves them to a team. MLflow 3 adds logged models with their own identifiers, referenced as models:/<model_id>, so that several checkpoints from one run can be logged, compared and loaded, and metrics can be linked to a particular checkpoint and dataset. Datasets are logged with mlflow.log_input as Dataset objects, which the documentation calls metadata tracking objects, carrying a name, an automatically computed digest (a hash identifying the dataset), the source the data came from and an optional schema and profile. DVC’s README describes a lighter, Git-based form of the same idea: experiments tracked in the local Git repository, compared on hyperparameters and metrics, and selected with dvc exp apply.
What a vision run should record follows from the list of things a model depends on: the image manifest hash, the annotation version, the preprocessing parameters (size, resize filter and library, normalization), the code commit, the library versions, the seeds and the hardware, along with the metrics on each evaluation set and slice. Without these, a table of accuracies cannot say why two runs differ.
The model registry
A registry is the list of models that may be deployed, separate from the many runs that produced candidates. MLflow 3.18.0’s documentation describes a registered model with a unique name that holds versions: the first model registered under a name is version 1, and each new one increments the number. Each version is linked to the run, logged model or notebook that produced it, which gives the lineage back to parameters and data. Aliases are mutable named references to a version, such as champion, so that a serving job can load models:/MyModel@champion and a deployment is changed by moving the alias rather than editing the serving code. Tags are key–value labels on models and versions, such as validation_status:pending and validation_status:approved. The registry’s earlier mechanism of fixed stages (Staging, Production and Archived) still appears in one sentence of the overview page, which names Staging and Production, but the workflow page at v3.18.0 states that model stages have been deprecated since MLflow 2.9.0 and will be removed in a future major release. For separate development, staging and production environments, the same page recommends, for mature set-ups, automated workflows that train and register models in each environment, with the machine-learning code promoted across environments through source control and CI/CD, and for simple cases copying a model version from one registered model to another.
The version a registry assigns is a name; the content hash of the model file is its identity. The deploying vision models article stores a checksum with the packaged model and the training run, data and code versions it came from, which is what lets a prediction be traced back through the registry to its lineage.
Continuous integration, delivery and training
The software practices of continuous integration (every change is built and tested automatically) and continuous delivery (every tested build can be released) extend to models in our framing as follows:
- Integration tests the code and the data: unit tests of preprocessing and postprocessing, a parity test of the serving pipeline against a golden set (deploying vision models), checks of new data and labels, and a short training run that must reach a known result.
- Delivery builds a candidate model from versioned inputs, evaluates it, and registers it only when the regression tests of testing and validating vision systems pass; release then follows the canary and rollback procedures of the deployment article.
- Continuous training re-runs the pipeline when new data or labels arrive, on a schedule or when monitoring raises an alarm, followed by re-validation against the original requirement. In a vision system the triggers include a new camera or site and an annotation correction as well as drift.
A pipeline that re-runs only changed stages makes these loops affordable, and a registry with aliases and tags keeps the result of each loop from reaching production until its checks pass.
Lineage
Lineage is the record, for each artefact, of the artefacts and settings it was made from, so that the graph can be followed in both directions: back from a deployed model to the images, annotations, code, environment and parameters that produced it, and forward from a changed input to everything that depends on it. Sculley et al.‘s “plain-old-data type smell” makes the same point for predictions: a prediction should know about the model that produced it and how it should be consumed. When each stage’s cache key is built from its inputs’ hashes, as in DVC’s pipelines and the example below, the keys themselves form the lineage: the model’s record names the hash of the preprocessed array, whose record names the hash of the image manifest.
Reproducibility
Rebuilding a model bit for bit needs the same inputs, the same code and libraries, the same seeds, and the same order of floating-point operations. The last is the hardest, because sums split across threads or devices are added in different orders; the neural network training article describes PyTorch’s deterministic settings and its statement that results are not guaranteed to be reproducible across releases or platforms. A practical position, in our reading, is to require bitwise reproducibility where it is cheap, as in the example below with one thread on one machine, and elsewhere to require that a rebuilt model pass the same tests as the original, which is a statistical rather than a bitwise claim. Seeds belong in the record either way; whether they change the result depends on the algorithm, as the example shows.
How to Apply It
- Put the code, the pipeline definition and the configuration in version control, and review configuration changes like code.
- Version images and annotations by content, with manifests in the repository and the files in a shared store; keep annotations in their own files.
- Write the pipeline as stages with declared inputs, including code, preprocessing parameters, labels and library versions, and let a tool decide what to re-run.
- Log every training run with its inputs’ versions, parameters, seeds, environment and metrics, and link each logged model to its run.
- Register only models that passed the regression tests; deploy through aliases, and record validation status in tags.
- Automate retraining and re-validation, and connect monitoring alarms to the pipeline rather than to a manual procedure.
Example: a content-addressed pipeline for a shape classifier
The script renders 300 grayscale images of disks, squares and triangles (100 of each) at random positions and sizes on noisy backgrounds, and stores them as PNG files with a separate CSV annotation file. Every fifth image is a test image (60 in all, 20 per class). The pipeline has three cached stages after an ingest step. Ingest hashes every file with SHA-256 and the sorted manifest of hashes; it runs every time, as the check for changes, and is not counted as a stage. Preprocess decodes the PNGs and resizes them to with a Pillow filter; train fits a multinomial logistic regression (scikit-learn 1.7.2, ) to the 240 training images’ pixels scaled to , and stores the model as the bytes of its classes, coefficients and intercepts; evaluate reads the model back from those bytes and counts correct test predictions. Each stage’s cache key is the SHA-256 of its name, the hash of its function’s source code, its inputs’ hashes, its parameters and the library versions, and a stage runs only when its key is not already in the cache. The cache is our analogue of DVC’s stage caching, with SHA-256 where DVC records md5 for local files, so the outcomes test the analogue, not DVC.
After a baseline run fills the cache, seven scenarios each start from a copy of it: S0 changes nothing; S1 re-saves one PNG without compression (compress_level=0, where Pillow 12.0.0’s PNG writer passes −1 when no level is given), which changes its bytes but not its pixels; S2 changes one training image’s label from square to triangle; S3 changes the resize filter from bilinear to box; S4 changes the seed passed to the default lbfgs solver; S5 switches to the saga solver; and S6 runs saga with another seed. A final run rebuilds everything from an empty cache. Before any script ran, we wrote down the stages each scenario should execute, from the graph: none for S0, preprocessing only for S1, training and evaluation for S2, all three for S3, training only for S4 (evaluation is keyed on the model’s hash, so it is skipped if the model is unchanged), and training and evaluation for S5 and S6. The script uses NumPy 2.3.5, SciPy 1.16.3, scikit-learn 1.7.2 and Pillow 12.0.0 with one thread, and runs in about one to two seconds.
import os
for v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
os.environ[v] = "1" # one thread: summation order fixed
import hashlib, inspect, io, json
import numpy as np, scipy, sklearn, PIL
from PIL import Image
from sklearn.linear_model import LogisticRegression
ENV = {"numpy": np.__version__, "scipy": scipy.__version__,
"scikit-learn": sklearn.__version__, "Pillow": PIL.__version__}
print("environment:", ENV)
N, SIZE, OUT, CLASSES = 300, 32, 16, ["disk", "square", "triangle"]
FILTERS = {"BILINEAR": Image.Resampling.BILINEAR, "BOX": Image.Resampling.BOX}
def sha(b):
return hashlib.sha256(b).hexdigest()
def canon(obj): # canonical JSON, so equal content gives equal bytes
return json.dumps(obj, sort_keys=True, separators=(",", ":")).encode()
def array_hash(a):
return sha(f"{a.dtype}{a.shape}".encode() + a.tobytes())
# ---- Raw data: rendered shapes stored as PNG files, labels in a CSV annotation file ----
def render(k, rng):
r = rng.uniform(5, 9)
cx, cy = rng.uniform(r + 2, SIZE - r - 2, 2)
yy, xx = np.mgrid[0:SIZE, 0:SIZE] + 0.5
if k == 0:
m = (xx - cx) ** 2 + (yy - cy) ** 2 <= r ** 2
elif k == 1:
m = (np.abs(xx - cx) <= r) & (np.abs(yy - cy) <= r)
else: # apex (cx, cy - r), base corners (cx -/+ r, cy + r)
m = (yy <= cy + r) & (np.abs(xx - cx) <= (yy - (cy - r)) / 2)
fg, bg = rng.uniform(150, 230), rng.uniform(40, 100)
img = np.where(m, fg, bg) + rng.normal(0, 12, m.shape)
return np.clip(img, 0, 255).round().astype(np.uint8), m
def png(a, **kw):
b = io.BytesIO(); Image.fromarray(a).save(b, format="PNG", **kw); return b.getvalue()
rng = np.random.default_rng(46)
labels = np.arange(N) % 3
imgs, masks = zip(*(render(k, rng) for k in labels))
FILES = {f"img_{i:03d}.png": png(a) for i, a in enumerate(imgs)}
def labels_csv(lab):
return ("name,label\n" + "".join(f"img_{i:03d}.png,{CLASSES[k]}\n" for i, k in enumerate(lab))).encode()
LABELS = labels_csv(labels)
def test_mask(n): # split rule "mod5": every fifth image is a test image
return np.arange(n) % 5 == 0
# ---- Stages ----
def preprocess(files, filter, size):
return np.stack([np.asarray(Image.open(io.BytesIO(files[n])).resize((size, size), FILTERS[filter]))
for n in sorted(files)])
def parse_labels(csv):
rows = sorted(line.split(",") for line in csv.decode().splitlines()[1:])
return np.array([CLASSES.index(lab) for _, lab in rows])
def train(X, csv, split, C, max_iter, solver, seed):
y, tr = parse_labels(csv), ~test_mask(len(X))
clf = LogisticRegression(C=C, solver=solver, max_iter=max_iter, random_state=seed)
clf.fit(X[tr].reshape(tr.sum(), -1) / 255.0, y[tr])
artifact = np.concatenate([clf.classes_.astype(np.float64), clf.coef_.ravel(), clf.intercept_])
return artifact.tobytes(), int(clf.n_iter_.max())
def evaluate(artifact, X, csv, split):
w = np.frombuffer(artifact, dtype=np.float64)
k = len(CLASSES); classes, coef, b = w[:k], w[k:-k].reshape(k, -1), w[-k:]
te = test_mask(len(X))
pred = classes[np.argmax(X[te].reshape(te.sum(), -1) / 255.0 @ coef.T + b, axis=1)]
return int((pred == parse_labels(csv)[te]).sum()), int(te.sum())
# ---- Pipeline runner: a stage runs only if its key (code, inputs, params, environment) is new ----
def run(files, csv, p, cache):
executed, lineage, n_iters = [], {}, []
manifest = {name: sha(b) for name, b in sorted(files.items())} # ingest: always re-hashed
images, annot = sha(canon(manifest)), sha(csv)
def stage(name, fn, inputs, params, compute):
key = sha(canon({"stage": name, "code": sha(inspect.getsource(fn).encode()),
"inputs": inputs, "params": params, "env": ENV}))
if key not in cache:
cache[key] = compute(); executed.append(name)
lineage[name] = {"key": key, "inputs": inputs, "params": params, "output": cache[key][1]}
return cache[key]
def do_pre():
X = preprocess(files, p["filter"], OUT); return X, array_hash(X)
X, x_hash = stage("preprocess", preprocess, {"images": images},
{"filter": p["filter"], "size": OUT}, do_pre)
tp = {"split": "mod5", "C": 1.0, "max_iter": 2000, "solver": p["solver"], "seed": p["seed"]}
def do_train():
art, it = train(X, csv, **tp); n_iters.append(it); return art, sha(art)
art, m_hash = stage("train", train, {"preprocessed": x_hash, "labels": annot}, tp, do_train)
def do_eval():
r = evaluate(art, X, csv, "mod5"); return r, sha(canon(r))
(correct, total), _ = stage("evaluate", evaluate,
{"model": m_hash, "preprocessed": x_hash, "labels": annot},
{"split": "mod5"}, do_eval)
return dict(executed=executed, lineage=lineage, n_iters=n_iters, images=images,
x_hash=x_hash, model=m_hash, correct=correct, total=total)
BASE = {"filter": "BILINEAR", "solver": "lbfgs", "seed": 0}
cache = {}
base = run(FILES, LABELS, BASE, cache)
print("baseline executed:", base["executed"], "| n_iter", base["n_iters"])
print("lineage of the baseline model:")
for name, rec in base["lineage"].items():
ins = {k: v[:16] for k, v in rec["inputs"].items()}
print(f" {name:10s} key {rec['key'][:16]} inputs {ins} params {rec['params']} -> {rec['output'][:16]}")
# ---- Data checks ----
border = sum(m[0].any() or m[-1].any() or m[:, 0].any() or m[:, -1].any() for m in masks)
print("C-data: images", len(imgs), "per class", np.bincount(labels), imgs[0].shape, imgs[0].dtype,
"| masks touching border", border, "| test per class", np.bincount(labels[test_mask(N)]))
s1_files = dict(FILES)
s1_files["img_007.png"] = png(np.asarray(Image.open(io.BytesIO(FILES["img_007.png"]))), compress_level=0)
same_px = np.array_equal(np.asarray(Image.open(io.BytesIO(s1_files["img_007.png"]))), imgs[7])
print(f"C-data S1: img_007 bytes {len(FILES['img_007.png'])} -> {len(s1_files['img_007.png'])}, "
f"hash changed {sha(FILES['img_007.png']) != sha(s1_files['img_007.png'])}, pixels identical {same_px}")
s2_lab = labels.copy(); s2_lab[1] = CLASSES.index("triangle")
diff = np.flatnonzero(s2_lab != labels)
print(f"C-data S2: labels differing {len(diff)} at {diff.tolist()}, in training split "
f"{bool((~test_mask(N))[diff].all())}")
# ---- Scenarios, each from a copy of the baseline cache ----
SCEN = {
"S0 rerun unchanged": (FILES, LABELS, BASE),
"S1 one PNG re-encoded": (s1_files, LABELS, BASE),
"S2 one label changed": (FILES, labels_csv(s2_lab), BASE),
"S3 resize filter BOX": (FILES, LABELS, {**BASE, "filter": "BOX"}),
"S4 seed 1 (lbfgs)": (FILES, LABELS, {**BASE, "seed": 1}),
"S5 solver saga, seed 0": (FILES, LABELS, {**BASE, "solver": "saga"}),
"S6 solver saga, seed 1": (FILES, LABELS, {**BASE, "solver": "saga", "seed": 1}),
}
EXPECT = {"S0": [], "S1": ["preprocess"], "S2": ["train", "evaluate"],
"S3": ["preprocess", "train", "evaluate"], "S4": ["train"],
"S5": ["train", "evaluate"], "S6": ["train", "evaluate"]}
res, all_iters = {}, list(base["n_iters"])
print(f"\n{'scenario':24s} {'executed':36s} {'expected?':9s} {'images':7s} {'preproc':7s} "
f"{'model hash':16s} {'model':7s} test accuracy")
for name, (files, csv, p) in SCEN.items():
r = run(files, csv, p, dict(cache)); res[name[:2]] = r; all_iters += r["n_iters"]
same = lambda a, b: "same" if a == b else "changed"
print(f"{name:24s} {str(r['executed']):36s} {str(r['executed'] == EXPECT[name[:2]]):9s} "
f"{same(r['images'], base['images']):7s} {same(r['x_hash'], base['x_hash']):7s} "
f"{r['model'][:16]} {same(r['model'], base['model']):7s} "
f"{r['correct']}/{r['total']} = {r['correct'] / r['total']:.4f}")
print("S6 model vs S5 model:", "same" if res["S6"]["model"] == res["S5"]["model"] else "changed")
cold = run(FILES, LABELS, BASE, {}); all_iters += cold["n_iters"]
print("\ncold rerun executed:", cold["executed"], "| output hashes equal to baseline:",
{s: cold["lineage"][s]["output"] == base["lineage"][s]["output"] for s in base["lineage"]})
# ---- Checks and hypotheses ----
c = base["correct"]
print(f"\nC-conv: n_iter of every executed fit {all_iters}, all < 2000: {all(i < 2000 for i in all_iters)}")
print(f"C-learn: baseline {c}/60, bar 30 <= c <= 57: {30 <= c <= 57}")
def verdict(ok): return "held" if ok else "failed"
B, R = base, res
print("H1", verdict(R["S0"]["executed"] == [] and R["S0"]["model"] == B["model"]))
print("H2", verdict(R["S1"]["images"] != B["images"] and R["S1"]["executed"] == ["preprocess"]
and R["S1"]["x_hash"] == B["x_hash"] and R["S1"]["model"] == B["model"]))
print("H3", verdict(R["S2"]["executed"] == ["train", "evaluate"] and R["S2"]["model"] != B["model"]
and R["S3"]["executed"] == ["preprocess", "train", "evaluate"]
and R["S3"]["x_hash"] != B["x_hash"] and R["S3"]["model"] != B["model"]))
print("H4", verdict(R["S4"]["executed"] == ["train"] and R["S4"]["model"] == B["model"]))
print("H5", verdict(R["S5"]["executed"] == ["train", "evaluate"] and R["S5"]["model"] != B["model"]
and R["S6"]["executed"] == ["train", "evaluate"]
and R["S6"]["model"] != R["S5"]["model"]))
print("H6", verdict(all(cold["lineage"][s]["output"] == B["lineage"][s]["output"] for s in B["lineage"])))
Output:
environment: {'numpy': '2.3.5', 'scipy': '1.16.3', 'scikit-learn': '1.7.2', 'Pillow': '12.0.0'}
baseline executed: ['preprocess', 'train', 'evaluate'] | n_iter [90]
lineage of the baseline model:
preprocess key 7e5d095b76688ed6 inputs {'images': '94e1ed8291e97741'} params {'filter': 'BILINEAR', 'size': 16} -> 1ab0022e67c6d799
train key 81b6b0d46f4d12c6 inputs {'preprocessed': '1ab0022e67c6d799', 'labels': 'a5a64d9aac25da7b'} params {'split': 'mod5', 'C': 1.0, 'max_iter': 2000, 'solver': 'lbfgs', 'seed': 0} -> 9e072c1d9cbeef71
evaluate key 06dcf85f09e38903 inputs {'model': '9e072c1d9cbeef71', 'preprocessed': '1ab0022e67c6d799', 'labels': 'a5a64d9aac25da7b'} params {'split': 'mod5'} -> 82d0a6450c2e0cc8
C-data: images 300 per class [100 100 100] (32, 32) uint8 | masks touching border 0 | test per class [20 20 20]
C-data S1: img_007 bytes 917 -> 1124, hash changed True, pixels identical True
C-data S2: labels differing 1 at [1], in training split True
scenario executed expected? images preproc model hash model test accuracy
S0 rerun unchanged [] True same same 9e072c1d9cbeef71 same 33/60 = 0.5500
S1 one PNG re-encoded ['preprocess'] True changed same 9e072c1d9cbeef71 same 33/60 = 0.5500
S2 one label changed ['train', 'evaluate'] True same same 6c0c1e0c1b534bb0 changed 33/60 = 0.5500
S3 resize filter BOX ['preprocess', 'train', 'evaluate'] True same changed 99cfee5bdec13feb changed 34/60 = 0.5667
S4 seed 1 (lbfgs) ['train'] True same same 9e072c1d9cbeef71 same 33/60 = 0.5500
S5 solver saga, seed 0 ['train', 'evaluate'] True same same 882544512428d93b changed 33/60 = 0.5500
S6 solver saga, seed 1 ['train', 'evaluate'] True same same 0a4b09e9a25eb7ab changed 33/60 = 0.5500
S6 model vs S5 model: changed
cold rerun executed: ['preprocess', 'train', 'evaluate'] | output hashes equal to baseline: {'preprocess': True, 'train': True, 'evaluate': True}
C-conv: n_iter of every executed fit [90, 78, 88, 90, 208, 211, 90], all < 2000: True
C-learn: baseline 33/60, bar 30 <= c <= 57: True
H1 held
H2 held
H3 held
H4 held
H5 held
H6 held
The lineage lines show the keys acting as records: the model’s training record names the hash of the preprocessed array and of the annotation file, and the preprocessing record names the hash of the image manifest, so the model can be traced back to every file and label it came from. In every scenario the executed stages matched the set written down beforehand.
Several outcomes follow from how the cache key is built, and show the mechanism working as designed; they are checks of this implementation rather than findings about data. H1, that an unchanged rerun executes nothing and reproduces the model hash, held. H2 held: re-saving img_007.png without compression grew it from 917 to 1124 bytes and changed its hash while leaving its pixels identical, so the image manifest changed and preprocessing re-ran, but its output hash did not change, and training and evaluation were skipped. H6 held: the rebuild from an empty cache reproduced the preprocessed array, the model and the metrics bit for bit, on one thread on one machine with these library versions; nothing is claimed here about other machines or versions.
H3 held: the label change (S2) re-ran training and evaluation and the filter change (S3) re-ran all three stages, and both changed the model hash, S3 the preprocessed hash as well. In S2 the model hash changed while the test accuracy stayed at 33/60.
H4 held. The seed is part of the training key, so S4 re-ran training, but the model hash came out unchanged and evaluation was skipped. That the model would be unchanged rests on scikit-learn 1.7.2’s docstring, which says random_state is “Used when solver == ‘sag’, ‘saga’ or ‘liblinear’”, and on its source, in which the lbfgs path starts from zero coefficients and does not use random_state; the printed hash confirms it for this run. H5 held: the saga solver, which shuffles the data with the seed, gave different model hashes for seeds 0 and 1, both with 33/60 correct. A content-addressed cache is therefore conservative: putting the seed in the key costs a retraining that, for lbfgs, produces the same bytes, and a changed hash says that the model’s bytes differ, not that its behaviour on the task does. Whether a changed model is acceptable is decided by tests, not by hashes.
The data checks held (300 images, 100 per class, uint8, no shape touching the border, 20 test images per class; one label changed in S2, on a training image), as did the convergence check (at most 211 of 2000 iterations) and the learnability check (33 of 60 test images correct, against a bar of 30 to 57 and 20 expected by chance). The classifier is weak, and the example does not claim a good one: it needs only a model whose training is real enough for its bytes to depend on its inputs. Two runs of the script printed identical output.
How the example was planned and changed. The set-up, the expected stage sets, the hypotheses and the checks were written before any script ran. A planning script then ran once, printing the data checks, the test accuracies and the iteration counts of the fits, but comparing no hashes and no seeds. The bar of the learnability check, 30 to 57 correct test images of 60, was first written after that script printed 33/60 for the baseline, so it is a bar set after a planning script. Before the plan was frozen and before any run, the script was edited to include the cold rebuild’s fit in the convergence check, as the plan required for every fit. After the pilot, one print-format change was made with the editor’s approval before the first run: the “executed” column was widened from 32 to 36 characters, because the S3 row overflowed it. Nothing else changed.
Variants
- Git-centred tools. DVC keeps metadata files in the Git repository and the data in a cache and remote store, defines pipelines in the repository, and tracks experiments in the local Git repository with “no servers needed” (its README). Its README compares the data side to Git LFS without a server.
- Tracking servers and registries. MLflow logs runs to a local directory or to a tracking server with a database backend and an artifact store, and keeps registered models with versions, aliases and tags. Its registry page describes managed variants, such as Databricks’ integration with Unity Catalog, that add access control and cross-workspace sharing.
- Hand-written manifests and caches, as in the example: hashes, manifests and keyed outputs in a few dozen lines, workable for one project and one machine, without remote storage, sharing or a user interface.
- Retraining regimes. Models retrained on a schedule, retrained on triggers from monitoring or new labels, or, as Sculley et al. mention, adapted online; each needs the versioned inputs and the tests above, and online adaptation also needs its updates recorded.
Best Practices
- Hash pixels as well as files. The file hash says that something changed; a hash of the decoded, preprocessed array says whether the change reached the model’s input.
- Declare every input of every stage, including the training code, the preprocessing library and its version, the resize filter, the label file and the seed. An input that is not in the key cannot invalidate the cache.
- Keep annotations separate from images, versioned on their own, so that a label correction retrains without re-preprocessing and every model records its label version.
- Deploy through aliases and record validation in tags, and move an alias only after the regression tests pass; MLflow’s stages are deprecated at 3.18.0.
- Record lineage with each deployed model and each logged prediction, so that a wrong prediction can be traced to the model version and from there to its data, labels and code.
- Remove dead experimental branches and keep configuration under review, as Sculley et al. recommend, so that the configuration that trained a model can be read and diffed.
Pitfalls
- Undeclared dependencies, such as a configuration file or environment variable read inside a stage, or a library upgraded in place, which leave cached outputs that look current but are stale.
- Treating file hashes as content in the visual sense. Re-encoding a PNG triggers re-runs that change nothing downstream, as S1 shows; re-encoding a JPEG from its decoded pixels quantizes again and can change the pixels, and a model trained on the re-encoded copy is then a different model.
- A test set that drifts with the versions. New images added to training can include copies of test images or frames from the same videos; the split rule belongs in the pipeline and dataset curation describes deduplication against evaluation sets.
- Expecting bitwise reproducibility everywhere. Changing threads, hardware or library versions changes floating-point sums; across such changes the claim to check is that the rebuilt model passes the same tests.
- Retraining without re-setting what depended on the old model: decision thresholds, as the vision system design article reports from Sculley et al., and undeclared consumers of its outputs.
- Deploying “the latest” model instead of a named, tested version, which lets an untested registration reach production.
Evidence and Impact
In our reading, the case for these practices rests mainly on practitioners’ reports rather than controlled studies. Sculley et al. (2015) argue from experience with production systems and say themselves that technical debt is a useful metaphor without a strict metric that can be tracked over time; they offer questions instead, such as how precisely the impact of a change can be measured and what the transitive closure of all data dependencies is. The neighbouring articles report the related empirical findings: the deploying vision models article reports Breck et al.’s (2017) interviews with 36 teams about production-readiness tests, and the vision system design article reports Amershi et al.‘s (2019) study at Microsoft, which found discovering, managing and versioning data much more complex than the corresponding work in other software engineering. The tools’ own documentation describes what they do rather than measuring its effect. The example shows the mechanism itself: with content hashes as keys, the work a change causes, and whether a retrained model is bit-identical, become things that can be printed and checked.
Related
- Designing a Computer Vision System
How a working vision system is designed from its requirements, through the choice of sensor, optics, lighting, compute and model, to the data it is trained on, its validation in the field and its maintenance once deployed, with the requirement arithmetic for a measurement station worked through in code.
- Testing and Validating Vision Systems
How a vision system is shown to meet its requirements before and after it goes into service, from the test plan and held-out tests to slice analysis, behavioural and metamorphic tests, stress tests at the requirement's limits, regression tests and the field acceptance test, with an inspection example in which the aggregate passes, one slice fails and a label-free test points to it.
- Dataset Curation
How a training set for a vision model is assembled and maintained, from defining what it should represent through filtering, deduplication, balancing, splitting, documentation and versioning, and how data pruning ranks examples so that a smaller set trains as well as a larger one.
- Active Learning
How a model chooses which unlabelled images to send for labelling next, by uncertainty, disagreement, coverage or typicality, and why the best choice depends on how many labels have been bought.
- Annotation Quality and Label Noise
How to measure and repair the quality of human labels for vision datasets; raw and chance-corrected agreement (Cohen's kappa, Scott's pi, Fleiss' kappa, Krippendorff's alpha) and why agreement is not accuracy, latent-truth aggregation with Dawid–Skene EM against the majority vote, finding wrong labels with confident learning, and what label noise does to training and evaluation.
- Training a Neural Network
The practical workflow for fitting a neural network to labeled images, from splitting the data and normalizing pixels through the training loop, learning curves and early stopping to hyperparameter search, debugging checks, reproducibility and memory.
References
- Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F. & Dennison, D. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28 (NIPS).
- DVC (Data Version Control) 3.67.1 (31 March 2026): README.rst and dvc/output.py, treeverse/dvc at tag 3.67.1.
- MLflow 3.18.0 documentation (4 November 2026): MLflow Tracking, ML Dataset Tracking, ML Model Registry and Model Registry Workflows (docs/docs/classic-ml), mlflow/mlflow at tag v3.18.0.
- scikit-learn 1.7.2 source: LogisticRegression (sklearn/linear_model/_logistic.py), installed package.
- Pillow 12.0.0 source: PNG writer (PIL/PngImagePlugin.py), installed package.