MLOps when the model touches patients | What changes when ML is diagnostic

Most of what gets written about MLOps assumes a particular kind of product. You ship a model, you watch a metric, and if the metric drops you roll back. You run two variants in production and let traffic decide. If the model is wrong, someone sees a bad recommendation and moves on.

I’ve spent several years working on the other kind, in a research group building diagnostic tools for histological and cytological images — the slides a pathologist looks at to decide whether a tumour is malignant. Almost none of the above transfers cleanly. You cannot A/B test a cancer diagnosis. “Roll back and see” is not a strategy when the output has already gone into someone’s chart.

This post is about what actually changes.

The problem isn’t the model

There’s a statistic we cited in our IDDM 2022 paper that I keep coming back to: of all projects using artificial intelligence, only around 22 percent successfully got as far as an implemented process for actually using their machine learning models.

Not 22 percent achieved good accuracy. Twenty-two percent got to the point of using the thing.

That gap is the whole subject. In a research group the failure mode is extremely recognisable: someone trains an excellent model in a notebook, it produces a number that goes in a paper, and eighteen months later nobody — including the author — can reproduce the result, because the environment was a laptop, the data was a folder, and the preprocessing was six cells that got edited in place.

The point of MLOps here isn’t velocity. It’s that a result you can’t reproduce isn’t a result.

The dataset is the hard part, and it’s small

Standard MLOps tooling quietly assumes data is abundant and cheap to regenerate. Biomedical imaging inverts both assumptions.

Acquiring the images is slow and expensive. They come from real patients, via a physical process involving a microscope and a person. When our group built a database of precancerous and cancerous breast tissue back in 2019, the result — BPCI2100 — was 2,100 images. That is a large dataset by the standards of this field, and it would be a rounding error in any natural-image benchmark.

This changes the engineering in concrete ways:

• Every sample matters, so every split matters. With 2,100 images, the difference between a good and a bad train/test split is larger than the difference between two architectures. The split has to be a versioned artefact, not something a script decides at runtime with an unseeded shuffle.
• You cannot just collect more. The usual answer to a data problem — get more data — requires ethics approval and months.
• The labels are themselves uncertain. This is the part engineers coming from web backends find hardest. Your ground truth is a human judgement, and humans disagree. In a 2025 paper on breast cancer subtypes the motivating problem is precisely this: scoring immunohistochemical biomarkers varies between individual pathologists, and part of the value of automating it is reducing that variability. But it also means your test-set accuracy is measured against a noisy oracle.

If you train on synthetic data, the generator is part of your pipeline

Because real images are scarce, a lot of our work has gone into generating artificial ones — GANs for biomedical image synthesis, and architecture search to find generator designs that work for this domain rather than for photographs.

This solves a data problem and creates a provenance problem.

The moment a classifier is trained on generated images, the generator becomes part of the classifier’s lineage. To reproduce the classifier you need the generator weights, the generator’s own training data, the sampling procedure, and the seed. If you version the classifier but not the generator, you have versioned the last link of the chain and lost the rest.

It also raises a question that has no equivalent in ordinary MLOps: is the synthetic data any good? “The loss went down” doesn’t answer it, and neither does eyeballing a grid of samples. This is awkward enough that it became its own paper on a combined quality metric for synthesized biomedical images. Treat synthetic-data quality as a pipeline gate with a number attached, not a vibe.

The environment is the bottleneck

The most practical finding from the 2023 segmentation paper is unglamorous and, I think, correct: the pipeline exists so that researchers can focus on model development rather than on setting up the environment.

That sounds like a platitude until you watch how the time is actually spent. Deep learning on medical images means CUDA versions, a specific PyTorch build, image-processing libraries with native dependencies, and datasets too large to sit in git. A new person joining the project loses their first week to environment setup. A paper from two years ago can’t be rerun because the library versions moved.

Docker fixes the first problem and half of the second. Pinning the whole stack — not just Python packages but the CUDA base image — is the difference between a reproducible result and a story about one. If you already containerise backend services, this is the part of your existing instinct that transfers directly.

For the segmentation work, the shape that came out of it was roughly:

data versioning        →  images + masks + the split, addressed by content hash
containerised training →  pinned CUDA base image, pinned framework versions
CI/CD                  →  build, run the pipeline on a fixed subset, publish artefacts
model registry         →  weights tagged with data version + code commit + metrics
inference service      →  the same container, serving the segmentation

Nothing exotic. U-Net for the segmentation itself, GitHub Actions to drive delivery. The value isn’t in any individual box — it’s that the arrows are automated, so the chain from a dataset to a published number is intact and inspectable.

What transfers from backend work, and what misleads

Coming from years of building Node.js services, some instincts helped and some actively hurt.

Transfers well:

• Containerising everything, and pinning versions like you mean it
• CI as the definition of “it builds”, rather than “it worked on my machine”
• Treating configuration as data, not as edits to a script
• Structured logging — knowing which run produced which artefact

Misleads:

• Ship fast, iterate in production. There is no production traffic to learn from, and the cost of a wrong answer isn’t a bad recommendation.
• Monitor a metric and alert on regressions. You often don’t get ground truth back. Nobody tells the model, weeks later, what the biopsy actually showed.
• Immutable inputs. Backend engineers assume the training data is a fixed thing you point at. In practice a pathologist relabels a batch because the earlier annotation was wrong, and your “fixed” dataset changed underneath a model you’ve already published a number for.
• The model is the deliverable. It isn’t. The reproducible path from data to result is the deliverable; the weights are a by-product.

That last one took me longest to internalise, and it’s the cleanest summary of the difference. In a web service the artefact is the running thing. Here the artefact is the argument — the model is only evidence for it, and evidence you can’t reconstruct isn’t worth much.

Caveat

Everything above is from a research setting: building diagnostic tools and publishing results, not shipping a regulated medical device. Clinical deployment brings a whole regulatory apparatus — approval regimes, audit trails, post-market surveillance — that I haven’t worked in and won’t pretend to advise on. The reproducibility argument is the part I can speak to, and it’s a floor, not a ceiling.

If you want the actual papers, they’re on the research page. The two MLOps ones are open access on CEUR: biomedical image classification (IDDM 2022) and automatic segmentation (IDDM 2023).