Automating the path from a commit to a deployed model, with gates that can actually stop a bad one — not automation for its own sake, but a system where "commit" doesn't mean "deployed," and a real evaluation gate stands between the two.
The world moved and the model did not. A model is a snapshot of relationships learned from historical data — when the world underlying those relationships changes, the model's assumptions quietly stop holding, and performance degrades in a way no error message will ever announce.
The model did not break. The upstream team renamed a column, and the model kept running — silently feeding a now-empty (or misaligned) feature into every prediction, with no error, no crash, and no obvious symptom until someone notices the metrics have quietly degraded.
Which exact rows produced the model currently serving traffic? For most teams, the honest answer is "we don't know" — the code is in Git, but the data that trained the model has moved on, been updated, or been silently overwritten since. A model is a function of code and data, and versioning only the code reproduces nothing.
The best result ever achieved on a project, and no memory of exactly how it was produced — which hyperparameters, which data version, which code state. Without disciplined tracking, this happens constantly, and it's completely avoidable: log everything about every run, or the run effectively did not happen.
The same feature, computed two different ways in two different places, gives two different answers — training pipeline code and serving pipeline code drift apart, quietly, until a model that scored well offline behaves differently in production for no reason anyone can immediately find.
The model scores 0.94 in the notebook, and nobody can use it. This is the single most common gap in applied machine learning — not a modelling problem, but a systems problem, and this section exists entirely to close it.
Making the model cheap enough to serve, without quietly making it wrong. Every optimisation technique on this page trades accuracy, latency, or memory for one another — measure all three, before and after, or the "optimisation" is a guess.
The file called modelfinalv2actuallyfinal.pkl — every team has one, and it's a symptom, not a joke. A model artefact is only genuinely useful with the preprocessing, schema, and metadata needed to call it correctly; package those together properly, or ship a bug the moment they drift apart.
Models fail silently. They keep returning confident, plausible-looking answers that are simply wrong, with no crash, no error, nothing that would trip a standard software alert — the only way to catch this is to watch the inputs and predictions continuously, because the ground truth needed to know an answer was wrong often arrives late, or never.
One aggregate number hides every failure that matters. A model at 94% accuracy can be failing badly on a 5% subgroup, systematically wrong in a specific, predictable direction, or actively worse than the model it's replacing on the exact cases that matter most — none of which a single headline metric will ever reveal.
The offline metric improved, and the business metric did not. This gap is common enough to expect it as the default outcome, not a surprising exception — offline metrics are proxies, and only a genuine online experiment measures the thing actually being optimised for.
Same code, same data, different result. This should be alarming, and it usually traces to one of a small, well-understood set of non-determinism sources — reproducibility is layered, and the right response is deciding which layer actually matters before paying for the strictest possible guarantee.
The ways a working model causes harm, and the ones an engineer can actually engineer against. Fairness, privacy, and security are design constraints with concrete, testable properties — not a review step bolted on at the end, and not something achieved by good intentions alone.
Choosing between an overnight batch job and a 50-millisecond synchronous API is not a minor implementation detail — it changes almost every other decision downstream, from infrastructure to model architecture to error handling. The serving pattern should be chosen deliberately, before the model is designed, not bolted on after.
The training bill arrives, and most of it bought idle GPU time — a genuinely common outcome, and an entirely preventable one. Utilisation, not hourly rate, decides training cost: a half-idle expensive GPU beats neither a well-fed cheap one nor, often, a smaller model trained more efficiently.