Version the serving contract
Give each candidate an immutable artifact reference and record the runtime image digest, model checksum, tokenizer version and inference configuration beside it. A friendly model alias can point at the active version, but it should never be the only identifier in logs or deployment history.
This makes a regression diagnosable. If output quality or latency changes, the team can tell whether the model, runtime, prompt wrapper or hardware profile changed, instead of comparing two mutable tags called latest.
Shift a small amount of traffic first
Bring up the candidate without sending user traffic, run a smoke suite against its health and inference endpoints, then direct a small canary slice to it. A service mesh, gateway or application-level router can make the split; the important part is that the routing decision is observable and reversible.
Warm the model before measuring. Cold-start downloads, kernel compilation and cache fill are real production costs, but they should not be confused with steady-state latency. Record both separately.
Gate on service and product signals
Use a short list of bounded signals: error rate, queue depth, time-to-first-token, tail latency and a task-specific quality check. Compare the canary with the current version over a representative sample. A generic HTTP health check cannot detect a semantically broken model response.
If a gate fails, stop the traffic increase and route back to the stable version automatically. Keep the previous deployment available until the new one has passed its observation window; deleting it immediately saves little and removes the simplest recovery path.
Keep the release auditable
Attach the rollout decision to the commit or release record: candidate digest, traffic percentages, gate thresholds, observed values and operator override. This is valuable during an incident and during the later conversation about whether a model is ready for a wider audience.
Canary is not a percentage. It is a bounded experiment with a control, a decision rule and a clear rollback action.