The MLOps Playbook Behind MSMEMate: How We Ship Honest AI

Why lifecycle matters more than launch accuracy

A model's accuracy on the day it ships is a snapshot. The world underneath it moves: new kinds of businesses register, vocabulary shifts, upstream services change behaviour. For AI that touches government workflows — deciding which ONDC category a business belongs to, or which seller apps suit it — the real question is not 'how accurate was it at launch?' but 'how will you know when it stops being accurate, and what happens then?'

That question is what MLOps answers. Here is the lifecycle we run, end to end: data with lineage → training with a locked evaluation protocol → versioned, stamped artifacts → gated deployment → drift monitoring → human feedback → retraining. Each stage below is live in production today.

Data: every row knows where it came from

VargBot v2 trains on 33,500 labelled examples from four sources: real e-commerce product listings, real product names from Andhra Pradesh's MEPMA self-help-group sellers, business descriptions derived from real Udyog registration data, and — only for the six ONDC domains nothing else covered — template-generated text built from the official taxonomy's own category names.

Every row carries a source column. The corpus builder is a deterministic, seeded script, so the exact dataset can be rebuilt from its inputs on any machine. This is data lineage in its simplest useful form: when a prediction looks wrong, we can trace what kind of data taught the model that behaviour.

Evaluation: report the number that makes you look worse

The protocol is standard and locked: stratified 80/10/10 split, hyperparameters tuned only on validation, 5-fold cross-validation for stability, and a test set touched exactly once. VargBot v2 scores 98.9% held-out accuracy with cross-validation at 0.984 ± 0.001.

But the more important number is the one we publish next to it: on real product text alone — excluding the template-generated rows whose near-duplicates appear on both sides of the split — accuracy is 98.5% and macro-F1 drops from 0.987 to 0.957. The evaluation report carries a written honesty note explaining exactly why the headline number is optimistic. If your eval report only contains numbers that flatter the model, it is marketing, not evaluation.

Confidence you can act on: the calibrated gate

The trained model only answers when its confidence clears a gate; below it, the case routes to a language model that handles Indic scripts and unusual text, and ultimately to a human reviewer. In v2 the gate stopped being a guess: we compute a full threshold-versus-precision table on validation data, and the shipped gate (0.55) corresponds to a measured coverage/precision trade-off. Saying 'the model is 95% confident' only means something if someone checked what 95% actually delivers.

Versioning and honest stamps

Model artifacts are versioned (v1 stays deployable for instant rollback via one environment variable), the training library version is pinned identically in training and serving, and — the part we consider non-negotiable — every stored prediction carries the exact engine that produced it: the trained model, the language model, or the keyword fallback. No result ever wears a label it did not earn. That per-prediction lineage is what makes everything downstream possible.

• Reproducible pipeline: dvc repro re-runs corpus build → training → artifact deployment as a declared DAG with dependency tracking.
• Model card: a published document covering architecture, data, metrics, limitations and ethics (docs/MODEL_CARD_VARGBOT.md).
• CI model gate: the build fails if the shipped artifact loses domain coverage, regresses on canonical Hinglish/English inputs, or stops deferring on ambiguous text.

Monitoring: the model watches itself, officers watch the model

The NSIC dashboard includes a Model Health monitor built entirely from data the platform already records: weekly confidence trends (average and 25th percentile), the mix of engines answering live traffic, how often officers reject registrations or override the AI's recommendations, and per-domain live confidence compared against the frozen evaluation baseline from training day.

Thresholds turn the card red when the fallback share or override rate climbs past configured limits — a plain-language signal that says 'time to retrain', backed by the exact evidence. Drift detection is not an enterprise luxury; ours is a read-only analytics layer over records an honest system was already keeping.

The feedback flywheel: humans close the loop

Every officer decision — approving a registration, rejecting one, overriding a recommended seller app — is recorded anyway for auditability. A dedicated export turns those decisions into weak-supervision retraining data, clearly labelled with what an approval does and does not confirm. Monitoring tells us when to retrain; the feedback export supplies human-vetted data to retrain with; the same locked evaluation protocol decides whether the new model actually earns its way to production. That loop — not any single model — is the product.

What about reinforcement learning? For a supervised classifier, true RL does not apply — but the feedback flywheel is its philosophical cousin: the system learns from human judgment over time. Genuine RL (contextual bandits over match acceptance) is a plausible future for the matching engine, once live acceptance data accumulates.

What we deliberately do not do (yet)

Honest MLOps also means not cosplaying at scale. We skip experiment-tracking servers (two model versions live comfortably in versioned JSON reports), canary infrastructure (traffic does not justify it; env-var rollback covers us), and feature stores. Each becomes worth adopting at a defined trigger — more models, more traffic, more people — and the DVC remote for large artifacts is the first upgrade queued.