CI/CD for Machine Learning: Tests, Quality Gates and Releases

At a good mithai shop before Diwali, nothing reaches the counter without the head halwai tasting it. A new batch of kaju katli is checked for sweetness, thickness and texture against the last good batch. If it is worse, it goes back, however nice it looks.

The check happens every batch, not once a season. And the counter staff never have to judge quality themselves; the taster's rule decides.

CI/CD for ML is that tasting rule, automated. Every change to code, data or settings produces a candidate model, and a fixed set of checks decides whether it reaches customers.

What is CI/CD for machine learning?

In the analogy, the new batch is a candidate model, the last good batch is the champion in production, the halwai's tasting is the quality gate, and sending it back is a failed pipeline that blocks the merge.

CI/CD for machine learning is the automation of building, testing and deploying ML systems, where continuous integration validates code, data and model quality on every change, and continuous delivery releases only the models that pass an agreed quality gate.

ML adds a third letter pair: CT, continuous training, which retrains models automatically when new data arrives or monitoring detects drift. This final lesson of the MLOps series wires together everything from what MLOps is onwards for Chappan Express, the fictional Indore delivery company, and its eta-model.

Code changes and drift alerts enter the same pipeline. Only the gate can promote a model.

CI vs CD vs CT in ML

What should you test in an ML pipeline?

Ordinary software tests check that code does what it says. ML needs that plus checks on data and on learned behaviour.

1. Unit tests for feature code. Units, time zones, defaults. These are fast and catch train/serve skew from the pipelines lesson.
2. Data tests. Schema, null rates, ranges and row counts on the training snapshot.
3. Model quality tests. Holdout MAE against the baseline and the champion.
4. Slice tests. Quality on important groups such as rainy orders or one zone.
5. Behavioural tests. Directional ("more distance, more minutes") and invariance ("order ID does not change the ETA") expectations.
6. Serving tests. Latency, memory, and that the API returns valid JSON for good and bad inputs.

Unit tests for feature code

These look like any pytest file. The block runs them with a tiny runner so you can see the output in the browser; in a real repository you would put them in tests/test_features.py and run pytest.

def build_features(order):
    """Feature code shared by training and serving (from the pipelines lesson)."""
    hour_ist = (order["hour_utc"] + 5.5) % 24
    return {"distance_km": order["distance_m"] / 1000,
            "is_peak": int(int(hour_ist) in (13, 14, 20, 21)),
            "is_raining": int(bool(order.get("rain_mm", 0) > 0.5))}

def test_distance_in_km():
    assert build_features({"distance_m": 3200, "hour_utc": 9})["distance_km"] == 3.2

def test_peak_hour_uses_ist():
    assert build_features({"distance_m": 1, "hour_utc": 14.5})["is_peak"] == 1  # 8 pm IST

def test_missing_rain_defaults_to_dry():
    assert build_features({"distance_m": 1, "hour_utc": 3})["is_raining"] == 0

def test_drizzle_is_not_rain():
    assert build_features({"distance_m": 1, "hour_utc": 3, "rain_mm": 0.2})["is_raining"] == 0

tests = [f for name, f in dict(globals()).items() if name.startswith("test_")]
passed = 0
for t in tests:
    try:
        t(); passed += 1; print("PASSED", t.__name__)
    except AssertionError:
        print("FAILED", t.__name__)
print(f"\n{passed} passed, {len(tests) - passed} failed")

Each test pins down one decision that once caused a bug somewhere: metres versus kilometres, UTC versus IST, a missing field, a threshold. They run in milliseconds, so they run on every push.

Building a quality gate for a candidate model

The gate is the most important part of ML CI. It answers one question: should this model replace the one customers use today?

In this simulation, a pull request proposes switching eta-model from linear regression to gradient boosting. The gate trains the candidate, scores it on a fixed holdout set and runs six checks.

import time
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(30)
def orders(n):
    X = np.column_stack([rng.uniform(0.5, 9, n), rng.integers(8, 25, n), rng.integers(0, 2, n)])
    return X, 6 + 3.2 * X[:, 0] + X[:, 1] + 5 * X[:, 2] + 0.4 * X[:, 0] * X[:, 2] + rng.normal(0, 3, n)
X_tr, y_tr = orders(2500)
X_te, y_te = orders(800)                                    # fixed holdout set

champion = LinearRegression().fit(X_tr, y_tr)               # live eta-model:14
candidate = GradientBoostingRegressor(random_state=0).fit(X_tr, y_tr)  # PR's model
mae = lambda m, X, y: float(np.abs(y - m.predict(X)).mean())
rain = X_te[:, 2] == 1
t0 = time.perf_counter(); candidate.predict(X_te[:1]); ms = (time.perf_counter() - t0) * 1000

checks = {
    "schema: 3 numeric columns, no NaN": X_te.shape[1] == 3 and not np.isnan(X_te).any(),
    "beats baseline MAE (5.0 min)": mae(candidate, X_te, y_te) < 5.0,
    "not worse than champion": mae(candidate, X_te, y_te) <= mae(champion, X_te, y_te) * 1.02,
    "rainy slice MAE under 4.5": mae(candidate, X_te[rain], y_te[rain]) < 4.5,
    "longer distance -> longer ETA": candidate.predict([[8, 15, 0]])[0] > candidate.predict([[2, 15, 0]])[0],
    "single prediction under 50 ms": ms < 50,
}
print(f"champion MAE {mae(champion, X_te, y_te):.2f} | candidate MAE {mae(candidate, X_te, y_te):.2f}")
for name, ok in checks.items():
    print(("PASS " if ok else "FAIL ") + name)
print("\nGATE:", "promote candidate" if all(checks.values()) else "block merge")

The candidate passes almost everything, including the baseline, the rainy slice and the directional check. It still fails one rule: it is slightly worse than the simpler champion on the same holdout. The gate blocks the merge, and nobody had to argue about it in a meeting.

That is the point of agreeing the rules in advance. A fancier algorithm, a bigger feature set or a longer training run earns its place only by measurable improvement. The baseline and target values come from the very first decisions in the ML lifecycle.

A GitHub Actions workflow for ML

GitHub Actions is GitHub's built-in CI/CD service. Workflows are YAML files in .github/workflows/. This one runs tests and the gate on every pull request, and on main builds and pushes the image.

# .github/workflows/eta-model.yml
name: eta-model CI/CD

on:
  pull_request:
    paths: ["src/**", "tests/**", "params.yaml", "dvc.lock"]
  push:
    branches: [main]

jobs:
  test-and-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - name: Unit and data tests
        run: pytest tests/ -q
      - name: Fetch versioned data
        run: dvc pull data/train.parquet data/holdout.parquet
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      - name: Train candidate and run quality gate
        run: |
          python src/train.py --out models/candidate.joblib
          python src/gate.py --candidate models/candidate.joblib --champion-alias champion
        env:
          MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_TRACKING_URI }}

  build-and-release:
    needs: test-and-gate
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - name: Build and push image
        run: |
          echo "${{ secrets.REGISTRY_TOKEN }}" | docker login registry.example.com -u ci --password-stdin
          docker build -t registry.example.com/chappan/eta-api:${{ github.sha }} .
          docker push registry.example.com/chappan/eta-api:${{ github.sha }}
      - name: Start 10% canary
        run: ./deploy/canary.sh ${{ github.sha }}

A few details are worth noticing:

• paths filter. Editing the README does not retrain a model.
• Secrets. Credentials come from repository secrets, never from the code.
• needs and if. Release runs only after the gate passes, and only on main.
• Image tag is the commit SHA, so every running container traces back to one commit, one data version and one model.

Action major versions change over time; check the GitHub Actions documentation and each action's release page when you copy a workflow. If you are new to the shell commands in the run: steps, they are the same ones from the Docker lesson.

Continuous delivery: canary, promote, roll back

Passing the gate on a holdout set is necessary, not sufficient. Live traffic is the real test. Chappan Express's delivery script follows these steps:

1. Deploy eta-api with the new image to a canary Deployment that takes 10% of traffic.
2. Watch latency, error rate and, once deliveries finish, live MAE for both versions for 24 to 48 hours.
3. If the canary is at least as good, move the MLflow alias champion to the new version and roll it out to 100%.
4. If not, scale the canary to zero. Rollback is one command because the old image and model version still exist.

# Promote after a healthy canary
kubectl set image deployment/eta-api eta-api=registry.example.com/chappan/eta-api:3f9c1a7
kubectl rollout status deployment/eta-api

# Something wrong? Go back to the previous ReplicaSet
kubectl rollout undo deployment/eta-api

Continuous training: closing the loop

The last piece connects monitoring back to training. When the nightly job sees PSI above 0.25 on a key feature, or rolling MAE above target, it starts the training pipeline. The new candidate goes through exactly the same gate as a human's pull request.

That is the full MLOps loop from the first lesson, now automated: data, train, evaluate, deploy, monitor, retrain. Generative AI systems run the same loop with different tests, covered next in LLMOps explained.

Common mistakes in ML CI/CD

• Testing only code. Most ML failures are data or model failures.
• Comparing only to a baseline. A candidate can beat the baseline and still be worse than production.
• Random holdout sets per run. Scores then move with noise, and the gate flips between pass and fail.
• Automatic promotion without a canary. Holdout data never captures everything about live traffic.
• Retraining on a timer without a gate. A broken data day becomes a broken model in production.
• Slow pipelines. If CI takes two hours, people skip it. Sample data for PR checks; run full training on main.

If you would like to build an end-to-end MLOps project with feedback, explore the generative AI track that Debugshala runs in Indore. For interview preparation, prepare for placement coding rounds on Debugshala Test.

Key takeaways

• ML CI/CD tests code, data and models, and adds continuous training to classic CI/CD.
• A quality gate with agreed rules decides promotion: baseline, champion, slices, behaviour and latency.
• Unit tests on feature code catch skew; behavioural tests catch silly models.
• GitHub Actions can run tests, pull DVC data, train, gate, build and push on every change.
• Release through a canary, promote by moving the registry alias, and keep rollback to one command.