ML Reproducibility and Versioning: Data, Code, Seeds, Models

Your grandmother's dal tastes the same every time. Ask her for the recipe and she says "a little of this, some of that". You try it and it tastes nothing like hers.

The missing pieces were never in the recipe: which brand of dal, which pot, how hot her stove runs, how long she actually waited. She carries them in her head.

Machine learning models have the same problem. The training script is the written recipe. The data snapshot, library versions and random seed are the unwritten parts. Unless you record them, nobody can cook the same model twice, including you next month.

What is reproducibility in machine learning?

In the kitchen analogy, the recipe is your code, the ingredients are the data, the quantities are the hyperparameters, the stove and pot are the software environment, and the finished dish is the model.

Reproducibility in ML is the ability to recreate a trained model and its results exactly, by re-running recorded code on a recorded data version with recorded settings in a recorded environment.

Versioning is the practice that makes it possible: giving each of those inputs an identifier you can go back to.

At Chappan Express, the fictional Indore delivery company in this course, the question comes up the moment a customer complaint arrives: "Why did the app promise 18 minutes for an order that took 40?"

To answer, you must load the exact eta-model version that was live that evening and the exact data it was trained on. The previous lesson on the ML lifecycle said each stage should leave an artefact. This lesson makes those artefacts traceable.

What should you version in an ML project?

Five things decide a model's behaviour. Change any one and you have a different model.

A model version is really a bundle of five versions. The run ID is the key that ties them together.

The rest of this lesson takes each row in turn with code you can run.

How do you version data? Content hashes

Git is built for small text files. A 2 GB orders table does not belong in it. The simplest workable idea is a content hash: a short fingerprint computed from the bytes of the data. If even one cell changes, the fingerprint changes.

The block hashes a small orders table, then fixes one wrong delivery time and hashes again. It also shows a subtle trap: shuffling rows changes the bytes, so you should sort into a canonical order before hashing.

import hashlib
import pandas as pd

def data_hash(df):
    canon = df.sort_values(list(df.columns)).reset_index(drop=True)
    raw = canon.to_csv(index=False).encode("utf-8")
    return hashlib.sha256(raw).hexdigest()[:12]

orders = pd.DataFrame({
    "order_id": [101, 102, 103, 104],
    "distance_km": [2.4, 5.1, 0.9, 7.3],
    "minutes": [21, 33, 14, 41],
})
v1 = data_hash(orders)

shuffled = orders.sample(frac=1, random_state=1)
fixed = orders.copy()
fixed.loc[fixed.order_id == 103, "minutes"] = 16   # corrected label

print("original :", v1)
print("shuffled :", data_hash(shuffled), "(same data, same hash)")
print("one fix  :", data_hash(fixed), "(different hash)")
print("raw bytes of shuffled match original?",
      shuffled.to_csv(index=False) == orders.to_csv(index=False))

Store that hash next to every training run. Months later, if someone asks whether the model saw the corrected label, you compare two short strings instead of two large files.

Data versioning with DVC

DVC (Data Version Control) is an open-source tool that applies the hashing idea at scale. It keeps large files in cheap storage such as S3 or a shared drive, and commits only a tiny pointer file to Git.

Checking out an old Git commit brings back the old pointer; dvc pull then fetches the matching data.

The usual first steps look like this, following the DVC documentation (DVC 3.x):

git init && dvc init
dvc add data/orders.parquet          # writes data/orders.parquet.dvc
git add data/orders.parquet.dvc data/.gitignore
git commit -m "Track October orders with DVC"

dvc remote add -d storage s3://chappan-ml/dvc-store
dvc push                             # upload the data blobs

# Later, on another machine or an older commit
git checkout <old-commit>
dvc pull                             # fetch the data that commit points to

DVC can also describe the training pipeline itself in a dvc.yaml file, so dvc repro reruns only the stages whose inputs changed:

stages:
  train:
    cmd: python train.py
    deps:
      - train.py
      - data/orders.parquet
    params:
      - train.n_estimators
      - train.window_days
    outs:
      - models/eta-model.joblib
    metrics:
      - metrics.json:
          cache: false

Why do random seeds matter?

Many algorithms use randomness: random forests sample rows and features, train/test splits shuffle, neural networks start from random weights. Run the same script twice without a fixed seed and you can get two different models.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error

rng = np.random.default_rng(0)
X = np.column_stack([rng.uniform(0.5, 9, 800), rng.integers(8, 25, 800)])
y = 6 + 3.2 * X[:, 0] + X[:, 1] + rng.normal(0, 3, 800)

def train_once(seed):
    m = RandomForestRegressor(n_estimators=20, max_features=1, random_state=seed)
    m.fit(X[:600], y[:600])
    return round(mean_absolute_error(y[600:], m.predict(X[600:])), 4)

print("no seed :", train_once(None), train_once(None))
print("seed=42 :", train_once(42), train_once(42))

Without a seed the two scores differ slightly from run to run. With random_state=42 they match exactly. Small differences like this make it impossible to tell whether a change to features helped or whether you were just lucky.

Fix the seed for numpy, for each scikit-learn estimator and for any splitting function, and store it in the configuration file.

Versioning the environment

The same pickle can behave differently, or fail to load, under another library version. scikit-learn warns when you unpickle a model saved with a different version, and the official docs on model persistence advise loading only with the same versions used for saving.

Pin exact versions in a lock file (requirements.txt with == pins, or a tool like uv or Poetry), and from the Docker lesson onwards, freeze the whole environment in an image. If you are new to isolated environments, read Python virtual environments first.

Model versioning with a registry

A model file on its own is anonymous. A model registry gives it a name, a version number and links back to the run that produced it.

First, the idea in plain Python. The block saves a model to bytes with pickle, fingerprints it, stores metadata beside it, then reloads and confirms the predictions are identical.

import hashlib, pickle, sys
import numpy as np
import sklearn
from sklearn.linear_model import Ridge

rng = np.random.default_rng(1)
X = np.column_stack([rng.uniform(0.5, 9, 300), rng.integers(8, 25, 300)])
y = 6 + 3.2 * X[:, 0] + X[:, 1] + rng.normal(0, 3, 300)
model = Ridge(alpha=1.0).fit(X, y)

blob = pickle.dumps(model)                     # what you would write to disk
registry = {"eta-model": {14: {
    "sha256": hashlib.sha256(blob).hexdigest()[:12],
    "sklearn": sklearn.__version__,
    "python": sys.version.split()[0],
    "data_hash": "3f9c1a7e02bd", "params": {"alpha": 1.0}}}}

loaded = pickle.loads(blob)
same = np.allclose(model.predict(X[:5]), loaded.predict(X[:5]))
print("registered:", registry["eta-model"][14])
print("predictions identical after reload:", same)

The same thing with MLflow

MLflow is an open-source platform for experiment tracking and model management. With MLflow 3, a training run that logs everything above looks like this:

import mlflow
from mlflow import MlflowClient
from sklearn.ensemble import RandomForestRegressor

mlflow.set_experiment("eta-model")
params = {"n_estimators": 300, "random_state": 42}

with mlflow.start_run():
    model = RandomForestRegressor(**params).fit(X_train, y_train)
    mlflow.log_params(params)
    mlflow.log_param("data_hash", data_hash(train_df))
    mlflow.log_metric("test_mae", mae)
    info = mlflow.sklearn.log_model(model, name="model",
                                    registered_model_name="eta-model")

client = MlflowClient()
version = info.registered_model_version
client.set_registered_model_alias("eta-model", "champion", version)

# The serving code loads whatever "champion" points to
live = mlflow.pyfunc.load_model("models:/eta-model@champion")

MLflow 3 uses name= in log_model (the older artifact_path= is deprecated), and aliases such as champion replace the old Staging and Production stages. Check the MLflow documentation for your version.

Common mistakes in ML versioning

• Committing big data to Git. The repository bloats and clones take forever. Use DVC or object storage with hashes.
• Overwriting model.pkl. Every saved model needs a unique version, never a reused file name.
• Seeds in code but not in config. Put them in the tracked parameters file so they are recorded with the run.
• Unpinned libraries. pip install scikit-learn today and next year give different versions.
• Versioning the model but not the features. If feature code changes, an old model fed new features behaves unpredictably. The data pipelines lesson covers this.

For learners who want the modelling fundamentals underneath all this, the Machine Learning course at Debugshala's Old Palasia campus is the natural companion, and you can pick a coding quiz on Debugshala Test between experiments.

Key takeaways

• A model version is five versions: code, data, configuration, environment and model file.
• Hash data in a canonical order, or use DVC to version large files beside Git.
• Fix random seeds and store them in tracked configuration.
• Pin library versions; models are only safe to load under the versions that saved them.
• A registry with aliases tells serving code exactly which model is live.