Feature Stores and Data Leakage: Point-in-Time Correct ML

Imagine a cricket fan in Indore who claims he can predict match results. His trick: he writes his "prediction" after checking the final score on his phone. His record is perfect, and completely useless at the next match.

ML models do this by accident all the time. A feature quietly includes information from after the moment of prediction, the model learns to rely on it, and the offline score looks wonderful. In production the future is not available, and accuracy drops.

Preventing this, and serving features the same way to training and to the live app, is what a feature store is for.

What is data leakage in machine learning?

In the analogy, checking the final score is a leaked feature, the perfect record is your inflated validation score, and the next match is production.

Data leakage is when a model is trained with information that would not be available at the moment it makes a real prediction, which makes offline evaluation look better than live performance.

There are three common kinds:

The previous lesson on data pipelines fixed skew from mismatched code. Leakage is subtler: the code can be identical everywhere and still use the future.

Seeing leakage happen in code

Chappan Express wants a feature "this restaurant's average delivery time", because some kitchens are much slower. The easy way in pandas is groupby().transform("mean") over the whole table. That average includes the current order's own delivery time and every later order.

The block trains one model with the easy (leaky) feature and one with an honest feature built only from earlier orders, then scores both.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

rng = np.random.default_rng(21)
n, kitchens = 3000, 600
df = pd.DataFrame({"restaurant": rng.integers(0, kitchens, n),
                   "distance_km": rng.uniform(0.5, 9, n)})
pace = rng.normal(0, 3, kitchens)                  # each kitchen's own speed
df["minutes"] = 10 + 3.2 * df.distance_km + pace[df.restaurant] + rng.normal(0, 6, n)

g = df.groupby("restaurant").minutes
df["avg_leaky"] = g.transform("mean")              # uses ALL orders, even this one
df["avg_past"] = g.transform(lambda s: s.shift().expanding().mean())
df["avg_past"] = df.avg_past.fillna(df.minutes.iloc[:2100].mean())  # earlier orders only

train, test = df.iloc[:2100], df.iloc[2100:]       # rows are in time order
X = lambda d, col: d[["distance_km", col]].to_numpy()
leaky = LinearRegression().fit(X(train, "avg_leaky"), train.minutes)
honest = LinearRegression().fit(X(train, "avg_past"), train.minutes)

def mae(model, col):
    return mean_absolute_error(test.minutes, model.predict(X(test, col)))

print(f"leaky model, offline test : {mae(leaky, 'avg_leaky'):.2f} min  (looks great)")
print(f"leaky model, live reality : {mae(leaky, 'avg_past'):.2f} min  (only the past exists)")
print(f"honest model, offline test: {mae(honest, 'avg_past'):.2f} min  (what you will get)")

The leaky model has the best offline score. In production, where only past orders exist, it becomes the worst of the three, because it learned to trust a feature that secretly contained the answer. The honest model's offline score is a fair forecast of live behaviour.

That is the danger of leakage: it does not just mislead you about accuracy, it makes you pick the wrong model.

What is a point-in-time join?

Features usually live in a table with a timestamp: "Sarafa Poha Corner's average prep time was 14.5 minutes as of 1 p.m.". Training rows also have a timestamp: the moment the prediction was needed. A point-in-time join attaches to each training row the latest feature value from at or before that row's timestamp, never after.

Each order looks backwards along the timeline. The naive join grabs the newest value and leaks the future into the past.

pandas has this built in as merge_asof. The block compares it with a naive "join the latest value" approach.

import pandas as pd

# Feature table: a restaurant's average prep time, recomputed every few hours
stats = pd.DataFrame({
    "restaurant": ["sarafa_poha"] * 3 + ["vijay_biryani"] * 2,
    "ts": pd.to_datetime(["2026-10-14 09:00", "2026-10-14 13:00", "2026-10-14 19:00",
                          "2026-10-14 10:00", "2026-10-14 18:00"]),
    "avg_prep_min": [9.0, 14.5, 21.0, 16.0, 24.0],
})
# Training rows: orders with the moment the prediction was needed
orders = pd.DataFrame({
    "order_id": [501, 502, 503, 504],
    "restaurant": ["sarafa_poha", "sarafa_poha", "vijay_biryani", "vijay_biryani"],
    "ts": pd.to_datetime(["2026-10-14 12:40", "2026-10-14 20:15",
                          "2026-10-14 09:30", "2026-10-14 17:55"]),
})

latest = stats.sort_values("ts").groupby("restaurant").tail(1)
naive = orders.merge(latest[["restaurant", "avg_prep_min"]], on="restaurant")
pit = pd.merge_asof(orders.sort_values("ts"), stats.sort_values("ts"), on="ts",
                    by="restaurant", direction="backward",
                    tolerance=pd.Timedelta(hours=6))

print("Naive join (latest value, leaks the future):")
print(naive[["order_id", "avg_prep_min"]].to_string(index=False))
print("\nPoint-in-time join (value known at order time):")
print(pit.sort_values("order_id")[["order_id", "ts", "avg_prep_min"]].to_string(index=False))

Read the output row by row. Order 501 at 12:40 correctly gets 9.0, the 9 a.m. value, not the evening's 21.0. Order 503 at 9:30 gets nothing, because Vijay Biryani had no stats yet. Order 504 at 5:55 p.m. also gets nothing: the only earlier value is from 10 a.m., older than the six-hour tolerance.

That tolerance is the same idea feature stores call a TTL (time to live): a value older than this is treated as missing rather than trusted. Missing values are then handled explicitly, for example with a default, as covered in handling missing data.

If you know SQL, this is like a join with a "latest row before this time" condition, which plain SQL joins need a window function or subquery to express.

What is a feature store?

Doing point-in-time joins by hand works for one model and a few features. Chappan Express soon has 40 features used by three models: delivery time, rider assignment and a "will this order be cancelled" model. Each team recomputing the same features in its own way brings back skew and leakage.

A feature store is a data system that keeps a registry of feature definitions, an offline store of historical feature values for building training sets with point-in-time joins, and an online store of the latest values for low-latency serving.

One definition, two stores. Training and serving read the same features computed by the same code.

A feature store with Feast

Feast is a widely used open-source feature store. You declare entities and feature views in Python, then ask for historical features for training or online features for serving. The pattern below follows the Feast quickstart (Feast 0.6x):

from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Int64

restaurant = Entity(name="restaurant", join_keys=["restaurant_id"])

restaurant_stats_source = FileSource(
    path="data/restaurant_stats.parquet",
    timestamp_field="event_timestamp",
)

restaurant_stats = FeatureView(
    name="restaurant_stats",
    entities=[restaurant],
    ttl=timedelta(hours=6),
    schema=[Field(name="avg_prep_min", dtype=Float32),
            Field(name="orders_last_hour", dtype=Int64)],
    source=restaurant_stats_source,
)

feast apply                                  # register the definitions
feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S")   # fill the online store

from feast import FeatureStore
store = FeatureStore(repo_path=".")

# Training: point-in-time correct join against an entity DataFrame
training_df = store.get_historical_features(
    entity_df=orders_df,          # columns: restaurant_id, event_timestamp, label
    features=["restaurant_stats:avg_prep_min", "restaurant_stats:orders_last_hour"],
).to_df()

# Serving: latest values for one restaurant, in milliseconds
online = store.get_online_features(
    features=["restaurant_stats:avg_prep_min", "restaurant_stats:orders_last_hour"],
    entity_rows=[{"restaurant_id": 1042}],
).to_dict()

Notice that both calls name the same features. That single list is what keeps training and serving consistent. Details are in the Feast documentation.

Do you need a feature store?

Not on day one. A feature store adds infrastructure to run and learn.

• Skip it when you have one or two models, batch predictions, and features computed in one pipeline. Point-in-time joins in pandas or SQL are enough.
• Adopt it when several models share features, you need fresh features online in milliseconds, or different teams keep reinventing the same feature.

Either way, the habit matters more than the tool: every feature row carries an event timestamp, and every training join looks only backwards. Good features themselves come from domain thinking, covered in feature engineering.

Common mistakes with features and leakage

• Using transform("mean") over the whole table. It mixes future rows into the past. Use expanding or rolling windows with a shift.
• Joining on entity only, ignoring time. Always join on entity and timestamp.
• Random train/test splits with repeat entities. Split by time, or group by customer.
• Scaling before splitting. Fitting a scaler on all data leaks test statistics. Fit inside a Pipeline on training data only.
• No TTL. A feature last updated three days ago is not "current". Set an expiry and handle missing values.

To practise these instincts, rehearse join logic with quizzes on Debugshala Test. If you want guided project reviews on problems like these, look at Debugshala's generative AI classes for engineering graduates.

Key takeaways

• Data leakage lets a model use future information, inflating offline scores and picking the wrong model.
• Point-in-time joins attach only values known at each row's timestamp; pd.merge_asof with direction="backward" does this.
• A TTL or tolerance stops stale values from being trusted.
• A feature store combines a registry, an offline store for training and an online store for serving.
• Next, the deployment lesson puts eta-model behind an API.