A rolling delay rate for a flight route — what fraction of departures on this origin-destination pair were delayed more than 15 minutes over the past 90 days — is a real predictive signal. Routes with consistently bad on-time performance tend to stay that way. The correlation with the label isn’t spurious.
The danger is that “rolling” is implementation-defined. If the window includes flights from the same day as the one you’re predicting, you’ve built a feature that contains future information relative to the moment of prediction. The model trains on data where the rolling rate is computed with that day’s delays included. At serving time, you don’t have that day’s delays yet — the flight hasn’t happened. You feed a different value into the same feature slot. The model’s learned weights, tuned during training when the feature was slightly corrupted with same-day data, don’t generalize correctly.
This is temporal leakage, and it’s not subtle. A model trained with leaky rolling features will score noticeably higher on your test set than it performs in production, and the gap only becomes obvious when someone notices the predictions are miscalibrated on live traffic.
What You Need Coming In
- Comfortable with PySpark DataFrames and Python
- Familiar with what a training/test split is and why it matters
- Basic understanding of window functions in SQL or Pandas
One Module That Owns the Contract
The first architectural decision: every stage of the pipeline imports a shared Python package rather than defining its own concept of which columns are safe and what the temporal boundary is. When there are four or five separate stages (ingestion, silver transformation, gold feature engineering, model training, serving), each with their own idea of “the feature list,” you’ll eventually find a column that’s allowed at one stage and banned at another, and the inconsistency won’t surface until the model is deployed.
The contract lives in shared/flight_contracts/contract.py:
# Prediction label
LABEL = "dep_del15" # 1 if departure delay ≥ 15 minutes, else 0
# Temporal split — explicit year boundary, never random
TRAIN_YEARS = [2022, 2023, 2024]
TEST_YEARS = [2025]
TRAIN_CUTOFF = date(2025, 1, 1)
# Columns that must never reach training or serving
BANNED_LEAKY_COLS = [
"dep_delay", # the delay itself — obviously the label
"arr_delay", # only known after the plane lands
"actual_elapsed_time", # same
"air_time",
"taxi_out", # gate departure to wheels-up — post-boarding info
"wheels_off",
"wheels_on",
"dep_time", # actual departure time (not scheduled)
"arr_time",
"cancellation_code",
"diverted",
"carrier_delay",
"weather_delay",
"nas_delay",
"security_delay",
"late_aircraft_delay",
]
# Explicit allowlist — only these columns reach the model
MODEL_FEATURES = [
"origin", "dest", "mkt_unique_carrier",
"dep_hour", "day_of_week", "month",
"is_holiday_window",
"route_prior_delay_rate",
"origin_prior_delay_rate",
"carrier_prior_delay_rate",
"precip_mm", "wind_kph", "visibility_km", "temp_c",
"origin_congestion",
]The BANNED_LEAKY_COLS list is explicit documentation of what not to do, not just policy. taxi_out is there because it’s computed after the aircraft pushes back — it exists in the BTS dataset because BTS records it post-flight, but at prediction time you don’t have it. Someone new to the data might include it because it’s a strong feature (longer taxi-out often correlates with delays). The list stops that.
The allowlist matters as much as the ban list. A feature not on MODEL_FEATURES can’t reach training even if it isn’t in BANNED_LEAKY_COLS. The contract is restrictive by default.
Spark Window Functions Bounded to Prior Days
The rolling delay rates in MODEL_FEATURES need to be computed from flights that occurred strictly before the day being predicted. The Spark implementation uses Window.rangeBetween with an upper bound of -1 day:
from pyspark.sql import functions as F, Window
def _hist_rate(df, partition_cols: list[str], label_col: str, alias: str):
"""Cumulative prior-days delay rate — never includes current day."""
w = (
Window
.partitionBy(*partition_cols)
.orderBy(F.col("fl_date").cast("long")) # order by epoch seconds
.rangeBetween(Window.unboundedPreceding, -86_400) # -1 day in seconds
)
return df.withColumn(
alias,
F.sum(F.col(label_col).cast("double")).over(w)
/ F.nullif(F.count(F.col(label_col)).over(w), 0),
)The orderBy needs a numeric type for rangeBetween to work on — casting fl_date to its epoch representation in seconds means -86_400 is exactly one day. Window.unboundedPreceding reaches back as far as data exists. The upper bound of -86_400 means: up to and including one day before the current row’s date, and no further. The flight being predicted never appears in its own feature computation.
F.nullif(..., 0) handles routes that have no prior history — a new route in the dataset has zero denominator on its first few flights. The null propagates through downstream filling rather than producing a division-by-zero or an artificially extreme rate.
Three rates get computed at different granularities:
def build_gold_features(silver_df):
df = silver_df
# Route-level: origin + destination pair
df = _hist_rate(df, ["origin", "dest"], LABEL, "route_prior_delay_rate")
# Origin airport: all departures regardless of destination
df = _hist_rate(df, ["origin"], LABEL, "origin_prior_delay_rate")
# Carrier: all flights for this carrier regardless of route
df = _hist_rate(df, ["mkt_unique_carrier"], LABEL, "carrier_prior_delay_rate")
return dfEach partitions differently. route_prior_delay_rate captures the specific reliability of this route. origin_prior_delay_rate captures how the departure airport is performing overall, which matters on days when a regional weather event is affecting all departures. carrier_prior_delay_rate captures systemic carrier issues.
The Schema Assertion at the End of Gold
After all features are computed, the gold layer runs a validation step before writing output:
from flight_contracts.contract import BANNED_LEAKY_COLS, MODEL_FEATURES, LABEL
def assert_no_leakage(df):
cols = set(df.columns)
# Check banned columns aren't present
leaked = cols & set(BANNED_LEAKY_COLS)
if leaked:
raise ValueError(f"Leaky columns found in gold output: {leaked}")
# Check all required features are present
missing = set(MODEL_FEATURES) - cols
if missing:
raise ValueError(f"Required features missing from gold: {missing}")
# Check label is present
if LABEL not in cols:
raise ValueError(f"Label column '{LABEL}' missing")This runs as part of the gold build job, before any Parquet is written. A pipeline that accidentally carried taxi_out through silver into gold fails here with an explicit message rather than silently producing a training dataset with a leaky feature.
The check is cheap — it’s column name comparison, not data validation — so it adds milliseconds to a job that takes minutes. Running it before every write means the contract is enforced continuously, not just at code review.
Why the Temporal Split Has to Be Explicit
The test split uses TEST_YEARS = [2025], and this decision is more significant than it looks.
A random 80/20 split on a time-series dataset produces a test set where every test flight has temporal neighbors in the training set. A 2024 flight randomly assigned to the test set is surrounded by 2024 training flights with similar rolling delay rates, similar seasonal patterns, similar carrier performance. The model has seen those patterns. It scores well.
The 2025 test set contains no training neighbors. The model has to generalize to a future year it’s never seen, with carriers whose recent performance it knows only from 2022–2024, with routes it’s estimated reliability from historical data. That’s the actual production scenario: you train on history and predict on future flights.
def temporal_split(df):
train = df[df["year"].isin(TRAIN_YEARS)].copy()
test = df[df["year"].isin(TEST_YEARS)].copy()
return train, testWithin the training set, calibration and early-stopping validation are also carved temporally — the final months of 2024 become the calibration slice, the months before that become the early-stopping validation set:
def train_valid_calib_split(train_df):
sorted_df = train_df.sort_values("fl_date")
n = len(sorted_df)
calib_start = int(n * 0.90) # last 10% of training data
valid_start = int(n * 0.80) # next-to-last 10% for early stopping
train = sorted_df.iloc[:valid_start]
valid = sorted_df.iloc[valid_start:calib_start]
calib = sorted_df.iloc[calib_start:]
return train, valid, calibThe three slices are chronological: train comes first, then validation (used for early stopping), then calibration (used to fit the isotonic calibrator after training). No data from the future appears in the past — not during early stopping, not during calibration, not during evaluation.
A model that scores well under this split is actually generalizing. A model that only scores well under a random split might be memorizing temporal patterns.
