PySpark

Making Data Leakage a Pipeline Failure, Not a Debugging Session

A rolling delay rate that includes same-day flights is leakage. A temporal split that shuffles randomly is leakage. In a flight delay prediction pipeline, leakage produces models that look great on held-out data and fail immediately in production. This tutorial covers the shared contract module, Spark window functions bounded to strictly prior days, schema assertions that catch banned features before training, and a temporal split that respects the direction of time.