Every flight delay dataset has a trap. The features that best predict whether a departure was late are often computed from information that wouldn’t exist at prediction time — same-day gate-change status, arrival delays from the same tail number earlier that morning, or rolling aggregate statistics that include the flight you’re predicting. A model built with those features looks excellent on a held-out test set and fails the moment you ask it about a flight that hasn’t landed yet. The problem isn’t the algorithm; it’s that you gave it the answer as input.
That trap is where this project starts. Building a trustworthy delay predictor from public data isn’t interesting because flight delays are hard to model — they’re not particularly. It’s interesting because doing it correctly requires an architecture that enforces temporal discipline across every layer: ingestion, feature engineering, training, serving, and evaluation.
The Shared Contract Module
The enforcement mechanism is a Python package called flight_contracts that every stage imports. It pins four things in one place: the prediction label (dep_del15, whether departure delay exceeds 15 minutes), the temporal split boundary (train on 2022–2024, test on 2025 — never random), the feature list the model is allowed to see, and a list of banned leaky columns that no stage is permitted to pass downstream.
# shared/flight_contracts/leakage_contract.py
LABEL = "dep_del15"
TRAIN_CUTOFF = date(2025, 1, 1) # everything before → train
TEST_START = date(2025, 1, 1) # everything from → test
BANNED_LEAKY_COLS = [
"dep_delay", # the delay itself
"arr_delay", # arrives after departure
"actual_elapsed_time", # only known post-flight
"air_time",
"taxi_out", # gate departure to wheels-up
"wheels_off",
"wheels_on",
...
]If a feature engineer adds a column that’s in BANNED_LEAKY_COLS, the pipeline fails at schema validation before any data moves. If a model training script tries to use any feature not in the explicit allowlist, same result. The contract makes leakage a compile-time problem rather than a debugging problem that surfaces months later when prediction calibration drifts.
The Lakehouse Stack
Four public sources feed a medallion lakehouse: BTS on-time performance records (the historical backbone), NOAA/Meteostat weather archives keyed to airport locations and departure hours, OpenSky live aircraft positions for real-time congestion signals, and OpenFlights reference data for airport and carrier metadata. Each Python ingester handles its own rate-limit strategy, writes partitioned Parquet to a bronze layer, and resumes safely from the last partition on failure — no reprocessing from scratch.
Bronze is raw fidelity: schema is enforced but values aren’t corrected. Silver normalizes across sources — weather joined to departures on airport and hour, carrier codes standardized, tail numbers cleaned — and is where the contract’s banned-column list is applied. Gold is the analytics layer: the feature set ready for training, plus reliability aggregates at the airport, route, carrier, and hour granularity.
A parallel dbt project builds the reliability marts directly in DuckDB from the gold layer, with not-null, uniqueness, relationship, and accepted-range tests on every model. The dbt path is the analytics-first layer; the Spark path is the ML-first layer. They read from the same bronze, they just walk the transformation differently. On Databricks, both layers are registered in Unity Catalog so lineage is tracked end to end — from source file to trained model artifact.
Leakage-Safe Rolling Features
The hardest feature engineering problem isn’t what to include — it’s what to exclude at what time boundary. Route reliability, carrier delay rate, and origin-airport congestion are genuinely predictive signals, but only if computed from information available before the flight departs.
The Spark window functions that build rolling reliability features enforce this with an explicit ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING frame, bounded to strictly prior days:
w = (
Window
.partitionBy("origin", "dest")
.orderBy("fl_date")
.rowsBetween(Window.unboundedPreceding, -1) # never same-day
)
df = df.withColumn(
"route_prior_delay_rate",
F.avg(F.col("dep_del15").cast("double")).over(w)
)A flight on January 15th sees only January 1–14 data in its route delay rate. Same for carrier reliability and airport congestion — every rolling feature is bounded to days strictly before the flight date. The temporal split (TRAIN_CUTOFF) then ensures the test set contains only 2025 flights, which never appeared in training windows.
The ML Lifecycle
Algorithm choice was deliberately unremarkable. LightGBM on tabular data with proper temporal splitting beats most other choices without requiring much tuning, and I wanted to spend engineering time on the parts that actually matter: calibration, leakage prevention, and SHAP-based explainability.
dep_del15 is a binary label on imbalanced data — most flights aren’t delayed more than 15 minutes. Class-imbalance weighting in LightGBM handles the majority/minority imbalance during training. Isotonic calibration handles a different problem: a model that predicts “32% delay probability” needs to actually be right 32% of the time, not just rank flights correctly. ROC-AUC tells you the ranking quality; Brier score tells you the calibration quality. A well-ranked but poorly calibrated model is useless for the risk gauge in the dashboard, which needs the probability to mean something.
MLflow tracks every experiment: hyperparameters, feature importance, ROC-AUC, PR-AUC, Brier score, and a baseline comparison against the trivial “predict the base delay rate for this route” model. Every training run registers the model artifact in the MLflow Model Registry before exporting.
ONNX export is validated by a parity test that runs the same input through both the native LightGBM model and the ONNX runtime and asserts prediction agreement within floating-point tolerance. The serving layer loads the ONNX bundle — model, calibrator, and SHAP explainer — so it doesn’t depend on LightGBM being installed in the serving container.
Two Streaming Paths
The streaming architecture has two tracks that serve different purposes.
The Kafka track is the batch-complement showcase: ingesters publish raw events to Kafka topics, and a Spark Structured Streaming job consumes them with event-time windowed aggregation — tumbling windows over live aircraft positions to compute airport-congestion signals (how many aircraft are currently in approach or taxi at each airport). This is the reference architecture for production ingestion if you’re already running Kafka.
The NATS JetStream track drives the live demo. Aircraft positions stream into a NATS subject, a lightweight subscriber normalizes them, publishes to Valkey for the serving layer to blend into real-time predictions, and the SvelteKit frontend’s live map consumes position updates directly from the NATS WebSocket bridge. The demo map shows current aircraft in flight; the risk lookup blends live congestion into the delay probability alongside the historical features.
The two paths aren’t redundant — they’re for different contexts. Kafka for durable, high-throughput production ingestion. NATS for the real-time demo where the simplicity of the pub/sub model matters and message durability is less critical than low latency.
Serving and Explainability
The FastAPI serving layer loads the ONNX model bundle on startup and keeps the DuckDB gold store open as a read-only connection. A delay risk lookup for a given route and departure time runs three things in parallel: the model inference on the precomputed feature set, a DuckDB query for historical reliability stats on that route and carrier, and a SHAP explanation that decomposes the prediction into per-feature contributions.
@router.post("/predict")
async def predict(req: PredictRequest) -> PredictResponse:
features = await feature_store.get_features(req)
prob = float(onnx_session.run(None, {"X": features})[1][0, 1])
prob_cal = float(calibrator.predict_proba([[prob]])[0, 1])
shap_vals = explainer.shap_values(features)[0]
reliability = await gold_store.route_reliability(req.origin, req.dest)
return PredictResponse(
delay_probability=prob_cal,
factors=build_factor_bars(shap_vals, features),
reliability=reliability,
)The SHAP values power the “factor bars” in the dashboard — which signals pushed the probability up (departure hour, origin congestion, carrier history) and which pulled it down. This is the part that makes the prediction legible rather than just a number. A 41% delay probability with “departure at 7am from ATL, low carrier delay rate” tells you something; a bare number doesn’t.
The SvelteKit dashboard surfaces three views: risk lookup with the gauge and SHAP factor bars, airport and route reliability rankings pulled from the dbt gold marts, and the live aircraft map. The entire stack runs from npm run up and the serving API and dashboard are live in under a minute against the bundled sample model.
What I’d Change
The reliability mart refresh is currently a full recompute — dbt runs the whole DAG from silver, which is fine for weekly batch runs but unnecessary for daily incremental updates where only the new day’s flights need to fold in. Incremental dbt models with appropriate unique_key and merge strategy would reduce the nightly job from minutes to seconds at scale.
The model retraining DAG exists in Airflow but training and serving aren’t connected to a monitoring loop. Deploying prediction monitoring — tracking calibration drift as new flight data arrives — would close the feedback loop that makes the calibration investment actually pay off in production. Right now, whether the model is still well-calibrated six months later requires manually running the evaluation notebook.
Try It Out
Check out the live demo or explore the source code on GitHub.
