You’ve changed how search results are ranked and you want to know if it’s better. The obvious metric is click-through rate. But CTR is polluted by position bias: users click rank-1 results far more than rank-5 results regardless of whether rank-1 is actually more relevant. A ranking that puts any document at the top will see higher CTR for that document just because of where it appears. This makes CTR useless for comparing two ranking algorithms directly — you’d be measuring position effects, not ranking quality.
The standard response is to collect relevance labels: have humans judge whether each result is relevant to each query, then score your ranking against those judgments with nDCG or MRR. That works, but relevance labeling at scale is expensive, slow, and doesn’t generalize to the full query distribution.
Team draft interleaving is the online alternative. It evaluates two ranking algorithms against each other by weaving their results into a single list that gives each algorithm equal position exposure. When a user clicks, the click is attributed to whichever algorithm placed that document in the list. Because both algorithms had equal positional opportunity, the algorithm that gets more clicks is genuinely performing better — not just getting lucky with position.
What You Need Coming In
- Familiarity with Python: lists, dicts, list comprehensions
- Basic statistics: what a t-test is conceptually, what a p-value means
- Understanding of what a search result list looks like: an ordered list of document IDs
Why Equal Position Exposure Is the Key Property
The insight behind interleaving is that position bias is symmetric. If both algorithms have equal exposure at every position, the position effects cancel out. If algorithm A produces a better ranking, users will click its results more often — not because they appeared higher, but because they were more relevant when they did appear.
Standard A/B testing doesn’t have this property. In a 50/50 split, half the users see ranking A and half see ranking B. Position effects are balanced across user groups over time, but you need a large sample to overcome per-query variance — the same query might produce very different results depending on which users happened to run it. Interleaving compares A and B on exactly the same queries, with exactly the same users, at exactly the same moment.
The Team Draft Algorithm
Team draft interleaving works like picking teams from a roster. Two coaches (the two ranking algorithms) take turns selecting players (documents) for their team:
# eval/interleaving.py
def team_draft(rank_a: list[str], rank_b: list[str], k: int = 10) -> tuple[list[str], dict[str, str]]:
"""Return (interleaved, owner_map) where owner_map[doc_id] is 'A' or 'B'."""
interleaved: list[str] = []
owner: dict[str, str] = {}
ai = bi = 0
pickA = True
while len(interleaved) < k and (ai < len(rank_a) or bi < len(rank_b)):
if pickA:
while ai < len(rank_a) and rank_a[ai] in owner:
ai += 1
if ai < len(rank_a):
interleaved.append(rank_a[ai])
owner[rank_a[ai]] = "A"
ai += 1
else:
while bi < len(rank_b) and rank_b[bi] in owner:
bi += 1
if bi < len(rank_b):
interleaved.append(rank_b[bi])
owner[rank_b[bi]] = "B"
bi += 1
pickA = not pickA
return interleaved, ownerThe process: A picks its top-ranked unselected document. B picks its top-ranked unselected document. Alternate until the interleaved list reaches length k.
The owner map records which algorithm selected each document. When a user clicks a document, owner[clicked_doc_id] tells you which algorithm gets credit. A document that both algorithms rank highly will be selected by whichever goes first — but since alternation is fair, this averages out across many queries.
Walk through a small example. Algorithm A ranks: [d1, d2, d3, d4]. Algorithm B ranks: [d3, d1, d4, d2]. With k=4:
- A picks d1 → interleaved: [d1], owner: {d1: A}
- B picks d3 → interleaved: [d1, d3], owner: {d1: A, d3: B}
- A picks d2 (d1 already taken) → interleaved: [d1, d3, d2], owner: {d1: A, d3: B, d2: A}
- B picks d4 (d3 already taken, d1 already taken) → interleaved: [d1, d3, d2, d4], owner: {…, d4: B}
If the user clicks d2, algorithm A wins that query. If they click d3, B wins.
Counting Wins Per Query
The evaluator runs over a set of queries, each with two ranked lists and a set of clicks:
def evaluate_from_clicks(
rankings_a: dict[str, list[str]],
rankings_b: dict[str, list[str]],
clicks_by_query: dict[str, list[str]],
) -> TeamDraftResult:
a_wins = b_wins = ties = 0
for q, a_rank in rankings_a.items():
b_rank = rankings_b.get(q)
clicks = clicks_by_query.get(q, [])
if not b_rank or not clicks:
continue
_, owner = team_draft(a_rank, b_rank, k=10)
a_clicks = sum(1 for c in clicks if owner.get(c) == "A")
b_clicks = sum(1 for c in clicks if owner.get(c) == "B")
if a_clicks > b_clicks:
a_wins += 1
elif b_clicks > a_clicks:
b_wins += 1
else:
ties += 1
return TeamDraftResult(a_wins, b_wins, ties, t_test_paired(a_wins, b_wins, ties))Each query gets a binary outcome: A wins, B wins, or tie. This reduces noisy per-click data to a clean per-query judgment. A query where A received 3 clicks and B received 1 counts the same as a query where A received 1 click and B received 0 — both are A wins. This prevents high-traffic queries from dominating the aggregate result.
Testing for Statistical Significance
Counting wins gives you a direction (A is ahead), but not a confidence level. With 60 A-wins, 40 B-wins, and 20 ties, how confident should you be that A is genuinely better rather than just lucky?
The paired t-test converts win/loss/tie counts into a confidence value:
def t_test_paired(a_wins: int, b_wins: int, ties: int) -> float:
n = a_wins + b_wins + ties
if n == 0:
return 0.0
# Represent each query as +1 (A win), -1 (B win), 0 (tie)
diffs = [1.0] * a_wins + [-1.0] * b_wins + [0.0] * ties
mean = statistics.fmean(diffs)
sd = statistics.pstdev(diffs)
if sd == 0:
return 0.0
t = mean / (sd / math.sqrt(n))
# One-tailed p approximation for large n via normal CDF
p = 0.5 * math.erfc(t / math.sqrt(2))
return 1.0 - p # confidence A > BThis creates a list of per-query outcome values (+1, -1, 0), computes a t-statistic from their mean and standard deviation, then converts it to a one-tailed confidence value. A result of 0.95 means there’s 95% confidence that A is better than B — 1 in 20 chance the observed advantage is noise.
The erfc-based normal CDF approximation is valid for large n (rule of thumb: n > 30). For smaller query sets, a proper t-distribution lookup or permutation test is more accurate, but in practice you want at least a few hundred queries before drawing conclusions anyway.
The output when run against the clicks table:
{
"variant_a": "control",
"variant_b": "vector_heavy",
"a_wins": 312,
"b_wins": 241,
"ties": 47,
"confidence_a_better": 0.9823
}98% confidence that the control ranking outperforms the vector_heavy variant on this query set. That’s enough to reject the candidate and move on.
The Connection to LambdaMART
Interleaving evaluates ranking variants against each other. The click data it generates — which document a user clicked, at what position, for which query, under which variant — feeds directly into the LambdaMART reranker’s training loop.
The clicks table in Postgres stores every click event with its position and variant:
clicks (
session_id, query_id, query_text, variant,
clicked_doc_id, clicked_position, result_set_size,
facets JSONB, client_ts, server_ts
)From these events, pairwise training examples are constructed: for a query where the user clicked the document at position 3, all documents above position 3 that weren’t clicked get a negative label relative to the clicked document. This is the standard way to derive pairwise relevance judgments from implicit feedback without any human annotation.
XGBoost LambdaMART (scorer/ltr.py) is then trained on these pairwise preferences using a feature vector that includes the raw sub-scorer outputs (BM25 score, vector cosine, PageRank, recency decay), document attributes (study type, full-text availability, COI flag), and query attributes (length, presence of drug terms). The model learns which combination of signals best predicts which document a user would prefer, given a specific query.
What This Evaluation Framework Can’t Tell You
Interleaving measures relative preference between two algorithms on the actual query distribution. It doesn’t tell you whether either algorithm is good in absolute terms. If both algorithms return irrelevant results, interleaving will still pick a winner — the one whose irrelevant results users clicked slightly more often.
It also doesn’t handle tail queries well. Queries with zero clicks provide no signal. Rare query types with only a handful of examples have high variance in win/loss ratios. The confidence threshold matters here: requiring 0.95 confidence before promoting a new variant protects against false positives on thin data, but also means you’ll need more traffic to detect smaller improvements.
For absolute quality measurement, eval/run.py runs nDCG@10 against a set of queries with known-relevant documents — a ground-truth evaluation that interleaving can’t provide. The two approaches are complementary: nDCG gives you a floor guarantee that the ranking isn’t actively bad, and interleaving tells you which of two valid options users actually prefer.
