FlyRank ML Internship · Capstone Research

From Search Signals to Content Actions

Which existing page deserves the next review hour? A client-grouped, out-of-fold ranking of 30,000 pseudonymous content items turns search and engagement signals into a ranked, reason-coded action queue for a human editor.

0.632AUC (model) · out-of-fold
0.708Precision@50 vs 0.542 base
16,262declining / 30,000 rows
24features · 32 clients grouped

Research pipeline

1
Data
30,000 pseudonymous content items · trailing-90-day
2
Features
24 validated inputs — metrics, type, intent · no IDs, no label
3
Model
Logistic regression, refit per fold · seed 42
4
Validation
Client-grouped GroupKFold(5) · out-of-fold, no client leaks
5
Ranked Actions
archetype → human-reviewed action · reason code on every row
Abstract

What this research does

Question. FlyRank works with large-scale search and content performance data, and in the content-refresh lane the practical decision is prioritization: which existing pages deserve a human reviewer's attention first?

Approach. Using the anonymized starter release — 30,000 pseudonymous content items, 32 clients, trailing-90-day window metrics plus a 30-day trend — a logistic model (24 features) estimates each page's probability of being in the observed decline.

Validation. Client-grouped out-of-fold GroupKFold(k=5); preprocessing is refit per fold; each row is scored only by the fold that held its client out; a transparent three-rule Week-4 baseline is scored on the identical folds.

Result. The model reaches out-of-fold AUC 0.632 (sd 0.037) vs. the baseline's 0.558 (sd 0.096), and precision@50 0.708 (sd 0.148) against a 0.542 base rate — both measured on the same five folds.

Use. A ranked queue with a reason code and archetype per row becomes a human-reviewed action playbook (refresh, expand, recheck position, monitor, human review). It is decision support, not automation, and not a causal claim that editing a page improves its future performance.

Introduction / Problem statement

Which pages deserve the next review hour?

FlyRank works with large-scale search and content performance data. Its content team has one recurring decision: among thousands of existing pages, which few should get this week's editorial hour? That is a prioritization problem — the cost of a wrong pick is an hour spent on a page that is not the best next candidate.

A human cannot re-derive an ordering over 30,000 pages by hand — but they can honestly verify the first 20–100 rows of a good ranking. This system exists to make that ordering cheap to audit and trust.

≠

What this model is

A ranked, evidence-backed queue — the model scores, the playbook maps each score to a concrete, human-reviewed content action. It does nothing more: no auto-edit, auto-publish, delete, redirect, or rank manipulation.

decision support ranking human-in-the-loop
Data

A public-safe, anonymized snapshot

Everything below comes from the in-repo starter release. No client names, domains, URLs, queries, tokens, or raw identifiers appear anywhere in this document.

30,000
content items (rows)
one pseudonymous page per row
32
pseudonymous clients
used only for grouped validation
44
raw columns
window metrics + trend + descriptors
90d
trailing windows
plus a 30-day vs. prior-30-day trend
f

Features (24)

22 numeric window/performance metrics (visibility, clicks, sessions, engagement, freshness, position, CTR, rates) + 2 categorical (content_type, main_intent). No identifier column is ever a feature.

#

Label (proxy)

decline_label = (trend_direction == 'down') from the 30-day trend. A proxy for "worth a human's refresh review", not ground truth or a causal claim.

⌀

Excluded on purpose

content_id (row id), client_id (grouping only), trend_direction / trend_pct (label sources) — asserted in code.

Methodology

How the ranking is built and validated

A single pipeline relative to the label, all checked for leakage before any number is reported.

Labeldecline = (trend_direction == 'down') — the observed 30-day trend at snapshot time. 16,262 of 30,000 rows (0.542) are declining.
Features — 2422 performance/window metrics + content_type + main_intent. Broken down in the data section; asserted disjoint from IDs and label sources.
Baseline — Week-4 rulestale (≥180d since update) + visible (impressions≥500) + ranking (avg_position≥10) → integer 0–3, ranked top-down. Raw and checkable.
Model — logistic regressionOne scikit-learn Pipeline: median / most-frequent imputer → scaler / one-hot → logistic (max_iter=1000, random_state=42). A transparent, calibratable ranker.
Validation — client-grouped OOFGroupKFold(k=5): each client in exactly one test fold; preprocessing refit per fold; every row scored only by the fold that held its client out. Baseline scored on the same folds.
Leakage checks① label sources excluded (asserted) · ② client_id only as groups · ③ a leak-probe (label as a feature) hits near-perfect AUC under this harness — the model's 0.63 is nowhere near, so leaks would have surfaced · ④ no future or overlapping windows.
✓

Why this workflow is useful

The deliverable is an ordering, so the honest test is: does the top of the queue contain more declining pages than the base rate or the simple rule? Everything else — grouped splits, refit-per-fold, leak probes — exists to make that claim believable.

GroupKFold out-of-fold refit per fold
↥

The honest check

Baseline and model are compared on identical folds with the same base rate reading. Apples to apples — that is the only fair comparison.

Results

Model vs. baseline — same folds, out-of-fold

Mean ± std across the five client-grouped test folds. Base rate shown for every precision number. All metrics are recomputed in capstone.ipynb from the raw CSV — nothing is hard-coded on this page.

0.632 ± 0.037
model AUC
0.558 ± 0.096
baseline AUC
0.708 ± 0.148
precision@50 (model)
0.542
pooled base rate
Model vs. Week-4 baseline AUC + P@50
Bar chart comparing the logistic model with the Week-4 rule on AUC and Precision@50, on the same client-grouped out-of-fold splits, with the 0.542 base rate as a dashed line.

Caption: on identical client-grouped, out-of-fold folds, the model beats the transparent rule on AUC (0.632 → 0.558) and precision@50 (0.708 → 0.604). Both sit above the base rate — ordering by the model concentrates "declining" items at the top of the queue.

Precision at the queue head operational
Bar chart of precision at top-20, top-50 and top-100 of the ranked queue vs the 0.542 base rate.

The operational view on all 30,000 rows (out-of-fold scores): precision@50 ≈ 0.64 and @100 ≈ 0.71 against a 0.542 base rate. The editor at the top of the queue sees a denser set of declining pages than random.

Score distribution model output
Histogram of out-of-fold model probabilities, with proposed priority thresholds near 0.50 and 0.65.

Out-of-fold probabilities spread across the range; thresholds at ≈0.50 and ≈0.65 are proposed priority cuts, not validated optima.

Archetype / action mix (W7 playbook)
Distribution of action archetypes from the Week-7 action playbook: ctr, refresh, monitor, engagement, human-review, recheck-position, thin content.

Which playbook action each queue row falls into (computed in w07_action_playbook.ipynb): most rows resolve to a visible, cheap reviewable action — CTR, refresh, engagement, monitor — and "insufficient data" routes to a human check rather than acting on missing numbers.

MetricModel (logistic)Baseline ruleBase rate
ROC–AUC0.632 ± 0.0370.558 ± 0.0960.542
Precision@200.680 ± 0.2140.640 ± 0.2630.542
Precision@500.708 ± 0.1480.604 ± 0.2240.542
Precision@1000.726 ± 0.1150.626 ± 0.1610.542

Queue-head precision on the whole out-of-fold-ranked queue: @20 ≈ 0.45, @50 ≈ 0.64, @100 ≈ 0.71 (base 0.542). The fold-to-fold spread (P@50 std ≈ 0.15) is a reminder — directional aid, not a per-page promise.

Action playbook

Ranked recommendations — always human-reviewed

Every queue row maps to exactly one archetype → one action, with a machine-readable reason code. Priority bands are proposed cuts of the model score (high ≥ 0.65, medium ≥ 0.50, low < 0.50) — heuristics, not validated optima.

high
8,367 rows
medium
10,905 rows
low
10,728 rows
1
CTR candidaterefresh_and_review_ctr
Visible in SERP, low clicks — a small title/CTR edit may move one metric. Reason: page_one_decay_risk / low_ctr_visible_page.
2
Thin content candidateexpand_and_refresh
Visible but short — depth may cap positioning. Reason: thin_visible_page.
3
Refresh candidaterefresh
Declining and visible demand. Reason: declining_with_demand.
4
Engagement candidaterefresh_and_review_engagement
Visible but losing sessions. Reason: low_engagement_visible_page.
5
Ranking candidaterecheck_position
Visible below page one with a decent CTR — a positioning audit. Reason: page_one_decay_risk.
6
Monitor candidate / healthymonitor
Model says risk but no concrete page signal — watch list, don't touch yet.
7
Insufficient datahuman_review
Missing position / keyword / word-count evidence — a reason to not act without checking.

Order of review (cost/value heuristic — clearly proposed, not measured)

Rows are ranked by model score first; inside a priority band we suggest the cheapest visible high-confidence fix first: CTR → thin → refresh → engagement → recheck position. There is no effort or cost datum in the repo, so this ordering is qualitative, not a measured result.

Human review is mandatory. The editor opens the queue at rank 1, reads the reason code, checks ① search intent ② content quality ③ SERP context ④ the single-number trend ⑤ business relevance ⑥ any "no-data" marker — and may always reject a row. No auto-refresh, no auto-publish at any point.

What this system must never do on its own

  • Publish content autonomously
  • Delete or redirect pages
  • Change canonical URLs or titles/meta without human review
  • Treat missing data as a confirmed problem
  • Prioritize solely from one score with no context
  • Be read as proof of Google ranking causality
Limitations & honest framing

What this work does not claim

Observational, single snapshot

One time cut with trailing windows. No seasonality, no outcome-feedback loop; numbers may not hold next month or for a different client mix.

No causal refresh experiment

There is no refresh-event column and no before/after design in this repo. "Declining" is an association; refreshing a page is never claimed here to cause recovery.

Proxy label

The trend label is a snapshot heuristic for "worth reviewing", not ground truth.

Missingness + imputation

search_volume/competition /cpc are missing on ~8% of rows, word-count on ~26%; the imputer fills the model, and "no data" rows carry explicit markers rather than pretending to decisions.

Wide fold spread

P@50 std ≈ 0.15 — one client's fold can look markedly better or worse. This is an ordering aid, not a per-page promise.

Heuristic thresholds

The 3-level priority bands are proposed cuts of the model score, not validated optima; the refresh rule is a conservative decision-support guideline, not a guarantee.

Honest vocabulary

Every claim uses measured, observed, directional, decision-support, proposed, requires human review. No proves, no guarantees, no will increase traffic, no causal refresh, no production serving claim.

Reproducibility

Run it yourself, top to bottom

Capstone notebook

work/notebooks/capstone.ipynb recomputes every number on this page from the raw anonymized CSV, out-of-fold, with zero cached figures.

python -m nbconvert --execute --inplace work/notebooks/capstone.ipynb

GroupKFold by client_id seed 42 Python 3.13 · sklearn 1.5.2

Tracked notebooks (in order)

  • w01_research_question → w02_ml_task_framing
  • w03_data_contract + leakage probe
  • w04_baseline_score (the Week-4 rule)
  • w05_model → w06_validation_audit
  • w07_action_playbook (archetype → action mapping)
  • capstone.ipynb — recomputes everything on this page

Repository → github.com/H-Mike-Yankee/flyrank-ml-internship