What actually predicts relevance — measured against a manual pass¶

Status: observations from a one-off LLM pass over a full gold build. Not a decision, and not measurements — see the provenance note below before relying on any number here.

V2 scoring (ADR-0020, docs/v2-plan.md) automates a judgement that was made once by hand, over every posting in gold. Doing it manually produced measurements about which signals decide relevance that the pipeline's current keyword rules cannot express. Those measurements are the input contract for the V2 prompt: without them, V2 rebuilds the keyword matcher this exercise disproved.

Source: a full local gold build across all active boards — 10,170 postings fetched, 1,179 surviving to gold.

Re-measured 2026-10-13 against the prod warehouse — 10,123 postings fetched, 3,126 silver survivors, 1,406 gold, under the deployed rules and the landed resume corpus. See Re-measured at the end: the counting claims were re-run and hold, each now on its third or fourth independent sample. Every precision claim below still rests on the single LLM pass described next.

Provenance, and why it limits what this file can claim. The pass was a single LLM run, not a human review. It was given a resume and asked to return the ~75–80 best postings by work match regardless of title, ordered by recency. So it produced a ranked shortlist, not a judgement on every gold row, and the counts below describe what one model selected rather than what is actually relevant.

Two consequences. These numbers cannot serve as an evaluation set for V2 — grading an LLM scorer against LLM-generated labels measures agreement with a predecessor, including its mistakes, so the eval set has to be human-labelled. And the run predates the resume corpus, so it matched against a single tailored resume version; anything that version omitted was invisible to it. Re-measure once the corpus exists.

What survives regardless is the shape of the argument: the pass was already doing resume-against-requirements matching, ad hoc and in one prompt. V2 formalizes what it did rather than inventing a new approach.

Title matching fails in both directions¶

This is the central result, and it is symmetric — which is what makes it interesting.

• The best-titled posting in the set was not relevant. Perfect title match, densest keyword hit count in the file, and the described work turned out to be month-end financial close.
• Six genuinely relevant postings sat in the reject pile, under titles naming the org chart rather than the work: Forward Deployed Engineer, Salesforce Integration Developer, AI Solutions Engineer, Lead Data Scientist, Senior People Data Scientist, ERP Specialist.

The generalization:

Companies name a role after the org chart the seat sits in, not the work being done.

Analytics work under an engineering VP gets an engineering title; the same work in a field organization gets a customer-facing one. This is why title cannot be the key, and why ADR-0015 made the title seed a soft signal rather than a filter.

Four instructions the V2 prompt has to encode¶

1. Score the requirements section, not the whole posting. A keyword pass over the title-only rejections flagged 21 as analytics-dense. 20 were false positives — finance managers, L&D managers, controllers, a head of procurement. Every corporate job now says "dashboards" and "stakeholders" somewhere. Re-scoring only the requirements section cut 21 flags to 13, of which 3 were real — 1/21 precision against 3/13.

A responsibilities blurb describes reporting. Only a data role requires dbt.

2. Eligibility is an output field, not a filter. The location gate keeps unqualified "Remote", so postings restricted to another country reach gold with entirely relevant titles — correctly ingested, and correctly not applied to. The pipeline cannot tell "Remote" from "Remote (US)"; a description usually can.

Emit eligibility ∈ {ok, restricted, unclear} from the description and let it rank. Never delete on it.

3. Level must be inferred from the described work, in both directions. The highest-ranked posting in the final set was titled Manager and its description listed no reports, no hiring, and deliverables that were entirely individual. Meanwhile five postings were rejected precisely because they managed people, including one whose tool stack was otherwise an exact match.

Infer level from scope statements — reports? hiring? headcount? — not from the title, and emit level_fit with a verify state rather than a binary.

4. Deal-breaker technologies demote, they do not delete. Across the gold set, 123 postings named a deal-breaker technology — Spark 88, Kafka 48, Scala 17, Flink 16, Hadoop 8, roughly 9% of the market. Enough were otherwise strong that deleting them would have cost real matches. Already encoded as ADR-0023; the measurement is here so it is not re-litigated.

What the pipeline cannot see¶

Gold drops postings that vanish from their board, or whose board stops responding, so every row is live in the strict sense. What no pipeline can determine is whether a posting is effectively filled — still published with an offer already out. Age is the only available proxy, which is why older postings warrant a different action rather than a lower score.

Market shape — for list building, not for scoring¶

• Only 5 postings in the entire set carried the title "Analytics Engineer", and every one was relevant — 5/5, against 31/43 for postings titled "Analyst". The scarcity is supply, not filtering, so no amount of better scoring produces more of them.
• 73 of the 113 companies with a live posting produced zero data-shaped roles of any kind. Ten companies produced two-thirds of everything relevant.
• Companies that sell data tooling have almost no analytics jobs; companies that run on data have plenty. Three large data-platform vendors contributed 1,639 postings and zero relevant ones — their analytics vocabulary lives in Finance and Sales Ops. The best conversion in the set was a mid-size company with 11 postings, 4 of them relevant.

That last point is the list-building rule, and it replaces "add large well-known companies".

Re-measured 2026-10-13¶

3,126 silver survivors, 1,406 gold, all of them scored, from a 10,123-posting fetch under the deployed rules and the landed corpus. Everything below is counting. Every precision claim above — 1/21 against 3/13, 5/5, 31/43 — still needs human labels by construction, per the provenance note, and none exist yet.

The deal-breaker rate holds a fourth time. The five terms the original pass counted appear in 300 of 3,126 postings (9.6%) — Spark 203, Kafka 119, Scala 31, Flink 31, Hadoop 18 — against 10.4%, 9.4% and ~9% on the three earlier fetches. Four samples inside a point and a half of each other makes "roughly 9-10% of the market" a property of the market, not of a snapshot; ADR-0023 rests on it and does not need revisiting. The deployed 35-term seed flags 668 (21.4%), against 22.2% — one posting in five carries a demotion. That is a heavier thumb on match_score than the rule was carrying when it was written, and two samples agreeing means it is the rule's real weight rather than a drift to watch.

Scarcity holds. Titles word-matching "Analytics Engineer": 11 of 3,126, 0.35% of the corpus, against 0.3% on each of the three earlier fetches — the share has not moved. "Analyst" appears in 117, "Data Engineer" in 15, "Data Analyst" in 15. The supply argument stands: no amount of better scoring produces more of them.

Concentration holds as a band, not a number. 78 of the 107 companies with a live posting produced no title-matched role at all — 73%, against 62%, 73% and 65% before. Four samples spread over eleven points, so the claim is "roughly two-thirds to three-quarters of boards carry nothing title-shaped", and anything tighter is reading noise. The three largest boards contribute 421 of 3,126 postings (13.5%, against 13%), so volume and relevance remain unrelated.

Title match is 3.8% (118 of 3,126) on the deployed 26-pattern seed, against 6.1% on the 31-pattern set it replaced and 3.6% on the seven-pattern set before that. The rate is not comparable across seed versions — it tracks the patterns, not the market — which is itself the argument for keeping it a soft signal: a number that moves when the rule moves cannot be a gate.

What the original pass could not measure¶

The V2 extraction fields did not exist when this file was written. They turn two of its arguments into measurements.

Instruction 3 is the strongest result here, and it reproduces. Of gold postings titled Manager, Lead or Head, 210 describe no people management against 149 that do, with 20 unclear — so 58.5% of manager-titled postings are individual-contributor work, against 60.5% and 58% on the two earlier corpora. The error runs the other way too: 147 of 1,027 (14.3%) postings not titled that way do manage people, against 12.9% and 13%. Three independent fetches inside three points of each other makes this a property of how companies name roles rather than a sampling artefact, and it is the org-chart claim at the top of this file with a number under it.

Instruction 2's predicted leak is real, sized and stable. Of 1,406 gold postings the location gate admitted, 169 (12.0%) are us_only on the description, against 12.2% and 12% before. The rule-based gate cannot see any of them, which is what the instruction argued; one posting in eight reaching gold un-appliable is the cost of a location rule that reads a board field instead of prose.

The fit-score distribution moved when the corpus did, and that settles a question.

The same postings, scored against a third of the corpus and against all of it. A partial corpus distorts the middle of the distribution and leaves the top intact: the trough moves from 2 to 3, while the count scoring 4-or-5 barely changes (69 against 65). So a thin corpus costs ordering among the also-rans, not the shortlist — and a score is only comparable to another score produced from the same corpus, which is why landed_version() digests the rendered prompt and re-scores when it changes.

Six weeks of posting turnover under that same prompt moved the bands again — 899 / 291 / 172 / 18 / 26 — leaving the top thinner still, 44 of 1,406 (3.1%) against 65 of 1,361 (4.8%). So a band count describes one corpus-and-market pair rather than the scorer, and only the shape carries across: a long tail at 1 and a shortlist in the low tens.

Telling the model to trust the board's location is worth 3׶

Instructing extraction to treat the job board's own location field as authoritative — rather than inferring eligibility from the description alone — is the single largest measured improvement to geo_restriction:

Same posting population, one instruction, unresolved eligibility down threefold. The first 555 rows under the fixed prompt read 6.8%; 2,683 rows later it sits at 10.5%, so the effect is "about 3×" and the 5× was a small-sample reading — worth recording, because it is the direction a measured improvement usually moves as the sample grows. Worth knowing for any future extraction field regardless: the board's structured metadata beats inference from prose whenever both are available.

It also exposes a property of the incremental guard. It compares text, model and scoring-prompt version — none of which move when the extraction prompt changes — so gold can hold rows extracted under two different prompts at once. 230 gold rows still carry the older, weaker answer, down from 559 six weeks earlier — postings turn over, so the stale vintage drains on its own at roughly 55 rows a week. Resolving the remainder means a full refresh, which re-bills the extraction backfill (ADR-0025): a costed choice that gets cheaper to decline the longer it waits.

Do the two rankings disagree enough to be worth keeping both?¶

Label-free, and the most decision-relevant thing here. Postings split into quartiles by match_score, against the LLM's verdict:

Three things follow, and all three survived the corpus turning over. The keyword score already finds almost everything the LLM rates highly — 42 of 44 high-fit postings, 95%, sit in its top quartile, against 56 of 65 (86%) six weeks earlier, so the cheap ranking is not being embarrassed. Its bottom half has no resolution at all: quartiles 1 and 2 are indistinguishable (1.30 against 1.20) and every posting in them scores zero or below, so ordering within them is noise — the quartile split is a tie-break, not a ranking. And two high-fit postings sit outside the top quartile — that, plus reordering inside it, is the LLM's entire marginal contribution.

Eighteen of the 42 top-quartile high-fit postings carry no title match, which is the earlier org-chart argument showing up in the ranking itself: the keyword score reached them through technology hits alone.

Whether two recovered postings and a re-sorted top quartile justify the scoring cost is exactly what human labels decide — and the margin narrowed rather than widened between the two measurements, which is the direction that argues for deciding sooner. The point of measuring it is that the question has a size: precision@k on the top quartile is where the two rankings actually differ, so that is where the labelling effort should concentrate rather than being spread evenly across 1,406 rows.