Ingestion cost as sources multiply — a general model¶

Status: proposal. Only the retention lever is built; the rest awaits a decision.

The problem, stated generally¶

Ingestion cost — requests, wall time, landed bytes — scales with everything a board advertises. Value scales with what survives the gates in silver. Those two quantities are unrelated, and nothing in the current design connects them.

The design goal is not "make the one expensive board cheap". It is:

Make cost visible per source without being asked, bound it by policy rather than by per-adapter special cases, and keep the filtering rules in one place as adapters multiply.

Measured baseline (2026-07-28, all 167 active boards, one full run)¶

10,170 postings fetched, 0 boards failed, ~11.5 min wall time.

Per run 209 MB → per day 418 MB → steady state 167 GB logical at the 400-day partition expiry in force when this was measured (75 GB at 180 days). BigQuery's free allowance is 10 GiB.

As built (2026-08-01): the expiry is now 180 days — see §4 below. The 167 GB figure is kept as measured; the live steady state is the 75 GB line. Six months was chosen on need, not on cost: a posting older than that is not a lead.

Three findings, none of which match where attention was going:

1. Greenhouse and Ashby are 85% of the bytes. SmartRecruiters — the board that prompted this whole investigation, at ~10 minutes of run time — is 7.3%. The expensive thing was not the one anybody suspected, and it had been expensive since long before V1.8.
2. 39% of all bytes are the same text stored twice. description_html (80.9 MB) is re-assembled from fields that raw also keeps verbatim — Greenhouse's content, Ashby's descriptionHtml (which also ships a descriptionPlain copy), SmartRecruiters' jobAd.sections. This is uniform across sources and will be true of every future adapter unless something prevents it.
3. Keep rate is inversely related to board size. The local-employer sources V1.8 added keep 77% / 47% / 42% of what they fetch; the big global boards keep 8–19%, and SmartRecruiters 0.9%. Coverage of large boards is what costs; coverage of local ones is nearly free.

Proposal¶

Four parts, deliberately separated because they address different costs. Parts 1 and 3 are general; part 2 helps a specific (but growing) class of source; part 4 is a policy dial.

1. Per-source cost instrumentation — do this first, regardless of the rest¶

ops.ingest_runs records rows_fetched and nothing about cost, so an expensive source is invisible until a human profiles the pipeline by hand. That is exactly how this document came to exist, three weeks late, and it found the wrong suspect twice.

Add to IngestRun → ops.ingest_runs:

Then one ops model joins it against what silver kept: bytes and requests per surviving posting, per source. Put the worst offender in the digest footer next to the skipped-board line that already exists.

This is the part that answers "who knows what'll appear there". You cannot predict which future ATS is pathological; you can make it announce itself on its first run.

Watch for: the counters are written from worker threads — per (source, board) accumulation must be thread-safe, not a module-level integer.

2. Two-phase adapter contract: list → gate → enrich¶

Generalize SourceAdapter from one method to two:

class SourceAdapter(Protocol):
    def list_candidates(self, session, board_ref) -> list[Candidate]: ...
    def enrich(self, session, candidate) -> RawPosting: ...

The pipeline, not the adapter, decides whether to enrich, by applying a shared seed-driven predicate between the phases. A new ATS implements two well-defined steps and inherits whatever cost policy exists at the time; the policy can change without reopening nine adapters.

Be honest about the ceiling: this saves requests and wall time, not bytes. It only bites where a source pays per posting — today BambooHR, SmartRecruiters, Rippling, i.e. 8.9% of bytes but a large share of run time. Greenhouse and Ashby return the whole board in one response; there is no per-posting call to skip, so the gate cannot touch the 85%.

It is still worth considering, because detail-fetching sources are likely to be the norm going forward (of the six ATS added in V1.8, three needed per-posting calls), and run time is what makes a schedule fragile.

Watch for: pagination and Rippling's collapse-by-uuid both belong in phase 1, before the gate sees candidates. description_html stays non-optional — a skipped enrichment lands empty text only for postings the gate already judged out, so "everything reaching gold has a description" still holds.

3. One gate definition, two engines — with a test that they agree¶

The predicate reads the same dbt seed silver reads (dbt/seeds/allowed_locations.csv), so "filter rules are data" (CLAUDE.md) still holds: the rule is not reimplemented in Python, it is read by Python.

• Silver stays authoritative — it re-applies the full rule to everything landed.
• Python's copy is advisory and conservative: skip enrichment only when a candidate is definitively disqualified (non-null location matching nothing). Null/unknown always enriches. Being wrong costs a request, never a posting.
• It self-heals: raw re-lands the whole board every run (WRITE_APPEND), so widening the seed means the next run fetches full text for everything skipped before. Full ingestion-side filtering — dropping rows outright — would destroy this property, and would also make "this source landed 0 rows" ambiguous between board is dead and nothing local this week.

The drift risk is the real price, and it grows with every adapter. Mitigation: a golden fixture set of location strings asserted to produce identical verdicts from the Python predicate and from regexp_word_ci in dbt, failing the build on disagreement.

Open question: is location the only pre-enrichment dimension, or is this a general "pre-filter" hook (title patterns next)? More dimensions = larger blast radius when the two engines disagree.

4. Payload discipline and storage policy — the biggest lever, and the most general¶

Ordered by measured impact:

• Stop storing the same text twice — worth ~39% of all bytes, every source, forever. raw should keep only what the mapping did not already extract. Needs a shared helper (drop mapped keys) plus a test that fails when description text appears in both places, or every future adapter re-introduces it. This is the change that best fits "who knows what'll appear there": it is a property of the schema, not of any one ATS.
• Retention is one knob: bq_raw_partition_expiry_days, 400 → 180 halves the footprint (167 GB → 75 GB) and was already the stated preference. Done 2026-08-01. One trap found in the doing: create_table(exists_ok=True) returns an existing table untouched, so the setting reached new tables only. _ensure_bigquery_objects now reconciles the expiry on every run, which is what makes a future change to this number mean anything.
• Compression is one setting: ALTER SCHEMA jobs_raw SET OPTIONS(storage_billing_model = 'PHYSICAL'). JSON of this shape typically compresses 5–10×. Caveat: physical billing also charges time-travel (7d) and fail-safe (7d) bytes, so measure rather than assume.
• Combined, those three take a 167 GB steady state to roughly 5–10 GB — back inside the free allowance — with no change to what the pipeline fetches or keeps.

4b. What those bytes cost in dollars — the check that settles the rest¶

Everything above counts bytes. Nobody had converted them, and the conversion is what decides whether any of it is worth building.

Steady state and billable volume, from the measured 0.42 GB/day (10 GiB free allowance):

Raw partitions are append-only and never modified, so a partition crosses into long-term storage at 90 days and stays there at roughly half the active rate; the split above is ~90 days active and the remainder long-term, with the free allowance charged against the active bytes, which is the pessimistic reading.

The rate is the one unverified input. Google's pricing page would not yield a region-specific table, so the assumed $0.023/GiB/month for active logical storage is the us-central1 figure inferred from a worked example in their docs. That is now the warehouse's own region (ADR-0026), which removes the region mismatch this note originally flagged; the authoritative number is still the project's billing console. The conclusion does not depend on it — that is why the table shows 2× and 3×.

Against that, the two remaining levers:

So both remaining levers are worth roughly $1.30/month between them, and under $4/month even if the real rate is triple the assumption. That is below the threshold where engineering time is the cheaper resource, so both are parked rather than scheduled — not because the analysis is uncertain, but because it is decisive in the other direction. The retention change was still worth doing: it was a config edit, and six months is what the data is actually worth.

This also retires the "back inside the free allowance" framing above. Getting under 10 GiB was never the goal; it was a proxy for a cost that turns out not to matter.

• GCS + Parquet + BigQuery external tables (the S3-equivalent archive tier) stays out of scope as a cost fix, with a stated trigger instead: steady-state logical raw > 50 GB after the three levers above, or a real analytical need for >6 months of history. Below that, the export job, Hive-partitioned layout, external table and lifecycle rules cost more attention than they save. Above it, the shape is EXPORT DATA … format='PARQUET' per expiring partition → gs://…/raw/{source}/dt=…/ → external table → GCS lifecycle Standard → Coldline.

Sequencing, if adopted¶

1. Instrumentation (part 1) — changes no behavior, produces the numbers that decide the rest.
2. Payload discipline + retention + compression (part 4) — ~39% + 55% + 5–10× against a problem that is 85% concentrated in two sources the other parts cannot help.
3. Two-phase contract (part 2) with enrich as a no-op everywhere — a pure refactor, provable by the existing adapter tests.
4. The gate (part 3) — only once part 1 shows which sources it would help and by how much.

The case against parts 2 and 3¶

Stated plainly, because it may be right: curating which boards are active is a lever that already exists and costs nothing. One row turns a bad board off, and the measurement shows the worst offenders are few and obvious once someone looks.

The counter-argument is that curation is a human noticing, per board, forever — which is what part 1 fixes. If part 1 lands and the digest names the worst source every run, parts 2 and 3 may never be needed.

Related finding, not about cost¶

Coverage is not the constraint — relevance is. Measured across the same run and recorded in docs/research/relevance-signals.md.