Documentation¶

This is the written half of the project: how the pipeline works, why it is shaped the way it is, and the measurements behind those choices. The code is on GitHub; the model-level lineage is in the dbt docs.

Where to start¶

Architecture is the system as it runs today: three dbt zones, two warehouses, the automation split by permission scope, and the four layers of health checks. If you read one page, read the failures that look like health — a scheduled pipeline's worst failures produce no signal at all, and each one needed its own gate.

Decisions are the why. Each record states what was decided, what it cost and what it ruled out. Three that shaped the most:

• Three medallion zones, not five layers — bronze, silver and gold map one-to-one onto dbt's staging, intermediate and marts.
• Dual-target dbt: DuckDB dev, BigQuery prod — one SQL codebase, dialect differences isolated in adapter.dispatch macros.
• The score orders, never filters — why the AI layer can fail without the pipeline shipping nothing.

Research is the evidence: probes against each ATS, a cost model for adding sources, and a measured account of which signals actually predict relevance.

• ATS feed survey — which platforms qualify for an adapter, and why.
• What actually predicts relevance — what V2 has to encode.
• Ingestion cost as sources multiply — the general model.

V2 plan is the implementation contract for the AI layer: in-warehouse extraction and scoring as dbt models.