analytics engineering · portfolio project

Job postings in.
Tested, ranked data out.

A scheduled pipeline that pulls every public, keyless ATS feed into BigQuery, models it through a bronze → silver → gold dbt project that runs identically on DuckDB, and delivers a ranked digest of what is new. Every invariant it relies on is enforced by a check that fails — not by intent.

Explore the lineage Read the architecture View the code ↗

make site
manifest22 models · 11 sources · 2 exposures
built as3 incremental · 9 table · 10 view
tests117 data · 18 unit
contracts2 enforced
docs40 pages · 0 broken links
commit76fca94 · 2026-11-04

22dbt modelsbronze → silver → gold

117data testsschema + relationship contracts

18dbt unit testsmodel logic on fixed inputs

368pytest functionsbehind an enforced coverage floor

29decision recordsevery non-obvious choice

01Hiring at the tracked companies rebuilt after every scheduled run

Not the whole job market: these figures describe a curated list of company boards that the pipeline reads, and only those. They are the gold-zone agg_market_* models, exported by the production run and drawn when this page was built.

15,380open postings

211tracked companies hiring

1,199new in the last 7 days

44 daysmedian time a posting stays up

Open postings per tracked board

Divided by the boards read that day, so adding companies to the list does not read as a hiring surge.

Table view

Day	Boards	Open	Per board	New	Companies
2026-07-15	7	511	73.0	3	7
2026-07-16	7	510	72.9	14	7
2026-07-17	7	520	74.3	22	7
2026-07-18	9	539	59.9	2	9
2026-07-19	9	539	59.9	0	9
2026-07-20	9	537	59.7	12	9
2026-07-21	9	536	59.6	8	9
2026-07-22	10	379	37.9	1	14
2026-07-23	10	378	37.8	3	14
2026-07-24	10	378	37.8	3	14
2026-07-25	10	378	37.8	1	14
2026-07-26	10	378	37.8	0	14
2026-07-27	10	376	37.6	3	14
2026-07-28	119	9,015	75.8	116	122
2026-07-29	118	9,074	76.9	229	122
2026-07-30	119	9,108	76.5	181	122
2026-07-31	119	9,168	77.0	194	122
2026-08-01	119	9,140	76.8	28	122
2026-08-02	159	10,409	65.5	1	162
2026-08-03	160	10,467	65.4	150	163
2026-08-04	161	10,508	65.3	213	164
2026-08-05	161	10,577	65.7	251	164
2026-08-07	161	10,666	66.2	416	164
2026-08-08	161	10,562	65.6	52	164
2026-08-09	161	10,563	65.6	9	164
2026-08-10	161	10,574	65.7	95	164
2026-08-14	161	10,691	66.4	949	164
2026-08-15	161	10,709	66.5	41	164
2026-08-16	161	10,706	66.5	3	164
2026-08-31	156	10,844	69.5	2,090	159
2026-09-01	156	10,573	67.8	222	159
2026-09-02	156	10,638	68.2	251	159
2026-09-03	156	10,629	68.1	221	159
2026-09-04	156	10,645	68.2	196	159
2026-09-05	156	10,589	67.9	21	159
2026-09-06	156	10,594	67.9	10	159
2026-09-07	156	10,597	67.9	75	159
2026-09-08	156	10,638	68.2	209	159
2026-09-09	156	10,653	68.3	244	159
2026-09-10	156	10,678	68.4	195	159
2026-09-11	156	10,737	68.8	234	159
2026-09-12	156	10,700	68.6	32	159
2026-09-13	156	10,698	68.6	6	159
2026-09-14	156	10,774	69.1	227	159
2026-09-15	155	10,800	69.7	249	158
2026-09-16	155	10,811	69.7	252	158
2026-09-17	155	10,826	69.8	221	158
2026-09-18	155	10,838	69.9	215	158
2026-09-19	155	10,796	69.7	23	158
2026-09-20	155	10,782	69.6	5	158
2026-09-21	155	10,781	69.6	179	158
2026-09-22	155	10,784	69.6	179	158
2026-09-23	155	10,806	69.7	224	158
2026-09-24	154	10,830	70.3	237	157
2026-09-25	153	10,806	70.6	202	156
2026-09-26	153	10,763	70.3	33	156
2026-09-27	153	10,762	70.3	5	156
2026-09-28	153	10,705	70.0	198	156
2026-09-29	153	10,693	69.9	231	156
2026-09-30	153	10,676	69.8	211	156
2026-10-01	152	10,625	69.9	216	155
2026-10-02	152	10,672	70.2	206	155
2026-10-03	151	10,538	69.8	15	154
2026-10-04	151	10,544	69.8	12	154
2026-10-06	150	10,724	71.5	454	153
2026-10-07	148	10,494	70.9	201	151
2026-10-08	149	10,508	70.5	218	152
2026-10-09	149	10,514	70.6	198	152
2026-10-10	148	10,468	70.7	13	151
2026-10-11	148	10,466	70.7	6	151
2026-10-12	148	10,436	70.5	109	151
2026-10-13	206	15,988	77.6	303	206
2026-10-14	206	15,852	77.0	329	206
2026-10-15	207	15,915	76.9	392	207
2026-10-16	208	16,018	77.0	355	208
2026-10-17	207	15,947	77.0	33	207
2026-10-18	207	15,954	77.1	13	207
2026-10-19	207	15,903	76.8	277	207
2026-10-20	208	15,919	76.5	282	208
2026-10-21	208	15,969	76.8	347	208
2026-10-22	208	15,985	76.9	345	208
2026-10-23	208	15,943	76.6	307	208
2026-10-24	208	15,865	76.3	31	208
2026-10-25	208	15,865	76.3	7	208
2026-10-26	209	15,811	75.7	271	209
2026-10-27	211	15,809	74.9	301	211
2026-10-28	211	15,711	74.5	292	211
2026-10-29	211	15,738	74.6	336	211
2026-10-30	210	15,710	74.8	277	211
2026-10-31	210	15,629	74.4	18	211
2026-11-01	210	15,628	74.4	11	211
2026-11-02	211	15,482	73.4	241	211
2026-11-03	211	15,687	74.3	257	211
2026-11-04	211	15,380	72.9	59	211

How long a posting stays up

Kaplan-Meier: still-open postings count as censored, not as closed.

Table view

Days up	Still up	At risk	Closed that day
0	97.9%	16,629	345
1	96.4%	16,202	251
2	94.2%	15,695	365
3	92.9%	15,105	209
4	91.9%	14,884	150
5	90.8%	14,711	174
6	89.4%	14,266	233
7	88.1%	13,714	185
8	87.1%	13,260	163
9	85.9%	12,842	177
10	84.9%	12,436	137
11	83.7%	12,295	173
12	82.5%	12,094	182
13	81.1%	11,656	196
14	79.7%	11,181	193
15	78.8%	10,708	115
16	77.8%	10,364	137
17	76.6%	10,002	145
18	75.7%	9,826	125
19	74.6%	9,670	138
20	73.6%	9,265	127
21	72.4%	8,843	138
22	71.5%	8,486	103
23	70.6%	8,174	113
24	69.5%	7,994	126
25	68.6%	7,861	92
26	67.7%	7,759	105
27	66.4%	7,511	143
28	65.4%	7,219	112
29	64.7%	6,962	76
30	63.6%	6,607	109
31	62.7%	6,498	93
32	61.8%	6,395	89
33	60.9%	6,284	98
34	59.8%	6,056	105
35	59.2%	5,824	56
36	58.1%	5,655	113
37	57.0%	5,420	100
38	56.1%	5,221	83
39	55.2%	5,137	78
40	54.2%	5,044	89
41	53.2%	4,847	93
42	52.2%	4,628	87
43	50.9%	4,414	107
44	50.0%	4,206	80
45	48.8%	4,043	94
46	47.8%	3,947	79
47	47.0%	3,854	70
48	45.9%	3,681	80
49	45.1%	3,520	61
50	44.3%	3,344	60
51	43.4%	3,174	68
52	42.5%	3,012	64
53	41.4%	2,946	71
54	40.8%	2,862	46
55	39.6%	2,723	79
56	38.8%	2,556	50
57	38.1%	2,417	44
58	37.4%	2,305	44
59	36.9%	2,238	27
60	36.0%	2,205	54
61	35.3%	2,141	41
62	34.3%	2,030	57
63	33.9%	1,892	26
64	33.2%	1,785	35
65	32.8%	1,681	21
66	32.5%	996	10
67	31.5%	986	28
68	30.9%	958	18
69	29.8%	940	36
70	29.3%	904	15
71	28.8%	889	15
72	28.1%	874	20
73	27.6%	854	17
74	26.9%	837	19
75	26.4%	818	16
76	26.1%	802	10
77	25.8%	792	8
78	25.6%	784	6
79	25.4%	778	6
80	25.1%	772	9
81	24.6%	761	16
82	24.1%	736	14
83	23.8%	516	8
84	23.2%	508	12
85	22.0%	496	26
86	21.6%	470	7
87	21.2%	431	9
88	20.6%	417	12
89	20.2%	392	7
90	19.8%	294	6
91	19.4%	288	6
92	19.1%	229	3
93	18.7%	193	4
94	18.2%	161	5
95	17.8%	156	3
96	17.7%	145	1
97	17.7%	107	0
98	17.4%	69	1
99	17.4%	32	0
100	17.4%	10	0
101	15.5%	9	1
102	13.6%	8	1
104	13.6%	7	0

Data roles by family

Open postings at the tracked companies, grouped by title.

Table view

Family	Open	Companies	Remote	Hybrid	On-site	Unstated
Other	14,824	211	2,758	1,335	892	9,839
Analytics	197	60	60	12	15	110
Data science	106	38	23	5	1	77
Data engineering	91	41	16	17	2	56
ML engineering	80	29	4	3	0	73
Analytics engineering	40	25	12	0	1	27
Data architecture	19	5	1	0	0	18
Data governance	13	7	0	2	0	11
Business intelligence	10	10	0	3	1	6

What data roles ask for

Share of tracked companies' open data-role postings naming each technology.

Table view

Technology	Category	Share	Postings	Companies
SQL	Language	65.3%	363	88
Python	Language	63.3%	352	86
Machine learning	AI	39.7%	221	63
Experimentation	Practice	25.9%	144	54
dbt	Transformation	25.4%	141	66
AWS	Cloud	24.8%	138	47
Spark	Processing	23.7%	132	38
Data modeling	Practice	23.2%	129	55
Snowflake	Warehouse	22.3%	124	58
LLMs	AI	21.2%	118	50
Databricks	Lakehouse	19.2%	107	30
Airflow	Orchestration	18.9%	105	44
GCP	Cloud	16.7%	93	38
Azure	Cloud	16.0%	89	29
CI/CD	Practice	13.8%	77	35
BigQuery	Warehouse	13.3%	74	40
Tableau	BI	13.1%	73	37
Looker	BI	11.5%	64	33
Git	Practice	10.3%	57	32
Power BI	BI	9.7%	54	34
Excel	BI	9.0%	50	17
Kafka	Streaming	8.8%	49	22
Kubernetes	Infrastructure	7.4%	41	22
Docker	Infrastructure	6.3%	35	21
Terraform	Infrastructure	5.0%	28	18
Redshift	Warehouse	4.9%	27	18
Scala	Language	4.9%	27	9
Java	Language	4.3%	24	12
Fivetran	Ingestion	4.0%	22	17
Dagster	Orchestration	2.5%	14	12
Flink	Streaming	2.5%	14	14
Postgres	Database	2.3%	13	13
Data contracts	Practice	2.2%	12	8
ClickHouse	Warehouse	1.4%	8	7
Metabase	BI	1.4%	8	7
Airbyte	Ingestion	1.1%	6	5

Where the postings come from

Which applicant-tracking systems the tracked companies post on.

Table view

ATS	Open postings	Boards
Greenhouse	8,936	70
Ashby	2,625	77
Lever	1,573	20
Workable	289	10
BambooHR	164	25
Other	1,793	9

A curated list of company boards, not a sample of the job market; counted before any filter. A figure covering fewer than 5 companies is withheld. Snapshot taken 2026-11-04 00:17 UTC.

02From API to inbox

The dbt DAG, generated from manifest.json when this page was built. Click any node to open it in dbt docs. The AI models in silver are typed stubs on this DuckDB build, because the functions they call exist only in BigQuery.

03Decisions that shaped it

Medallion, dbt-native

Three zones map one-to-one onto staging → intermediate → marts, each in its own BigQuery dataset. Raw is append-only and every transform rebuilds from it, so a rule change applies retroactively with zero migrations.

ADR-0010 ADR-0014

One SQL codebase, two warehouses

Every model runs on DuckDB for development and CI — with no secrets — and on BigQuery in production. Dialect differences live in adapter.dispatch macros, including the prod-only type traps that naive cross-database macros hide.

ADR-0002

Lifecycle, not snapshots

The source APIs have no date filters, so novelty is derived, never requested: first_seen_at, last_seen_at and is_active come from the append-only landing, and a staleness rule ages out a dropped board instead of leaving zombie postings.

ADR-0016 ADR-0008

Rules are data

Locations, technologies and titles are dbt seeds, word-boundary matched and regex-safe (C++ and .NET work). Only location removes a row; everything else ranks, so recall stays high until a model can judge fit.

ADR-0015 ADR-0023 ADR-0028

Security is structural

No secret value exists anywhere in the repo. BigQuery auth is Workload Identity Federation, so there is no key file to leak; workflows are split by permission scope, actions are SHA-pinned, and gitleaks backstops every commit.

ADR-0007 ADR-0011

AI that orders, never filters

LLM extraction and scoring run as incremental dbt models against BigQuery's in-SQL AI functions, keyed on a content hash so each text is billed once. The score sorts delivery and can never drop a posting: if it fails, the V1 result still ships.

ADR-0004 ADR-0020 ADR-0024

04Built to fail loudly

A scheduled pipeline's worst failures produce no signal at all. An invalid workflow file never starts, so nothing emails. A suspended schedule is silent. A run that lands almost nothing still exits zero. Each of these has its own gate here: workflow linting, a dead-man's switch, volume warnings and a source-freshness check.

The failures that look like health →

05Even the prose is tested

Docs drift silently: a file is renamed and the sentence pointing at it still reads fine. A checker in CI verifies every path, make target, model name, heading anchor and ADR number the docs mention, and lets a sentence assert a count against the code — so a number cannot rot under an edit.

All quality gates ↗

06Working on LLM scoring vs. embeddings

Two ways to rank a posting against a resume corpus, built to answer the same question by different means. Choosing between them without measuring is how a project ends up carrying both forever, so neither replaces the keyword score until labels say it should.

LLM fit score running

Reads the posting's requirements against the corpus and returns a score from 1 to 5. It reads polarity, so "we sponsor visas" and "we do not" score differently. The dearer half: one generation per posting, billed once per text by the content-hash guard.

Embedding similarity built, off

Finds the single resume bullet closest to the requirements and names it, so a ranking explains itself. The cheaper half, but blind to polarity: those two visa sentences are nearly the same vector, so location limits and people management stay with the LLM.

Human labels. Labels from an LLM cannot settle it: grading a scorer against its predecessor rewards repeating the predecessor's mistakes.
Precision@k over those labels for all three rankings, the keyword baseline included.
Embeddings on, both rankings live side by side for a month, then the loser goes — with its column and its cost.

How V2 works →

07Decision log 29 records, newest first

Job postings in.Tested, ranked data out.

01Hiring at the tracked companies rebuilt after every scheduled run

Open postings per tracked board

How long a posting stays up

Data roles by family

What data roles ask for

Where the postings come from

02From API to inbox

03Decisions that shaped it

Medallion, dbt-native

One SQL codebase, two warehouses

Lifecycle, not snapshots

Rules are data

Security is structural

AI that orders, never filters

04Built to fail loudly

05Even the prose is tested

06Working on LLM scoring vs. embeddings

LLM fit score running

Embedding similarity built, off

07Decision log 29 records, newest first

Job postings in.
Tested, ranked data out.