I built the dbt agent-readiness audit to answer a practical question: if you point an analytics agent at your dbt project as it stands, what does the project establish clearly enough for the agent to answer correctly, and where does it leave the agent guessing? The tool started as a Claude Code skill that I shared with practitioners. I then ran it on 13 public projects covering 5,284 models and manually checked every finding used in this article against a pinned commit.
The audit reads model SQL, schema files, the doc blocks behind {{ doc() }}, and repository READMEs. It checks the evidence that names, descriptions, tests, joins, and declared grain provide. For these runs I supplied no warehouse access and no compiled SQL.
The projects include a 2024 public mirror of GitLab’s former analytics repository, Mattermost’s data warehouse, Cal-ITP’s live public transit project, and packages such as Stripe, GA4, and Snowplow. Several are mature and extensively documented. I could inspect them only because their maintainers chose to work in public.
This is a source-level case study. It identifies unresolved assumptions and direct metadata errors in repository evidence. It is not an agent benchmark or a prevalence estimate, and it does not measure how often an agent will make a particular mistake.
One name, two populations
Mattermost’s warehouse tracks per-server product usage, and count_registered_users appears in seven models across the intermediate, mart, and report layers. Two fact tables define it one word apart.
fct_active_users (_product__models.yml):
- name: count_registered_users
description: Total number of users, including deleted users. Reported by mattermost server.fct_board_activity (_product__board__models.yml):
- name: count_registered_users
description: Total number of users, excluding deleted users. Reported by mattermost server.Including, excluding. The same column name covers opposite populations. Both descriptions may be correct for their tables, but the repository does not identify an owner for the unqualified concept or say whether the measures are comparable. A repository-grounded agent can select the wrong population.
The model feeding fct_active_users, int_server_active_days_spined, adds another unresolved case. It is documented as including deleted users, and its SQL falls back to a third field when both registered-user sources are empty (int_server_active_days_spined.sql):
coalesce(activity.count_registered_users, legacy_activity.count_registered_users, d.count_users, 0) as count_registered_userscount_users is a different field, and nothing here establishes that it preserves the same deleted-user semantics. The fallback’s meaning is undocumented.
A uniqueness test on a different key
Cal-ITP, California’s public transit data platform, tests int_gtfs_quality__organization_dataset_map as unique on (date, organization_key, gtfs_dataset_key) (_int_gtfs_quality.yml):
data_tests:
- dbt_utils.unique_combination_of_columns:
arguments:
combination_of_columns:
- date
- organization_key
- gtfs_dataset_keyThe downstream route-change model joins it on (date, organization_source_record_id) instead (fct_monthly_route_id_changes.sql):
LEFT JOIN organization_dataset_map AS orgs ON (reports_index.date_start = orgs.date)
AND (reports_index.organization_source_record_id = orgs.organization_source_record_id)The test guarantees uniqueness for one tuple. It says nothing about uniqueness for the key the join actually uses, so the repository leaves open whether multiple datasets can match one report row. The next step expands datasets into routes, and that may be intentional. Proving an overcount would require duplicated logical routes in the warehouse or an explicit output-grain contract that this model violates. The source finding is narrower: no declared uniqueness guarantee covers the actual join key.
The description points at the wrong column
GitLab’s headcount report counts separations two ways, voluntary and involuntary, the difference between someone quitting and being let go. Both columns carry the same description (schema.yml):
- name: rolling_12_month_voluntary_separations
description: Provides the total number of the employees separated voluntarily for the current month and previous 11 months.
- name: rolling_12_month_involuntary_separations
description: Provides the total number of the employees separated voluntarily for the current month and previous 11 months.The second description is wrong. It defines involuntary separation as voluntary. For a question about people who were let go, one of the repository signals available for choosing a column points at the opposite concept.
Cal-ITP does the same in its transit metrics. One model exposes both a count, n_tu_trips, and a ratio, pct_tu_trips, of trips carrying a real-time feed (tu is trip updates), and documents both with the count’s description (_mart_gtfs_fcts.yml):
- name: n_tu_trips
description: '{{ doc("column_n_tu_trips") }}'
- name: pct_tu_trips
description: '{{ doc("column_n_tu_trips") }}'The ratio points to the count’s definition. For a question about the share of trips with a real-time feed, the ratio’s description points at a raw count.
Both errors are one-line fixes. They are hard to spot because the descriptions exist and look complete until two neighboring fields are compared. From the descriptions alone, the intended column is unresolved.
Coverage hides where meaning is missing
Mattermost’s intermediate/sales/hightouch directory contains four models that compute ARR and seat counts (all four). The directory has no schema file, so that revenue-related logic carries no model or column descriptions.
Cal-ITP shows a different coverage problem. In dim_annual_service_mode_time_periods, nineteen quality flags inherit one description through a YAML anchor (_mart_ntd.yml):
- &questionable_data
name: questionable_data
description: '{{ doc("ntd_questionable_data") }}'
...
- <<: *questionable_data
name: mode_voms_questionable
- <<: *questionable_data
name: vehicle_miles_questionable
- <<: *questionable_data
name: deadhead_miles_questionableAll nineteen columns count as documented. They still share one sentence.
A semantic layer governs part of this surface. MetricFlow defines metrics, entities, and dimensions, with entities as the join keys between semantic models. Two projects in the audit define that layer while leaving ordinary dbt columns undocumented: zero descriptions across 63 columns in full-funnel and zero across 137 in jaffle_corp. In full-funnel, the model description states its grain, then the columns carry tests and no descriptions (schema.yml):
- name: fct_channel_performance
description: >
Aggregated spend, revenue, orders, ROAS, and CAC per (date × channel).
Grain is daily × channel after the time-dimension refactor — channel
alone is not unique.
columns:
- name: date
tests:
- not_null
- name: channel
tests:
- not_null
- name: total_spend
tests:
- not_nullInside its modeled surface, MetricFlow governs measures, dimensions, and entity joins. Outside it, an agent has only the context provided by the underlying models and columns. The semantic layer reduces ambiguity for what it models. It does not supply meaning for every ordinary column an agent may need to query.
What source alone cannot establish
My audit got its most obvious findings wrong. I manually checked 35 GitLab flags for broken references or undefined columns. All 35 were false positives.
The causes were structural. A parser treated a date unit inside DATEADD as a column. It could not see columns generated by macros and Jinja loops. It flagged package models whose references resolve after dbt deps. In the Stripe package, 218 flags traced to one macro that builds its column list at compile time (stg_stripe__balance_transaction.sql):
{{
fivetran_utils.fill_staging_columns(
source_columns=adapter.get_columns_in_relation(ref('stg_stripe__balance_transaction_tmp')),
staging_columns=get_balance_transaction_columns()
)
}}I changed the tool after those checks. It now recognizes date-part keywords as units, treats Fivetran’s column-generating macros as unresolvable, skips those models, and holds back unresolved references when declared packages are absent. The known GitLab class fell from 35 flags to zero. Stripe fell from 218 to zero. A skipped model remains a blind spot, not evidence that the project is clean.
Warehouse-aware macros require compiled SQL. A parsed manifest.json describes the project, but it does not include compiled SQL for every node. dbt compile can require a data platform connection when macros run introspective queries. An audit can instead consume compiled artifacts produced by CI. If neither is available, raw source cannot expose the generated columns.
Source also sees code, not rows. The messiest project in the set was a RevOps warehouse whose author had deliberately seeded duplicate domains and duplicate emails. It came back nearly clean because the mess lived in the data, while the code handling it was correct.
Source can prove direct description errors and show that a join key lacks a declared uniqueness guarantee. It cannot decide whether the warehouse rows make every unresolved assumption true.
What the agent still needs
For the models an analytics agent is expected to query, the repository needs two kinds of evidence:
- Semantic: what each critical column means, what grain each model has, which table owns each business concept, and which joins are valid.
- Verification: tests on the keys actually used, compiled artifacts where source is dynamic, warehouse checks for data-dependent claims, and a way for the agent to abstain when evidence conflicts.
That does not require the same documentation depth on every model. It does require enough evidence on the path the agent queries, plus permission to stop when the repository cannot settle the question.
The audit is open source and runs on a dbt project’s source without warehouse credentials: github.com/GetCassis/dbt-agent-readiness. If you run it on yours, I’d genuinely like to compare notes on what it finds.
Appendix: the projects
Every example above is quoted verbatim from a public repository, under its open-source license, linked to a pinned commit.
Model counts are from the audit’s own inventory pass. The jaffle_corp repo holds several dbt projects; the count is its platform project.
The GitLab project is a public mirror of GitLab’s former internal analytics repo, MIT-licensed and last active in 2024, not GitLab’s current repository. Cal-ITP is actively maintained, which is why it appears alongside the older mirror.