Skip to main content

dbt Semantic Layer quickstart

📌

The dbt Semantic Layer is currently available in Public Preview for multi-tenant dbt Cloud accounts hosted in North America. If you log in via https://cloud.getdbt.com/, you can access the Semantic Layer. If you log in with another URL, the dbt Semantic Layer will be available in the future.

For more info, review the Prerequisites, Public Preview, and Product architecture sections.

Public Preview​

We're excited to announce the dbt Semantic Layer is currently available for Public Preview, which means:

— Who? The dbt Semantic Layer is open to all dbt Cloud tiers (Developer, Team, and Enterprise) during Public Preview. Review Product architecture for more info on plan availability.

  • Team and Enterprise accounts will be able to set up the Semantic Layer and Metadata API in the integrated partner tool to import metric definition.
  • Developer accounts will be able to query the Proxy Server using SQL, but will not be able to browse dbt metrics in external tools, which requires access to the Metadata API.

— What? Public Previews provide early access to new features. The Semantic Layer is stable and you can use it for production deployments, but there may still be some planned additions and modifications to product behaviors before moving to General Availability. We may also introduce new functionality that is not backwards compatible. dbt Labs provides support, and relevant service level objectives (SLOs) apply. We will introduce pricing for the dbt Semantic Layer alongside the General Available (GA) release (future GA date to be announced).

— When? Public Preview will end once the dbt Semantic Layer is available for GA. After GA, the dbt Semantic Layer will only be available to dbt Cloud Team and Enterprise plans.

— Where? Public Preview is enabled at the account level so you don’t need to worry about enabling it per user.

Introduction​

To try out the features of the dbt Semantic Layer, you first need to have a dbt project set up. This quickstart guide will lay out the following steps, and recommends a workflow that demonstrates some of its essential features:

  • Install dbt metrics package
  • Define metrics
  • Query, and run metrics
  • Configure the dbt Semantic Layer

Prerequisites​

To use the dbt Semantic Layer, you’ll need to meet the following:

Considerations

Some important considerations to know about using the dbt Semantic Layer during the Public Preview:

  • Support for Snowflake data platform only (additional data platforms coming soon)
  • Support for the deployment environment only (development experience coming soon)
📌

New to dbt or metrics? Check out our quickstart guide to build your first dbt project! If you'd like to define your first metrics, try our Jaffle Shop example project.

Installing dbt metrics package​

The dbt Semantic Layer supports the calculation of metrics by using the dbt metrics package. You can install the dbt metrics package in your dbt project by copying the below code blocks.

  1. Paste the dbt metrics package code in your packages.yml file.
  2. Run the dbt deps command to install the package.
  3. If you see a successful result, you have now installed the dbt metrics package successfully!
  4. If you have any errors during the dbt deps command run, review the system logs for more information on how to resolve them. Make sure you use a dbt metrics package that’s compatible with your dbt environment version.
Running dbt deps in the dbt Cloud IDERunning dbt deps in the dbt Cloud IDE

Design and define metrics​

Review our helpful metrics video below, which explains what metrics are, why they're important and how you can get started:

Design metrics​

To read about best practices on structuring and organizing your metrics, review our How to design and structure dbt metrics: Recommendations for getting started blog post first.

Define metrics​

Now that you've organized your metrics folder and files, you can define your metrics in .yml files nested under a metrics key.

  1. Add the metric definitions found in the Jaffle Shop example to your dbt project. For example, to add an expenses metric, reference the following metrics you can define directly in your metrics folder:

  1. Click Save and then Compile the code.

  2. Commit and merge the code changes that contain the metric definitions.

  3. If you'd like to further design and define your own metrics, review the following documentation:

Develop and query metrics​

You can dynamically develop and query metrics directly in dbt and verify their accuracy before running a job in the deployment environment by using the metrics.calculate and metrics.develop macros.

To understand when and how to use the macros above, review dbt metrics and make sure you install the dbt_metrics package first before using the above macros.

📌

Note: You will need access to dbt Cloud and the dbt Semantic Layer from your integrated partner tool of choice.

Run your production job​

Once you’ve defined metrics in your dbt project, you can perform a job run in your deployment environment to materialize your metrics. The deployment environment is only supported for the dbt Semantic Layer at this moment.

  1. Go to Deploy in the navigation and select Jobs to re-run the job with the most recent code in the deployment environment.
  2. Your metric should appear as a red node in the dbt Cloud IDE and dbt directed acyclic graphs (DAG).
DAG with metrics appearing as a red nodeDAG with metrics appearing as a red node

What’s happening internally?

  • Merging the code into your main branch allows dbt Cloud to pull those changes and builds the definition in the manifest produced by the run.
  • Re-running the job in the deployment environment helps materialize the models, which the metrics depend on, in the data platform. It also makes sure that the manifest is up to date.
  • Your dbt Metadata API pulls in the most recent manifest and allows your integration information to extract metadata from it.

Set up dbt Semantic Layer​

Before you continue with the following steps, you must have a multi-tenant dbt Cloud account hosted in North America.

  • Team and Enterprise accounts can set up the Semantic Layer and Metadata API in the integrated partner tool to import metric definition.
  • Developer accounts can query the Proxy Server using SQL but won't be able to browse dbt metrics in external tools, which requires access to the Metadata API.

You can set up the dbt Semantic Layer in dbt Cloud at the environment level by following these steps:

  1. Login to your dbt Cloud account
  2. Go to Account Settings, and then Service Tokens to create a new create a service account API token. Save your token somewhere safe.
  3. Assign permissions to service account tokens depending on the integration tool you choose. You can review the integration partner documentation to determine the permission sets you need to assign.
  4. Go to Deploy and then Environment, and select your Deployment environment.
  5. Click on Settings on the top right side of the page.
  6. Click Edit on the top right side of the page.
  7. Select dbt version 1.2 or higher.
  8. Toggle the Semantic Layer On.
  9. Copy the full proxy server URL (like https://eagle-hqya7.proxy.cloud.getdbt.com) to connect to your integrated partner tool.
  10. Use the URL in the data source configuration of the integrated partner tool.
  11. Use the data platform login credentials that make sense for how the data is consumed.
📌

Note - It is not recommended that you use your dbt Cloud credentials due to elevated permissions. Instead, you can use your specific integration tool permissions.

  1. Set up the Metadata API (Team and Enterprise accounts only) in the integrated partner tool to import the metric definitions. The integrated partner tool will treat the dbt Server as another data source (like a data platform). This requires:
  • The account ID, environment ID, and job ID (visible in the job URL)
  • An API service token with job admin and metadata permissions
  • Add the items above to the relevant fields in your integration tool

Troubleshooting​

If you're encountering some issues when defining your metrics or setting up the dbt Semantic Layer, check out a list of answers to some of the questions or problems you may be experiencing.

How are you storing my data?
The dbt Semantic Layer does not store, or cache, or log your data. On each query to the Semantic Layer, the resulting data passes through dbt Cloud servers where it is never stored, cached, or logged. The data from your data platform gets routed through dbt Cloud servers, to your connecting data tool.
Is the dbt Semantic Layer open source?
Some components of the dbt Semantic Layer are open source like dbt-core, the dbt_metrics package, and the BSL licensed dbt-server. The dbt Proxy Server (what is actually compiling the dbt code) and the Metadata API are not open source.

During Public Preview, the dbt Semantic Layer is open to all dbt Cloud tiers (Developer, Team, and Enterprise).

  • dbt Core users can define metrics in their dbt Core projects and calculate them using macros from the metrics package. To use the dbt Semantic Layer integrations, you will need to have a dbt Cloud account.


  • Developer accounts will be able to query the Proxy Server using SQL, but will not be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API.


  • Team and Enterprise accounts will be able to set up the Semantic Layer and Metadata API in the integrated partner tool to import metric definitions.
The dbt_metrics_calendar_table does not exist or is not authorized?
All metrics queries are dependent on either the dbt_metrics_calendar_table or a custom calendar set in the users dbt_project.yml. If you have not created this model in the database, these queries will fail and you'll most likely see the following error message:Object DATABASE.SCHEMA.DBT_METRICS_DEFAULT_CALENDAR does not exist or not authorized.
Fix:
  • If developing locally, run dbt run --select dbt_metrics_default_calendar

  • If you are using this in production, make sure that you perform a full dbt build or dbt run. If you are running specific selects in your production job, then you will not create this required model.
Ephemeral Models - Object does not exist or is not authorized
Metrics cannot be defined on ephemeral models because we reference the underlying table in the query that generates the metric so we need the table/view to exist in the database. If your table/view does not exist in your database, you might see this error message:Object 'DATABASE.SCHEMA.METRIC_MODEL_TABLE' does not exist or not authorized.
Fix:
  • You will need to materialize the model that the metric is built on as a table/view/incremental.
Mismatched Versions - metric type is ‘’
If you’re running dbt_metrics ≥v0.3.2 but have dbt-core version ≥1.3.0, you’ll likely see these error messages:
  • Error message 1: The metric NAME also references ... but its type is ''. Only metrics of type expression can reference other metrics.
  • Error message 2: Unknown aggregation style: > in macro default__gen_primary_metric_aggregate (macros/sql_gen/gen_primary_metric_aggregate.sql)
The reason you're experiencing this error is because we changed the type property of the metric spec in dbt-core v1.3.0. The new name is calculation_method and the package reflects that new name, so it isn’t finding any type when we try and run outdated code on it.
Fix:

Next steps​

Are you ready to define your own metrics and bring consistency to data consumers? Review the following documents to understand how to structure, define, and query metrics, and set up the dbt Semantic Layer:

0
Edit this page