Skip to main content

Move from dbt Core to dbt Cloud: Get started

Updated Total estimated time: 3-4 hours
Migration
dbt Core
dbt Cloud
Intermediate
Menu

    Introduction​

    Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service.

    Explore our 3-part-guide series on moving from dbt Core to dbt Cloud. This series is ideal for users aiming for streamlined workflows and enhanced analytics:

    Guide
    Information
    Audience
    Move from dbt Core to dbt Cloud: What you need to knowUnderstand the considerations and methods needed in your move from dbt Core to dbt Cloud.Team leads
    Admins
    Move from dbt Core to dbt Cloud: Get startedLearn the steps needed to move from dbt Core to dbt Cloud.Developers
    Data engineers
    Data analysts
    Move from dbt Core to dbt Cloud: Optimization tipsLearn how to optimize your dbt Cloud experience with common scenarios and useful tips.Everyone

    dbt Cloud is the fastest and most reliable way to deploy dbt. It enables you to develop, test, deploy, and explore data products using a single, fully managed service. It also supports:

    Learn more about dbt Cloud features.

    dbt Core is an open-source tool that enables data teams to define and execute data transformations in a cloud data warehouse following analytics engineering best practices. While this can work well for ‘single players’ and small technical teams, all development happens on a command-line interface, and production deployments must be self-hosted and maintained. This requires significant, costly work that adds up over time to maintain and scale.

    What you'll learn​

    This guide outlines the steps you need to take to move from dbt Core to dbt Cloud and highlights the necessary technical changes:

    • Account setup: Learn how to create a dbt Cloud account, invite team members, and configure it for your team.
    • Data platform setup: Find out about connecting your data platform to dbt Cloud.
    • Git setup: Learn to link your dbt project's Git repository with dbt Cloud.
    • Developer setup: Understand the setup needed for developing in dbt Cloud.
    • Environment variables: Discover how to manage environment variables in dbt Cloud, including their priority.
    • Orchestration setup: Learn how to prepare your dbt Cloud environment and jobs for orchestration.
    • Models configuration: Get insights on validating and running your models in dbt Cloud, using either the dbt Cloud IDE or dbt Cloud CLI.
    • What's next?: Summarizes key takeaways and introduces what to expect in the following guides.

    Prerequisites​

    Account setup​

    This section outlines the steps to set up your dbt Cloud account and configure it for your team.

    1. Create your dbt Cloud account.

    2. Provide user access and invite users to your dbt Cloud account and project.

    3. Configure Single Sign-On (SSO) or Role-based access control (RBAC) for easy and secure access. enterprise

      • This removes the need to save passwords and secret environment variables locally.

    Additional configuration​

    Explore these additional configurations for performance and reliability improvements:

    1. In Account settings, enable partial parsing to only reparse changed files, saving time.

    2. In Account settings, enable Git repo caching for job reliability & third-party outage protection. enterprise

    Data platform setup​

    This section outlines the considerations and methods to connect your data platform to dbt Cloud.

    1. In dbt Cloud, set up your data platform connections and environment variables. dbt Cloud can connect with a variety of data platform providers including:

    2. You can verify your data platform connections by clicking the Test connection button in your deployment and development credentials settings.

    Additional configuration​

    Explore these additional configurations to optimize your data platform setup further:

    1. Use OAuth connections, which enables secure authentication using your data platform’s SSO. enterprise

    Git setup​

    Your existing dbt project source code should live in a Git repository. In this section, you will connect your existing dbt project source code from Git to dbt Cloud.

    1. Ensure your dbt project is in a Git repository.

    2. In Account settings, select Integrations to connect your Git repository to dbt Cloud:

      • (Recommended) Connect with one of the native integrations in dbt Cloud (such as GitHub, GitLab, and Azure DevOps).

        This method is preferred for its simplicity, security features (including secure OAuth logins and automated workflows like CI builds on pull requests), and overall ease of use.

      • Import a Git repository from any valid Git URL that points to a dbt project.

    Additional configuration​

    Explore these additional configurations to optimize your Git setup further:

    1. Log into dbt Cloud using OAuth connections to integrate with your source code platform. It automatically links to the repository using one of the native integrations set at the account level. enterprise

      Set up groups for dbt project access with those configured for repository access to streamline permissions.

    Developer setup​

    This section highlights the development configurations you’ll need for your dbt Cloud project. The following categories are covered in this section:

    dbt Cloud environments​

    The most common data environments are production, staging, and development. The way dbt Core manages environments is through target, which are different sets of connection details.

    dbt Cloud environments go further by:

    • Integrating with features such as job scheduling or version control, making it easier to manage the full lifecycle of your dbt projects within a single platform.
    • Streamlining the process of switching between development, staging, and production contexts.
    • Making it easy to configure environments through the dbt Cloud UI instead of manually editing the profiles.yml file. You can also set up or customize target names in dbt Cloud.
    • Adding profiles.yml attributes to dbt Cloud environment settings with Extended Attributes.
    • Using Git repo caching to protect you from third-party outages, Git auth failures, and more. enterprise

    Initial setup steps​

    1. Set up development environment — Set up your development environment and development credentials. You’ll need this to access your dbt project and start developing.

    2. dbt Core version — In your dbt Cloud environment and credentials, use the same dbt Core version you use locally. You can run dbt --version in the command line to find out which version of dbt Core you’re using.

      • When using dbt Core, you need to think about which version you’re using and manage your own upgrades. When using dbt Cloud, leverage Keep on latest version so you don’t have to.

    3. Connect to your data platform — When using dbt Cloud, you can connect to your data platform directly in the UI.

      • Each environment is roughly equivalent to an entry in your profiles.yml file. This means you don't need a profiles.yml file in your project.

    4. Development tools — Set up your development workspace with the dbt Cloud CLI (command line interface or code editor) or dbt Cloud IDE (browser-based) to build, test, run, and version control your dbt code in your tool of choice.

      • If you've previously installed dbt Core, the dbt Cloud CLI installation doc has more information on how to install the dbt Cloud CLI, create aliases, or uninstall dbt Core for a smooth transition.

    Additional configuration​

    Explore these additional configurations to optimize your developer setup further:

    1. Custom target names — Using custom target.names in your dbt projects helps identify different environments (like development, staging, and production). While you can specify the custom target.name values in your developer credentials or orchestration setup, we recommend using environment variables as the preferred method. They offer a clearer way to handle different environments and are better supported by dbt's partial parsing feature, unlike using {{ target }} logic which is meant for defining the data warehouse connection.

    dbt Cloud commands​

    1. Review the dbt commands supported for dbt Cloud development. For example, dbt init isn’t needed in dbt Cloud as you can create a new project directly in dbt Cloud.

    Environment variables​

    This section will help you understand how to set up and manage dbt Cloud environment variables for your project. The following categories are covered:

    In dbt Cloud, you can set environment variables in the dbt Cloud user interface (UI). Read Set up environment variables for more info.

    In dbt Core, environment variables, or the env_var function, are defined manually by the developer or within the external application running dbt.

    Environment variables in dbt Cloud​

    • dbt Cloud environment variables must be prefixed with DBT_ (including DBT_ENV_CUSTOM_ENV_ or DBT_ENV_SECRET_).
    • If your dbt Core environment variables don’t follow this naming convention, perform a “find and replace” in your project to make sure all references to these environment variables contain the proper naming conventions.
    • dbt Cloud secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with DBT_ENV_SECRET_ to obscure them in logs and the UI.
    Setting project level and environment level valuesSetting project level and environment level values

    dbt Cloud environment variables order of precedence​

    Environment variables in dbt Cloud are managed with a clear order of precedence, allowing users to define values at four levels (highest to lowest order of precedence):

    • The job level (job override) or in the IDE for an individual developer (personal override). Highest precedence
    • The environment level, which can be overridden by the job level or personal override.
    • A project-wide default value, which can be overridden by the environment level, job level, or personal override.
    • The optional default argument supplied to the env_var Jinja function in the code. Lowest precedence
    Environment variables order of precedenceEnvironment variables order of precedence

    Set environment variables in dbt Cloud​

    • To set these variables for an entire project or specific environments, navigate to Deploy > Environments > Environment variables tab.
    • To set these variables at the job level, navigate to Deploy > Jobs > Select your job > Settings > Advanced settings.
    • To set these variables at the personal override level, navigate to Profile Settings > Credentials > Select your project > Environment variables.

    Orchestration setup​

    This section outlines the considerations and methods to set up your dbt Cloud environments and jobs for orchestration. The following categories are covered in this section:

    dbt Cloud environments​

    To use the dbt Cloud's job scheduler, set up one environment as the production environment. This is the deployment environment. You can set up multiple environments for different stages of your deployment pipeline, such as development, staging/QA, and production.

    Initial setup steps​

    1. dbt Core version — In your environment settings, configure dbt Cloud with the same dbt Core version.

      • Once your full migration is complete, we recommend upgrading your environments to Keep on latest version to always get the latest features and more. You only need to do this once.

    2. Configure your jobs — Create jobs for scheduled or event-driven dbt jobs. You can use cron execution, manual, pull requests, or trigger on the completion of another job.

    Additional configuration​

    Explore these additional configurations to optimize your dbt Cloud orchestration setup further:

    1. Custom target names — Use environment variables to set a custom target.name for every corresponding dbt Cloud job at the environment level.

    2. dbt commands — Add any relevant dbt commands to execute your dbt Cloud jobs runs.

    3. Notifications — Set up notifications by configuring email and Slack alerts to monitor your jobs.

    4. Monitoring tools — Use monitoring tools like run history, job retries, job chaining, dashboard status tiles, and more for a seamless orchestration experience.

    5. API access — Create API auth tokens and access to dbt Cloud APIs as needed. teamenterprise

    6. dbt Explorer — If you use dbt Explorer and run production jobs with an external orchestrator, ensure your production jobs run dbt run or dbt build to update and view models and their metadata in dbt Explorer. Running dbt compile alone will not update model metadata. In addition, features like column-level lineage also requires catalog metadata produced through running dbt docs generate. teamenterprise

    CI/CD setup​

    Building a custom solution to efficiently check code upon pull requests is complicated. With dbt Cloud, you can enable continuous integration / continuous deployment (CI/CD) and configure dbt Cloud to run your dbt projects in a temporary schema when new commits are pushed to open pull requests.

    This build-on-PR functionality is a great way to catch bugs before deploying to production, and an essential tool for data practitioners.

    1. Set up an integration with a native Git application (such as Azure DevOps, GitHub, GitLab) and a CI environment in dbt Cloud.
    2. Create a CI/CD job to automate quality checks before code is deployed to production.
    3. Run your jobs in a production environment to fully implement CI/CD. Future pull requests will also leverage the last production runs to compare against.

    Model development and discovery​

    In this section, you’ll be able to validate whether your models run or compile correctly in your development tool of choice: The dbt Cloud IDE or dbt Cloud CLI.

    You’ll want to make sure you set up your development environment and credentials.

    1. In your development tool of choice, you can review your dbt project, ensure it's set up correctly, and run some dbt commands:

      • Run dbt compile to make sure your project compiles correctly.
      • Run a few models in the dbt Cloud IDE or dbt Cloud CLI to ensure you’re experiencing accurate results in development.

    2. Once your first job has successfully run in your production environment, use dbt Explorer to view your project's resources (such as models, tests, and metrics) and their data lineage  to gain a better understanding of its latest production state. teamenterprise

    What’s next?​

    Congratulations on completing the first part of your move to dbt Cloud 🎉!

    You have learned:

    • How to set up your dbt Cloud account
    • How to connect your data platform and Git repository
    • How to configure your development, orchestration, and CI/CD environments
    • How to set up environment variables and validate your models

    For the next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud:

    Guide
    Information
    Audience
    Move from dbt Core to dbt Cloud: What you need to knowUnderstand the considerations and methods needed in your move from dbt Core to dbt Cloud.Team leads
    Admins
    Move from dbt Core to dbt Cloud: Get startedLearn the steps needed to move from dbt Core to dbt Cloud.Developers
    Data engineers
    Data analysts
    Move from dbt Core to dbt Cloud: Optimization tipsLearn how to optimize your dbt Cloud experience with common scenarios and useful tips.Everyone
    0