Upgrading dbt versions in Cloud

This Q&A guide should help you figure out what changes you might need to make to successfully upgrade your version of dbt Core in dbt Cloud. As a reminder, we recommend everyone upgrade to the most recent version of dbt, as we will not support running all versions of dbt in Cloud indefinitely. We document which versions of dbt Core we support here.

There aren't many breaking changes between versions, and it may be the case that you don't need to change any code to upgrade to a newer version of dbt in dbt Cloud. You may only need to change the settings in your environment or job to call a more recent version of dbt - directions to do so can be found here.

How to run the latest version of dbt in Cloud

Changes between minor versions of dbt that will affect your project

Below we try to help you answer the question of whether a known breaking change between minor versions of dbt will affect your project. If you answer "yes" to any of the questions below, we recommend that you read the migration guides that we've put together for every dbt minor version release.

An Important Note on Packages

If you use any packages from dbt Hub, make sure you also upgrade to a version of the package that supports the dbt version you intend to upgrade to. You can see which dbt versions a package supports by checking on the require-dbt-version: in the package's dbt_project.yml file on GitHub.

As an example, dbt-utils version 0.7.1 supports dbt v0.20 and v0.21, as described in its dbt_project.yml.

After you've changed the package version in your packages.yml file, be sure to run dbt deps in the IDE to install the updated version.

Upgrading to v0.21.latest from v0.20
  • Do you select specific sources to check freshness (dbt snapshot-freshness --select <source_name>)?
  • Do you have custom scripts that parse dbt JSON artifacts?
  • (Snowflake only) Do you have custom macros or materializations that depend on using transactions, such as statement blocks with auto_begin=True?

If you believe your project might be affected, read more details in the migration guide here.

Upgrading to v0.20.latest from v0.19
  • Does your project define any custom schema tests?
  • Does your project use adapter.dispatch or the spark_utils package?
  • Do you have custom scripts that parse dbt JSON artifacts?

If you believe your project might be affected, read more details in the migration guide here.

Upgrading to v0.19.latest from v0.18
Important

If you have not already, you must add config-version: 2 to your dbt_project.yml file. See Upgrading to v0.17.latest from v0.16 below for more details.

  • Do you have custom scripts that parse dbt JSON artifacts?
  • Do you have any custom materializations?

If you believe your project might be affected, read more details in the migration guide here.

Upgrading to v0.18.latest from v0.17
  • Do you directly call adapter_macro?

If you believe your project might be affected, read more details in the migration guide here.

Upgrading to v0.17.latest from v0.16
Universal change

You must add config-version: 2 to your dbt_project.yml file.

dbt_project.yml
name: my_project
version: 1.0.0
config-version: 2
vars:
my_var: 1
another_var: true
models:
...
Universal change

vars: are now defined not in your models: but are a separate section in dbt_project.yml file.

dbt_project.yml
name: my_project
version: 1.0.0
config-version: 2
vars:
my_var: 1
another_var: true
models:
...
  • Do you have dictionary configs in your dbt_project.yml such as partition_by or persist_docs? If yes, you need to add a preceding +.
dbt_project.yml
models:
my_project:
reporting:
+partition_by:
field: date_day
data_type: timestamp

If you believe your project might be affected, read more details in the migration guide here.

Upgrading to v0.16.latest from v0.15
  • Do you use the custom generate_schema_name macro?
  • Do you use partition_by config for BigQuery models?

If you believe your project might be affected, read more details in the migration guide here.

Upgrading to v0.15.latest from v0.14
  • Do you have a custom materialization?
  • Do you have a macro that accesses Relations directly?

If you believe your project might be affected, read more details in the migration guide here.

Upgrading to v0.14.latest from v0.13
  • Do you still use Archives?
  • Do you use the custom generate_schema_name macro?
  • Do you use the —non-destructive flag?

If you believe your project might be affected, read more details in the migration guide here.

Testing your changes before upgrading

Once you have an idea about what code changes you'll need to make, you can start implementing them. We recommend that you create a separate dbt project, Upgrade Project, to test your changes before making them live in your main dbt project. In your Upgrade Project, connect to the same repository that you use for your main dbt project, but this time, set the development environment settings to run the latest version of dbt Core. Next check out a branch dbt-version-upgrade, make the appropriate updates to your project (if needed), and see if your dbt project compiles and runs with the new version of dbt in the IDE. If jumping directly to the latest version of dbt is too far of a leap for your project, try iteratively getting your project to work on each successive minor version. There are years of development and a handful of breaking changes between two distant versions of dbt (e.g. 0.14 --> 0.20). There are far fewer between two subsequent versions of dbt, which is why upgrading regularly is important.

Once you have your project compiling and running on the latest version of dbt in the development environment for your dbt-version-upgrade branch, try replicating one of your production jobs to run off your branch's code. You can do this by creating a new deployment environment for testing, setting the custom branch to 'ON' and referencing your dbt-version-upgrade branch. You'll also need to set the dbt version in this environment to the latest dbt Core version.

Setting your testing environment

Setting your testing environment

Then add a job to the new testing environment that replicates one of the production jobs your team relies on. If that job runs smoothly, you should be all set to merge your branch into main and change your development and deployment environments in your main dbt project to run off the newest version of dbt Core.