Packages
What is a package?
Software engineers frequently modularize code into libraries. These libraries help programmers operate with leverage: they can spend more time focusing on their unique business logic, and less time implementing code that someone else has already spent the time perfecting.
In dbt, libraries like these are called packages. dbt's packages are so powerful because so many of the analytic problems we encountered are shared across organizations, for example:
- transforming data from a consistently structured SaaS dataset, for example:
- turning Snowplow or Segment pageviews into sessions
- transforming AdWords or Facebook Ads spend data into a consistent format.
- writing dbt macros that perform similar functions, for example:
- generating SQL to union together two relations, pivot columns, or construct a surrogate key
- creating custom schema tests
- writing audit queries
- building models and macros for a particular tool used in your data stack, for example:
dbt packages are in fact standalone dbt projects, with models and macros that tackle a specific problem area. As a dbt user, by adding a package to your project, the package's models and macros will become part of your own project. This means:
- Models in the package will be materialized when you
dbt run
. - You can use
ref
in your own models to refer to models from the package. - You can use macros in the package in your own project.
Defining and installing dbt packages is different from defining and installing Python packages.
How do I add a package to my project?
- Add a
packages.yml
file to your dbt project. This should be at the same level as yourdbt_project.yml
file. - Specify the package(s) you wish to add using one of the supported syntaxes, for example:
packages:
- package: dbt-labs/snowplow
version: 0.7.0
- git: "https://github.com/dbt-labs/dbt-utils.git"
revision: 0.9.2
- local: /opt/dbt/redshift
- Run
dbt deps
to install the package(s). Packages get installed in thedbt_packages
directory – by default this directory is ignored by git, to avoid duplicating the source code for the package.
How do I specify a package?
You can specify a package using one of the following methods, depending on where your package is stored.
Hub packages (recommended)
dbt Hub is a registry for dbt packages. Packages that are listed on dbt Hub can be installed like so:
packages:
- package: dbt-labs/snowplow
version: 0.7.3 # version number
Hub packages require a version to be specified – you can find the latest release number on dbt Hub. Since Hub packages use semantic versioning, we recommend pinning your package to the latest patch version from a specific minor release, like so:
packages:
- package: dbt-labs/snowplow
version: [">=0.7.0", "<0.8.0"]
Where possible, we recommend installing packages via dbt Hub, since this allows dbt to handle duplicate dependencies. This is helpful in situations such as:
- Your project uses both the dbt-utils and Snowplow packages; and the Snowplow package also uses the dbt-utils package.
- Your project uses both the Snowplow and Stripe packages, both of which use the dbt-utils package.
In comparison, other package installation methods are unable to handle the duplicate dbt-utils package.
Prerelease versions
Some package maintainers may wish to push prerelease versions of packages to the dbt Hub, in order to test out new functionality or compatibility with a new version of dbt. A prerelease version is demarcated by a suffix, such as a1
(first alpha), b2
(second beta), or rc3
(third release candidate).
By default, dbt deps
will not include prerelease versions when resolving package dependencies. You can enable the installation of prereleases in one of two ways:
- Explicitly specifying a prerelease version in your
version
criteria - Setting
install-prerelease
totrue
, and providing a compatible version range
Both of the following configurations would successfully install 0.4.5a2
of dbt_artifacts
:
packages:
- package: brooklyn-data/dbt_artifacts
version: 0.4.5a2
packages:
- package: brooklyn-data/dbt_artifacts
version: [">=0.4.4", "<0.4.6"]
install-prerelease: true
Git packages
Packages stored on a Git server can be installed using the git
syntax, like so:
packages:
- git: "https://github.com/dbt-labs/dbt-utils.git" # git URL
revision: 0.9.2 # tag or branch name
Add the Git URL for the package, and optionally specify a revision. The revision can be:
- a branch name
- a tagged release
- a specific commit (full 40-character hash)
Example of a revision specifying a 40-character hash:
packages:
- git: "https://github.com/dbt-labs/dbt-utils.git"
revision: 4e28d6da126e2940d17f697de783a717f2503188
We strongly recommend "pinning" your package to a specific release by specifying a release name.
If you do not provide a revision, or if you use master
, then any updates to the package will be incorporated into your project the next time you run dbt deps
. While we generally try to avoid making breaking changes to these packages, they are sometimes unavoidable. Pinning a package revision helps prevent your code from changing without your explicit approval.
To find the latest release for a package, navigate to the Releases
tab in the relevant GitHub repository. For example, you can find all of the releases for the dbt-utils package here.
As of v0.14.0, dbt will warn you if you install a package using the git
syntax without specifying a version (see below).
Private packages
SSH Key Method (Command Line only)
If you're using the Command Line, private packages can be cloned via SSH and an SSH key.
When you use SSH keys to authenticate to your git remote server, you don’t need to supply your username and password each time. Read more about SSH keys, how to generate them, and how to add them to your git provider here: Github and GitLab.
packages:
- git: "git@github.com:dbt-labs/dbt-utils.git" # git SSH URL
If you're using dbt Cloud, the SSH key method will not work, but you can use the HTTPS Git Token Method.
Git Token Method
This method allows the user to clone via HTTPS by passing in a git token via an environment variable. Be careful of the expiration date of any token you use, as an expired token could cause a scheduled run to fail. Additionally, user tokens can create a challenge if the user ever loses access to a specific repo.
If you are using dbt Cloud, you must adhere to the naming conventions for environment variables. Environment variables in dbt Cloud must be prefixed with either DBT_
or DBT_ENV_SECRET_
. Environment variables keys are uppercased and case sensitive. When referencing {{env_var('DBT_KEY')}}
in your project's code, the key must match exactly the variable defined in dbt Cloud's UI.
In GitHub:
packages:
# use this format when accessing your repository via a github application token
- git: "https://{{env_var('DBT_ENV_SECRET_GIT_CREDENTIAL')}}@github.com/dbt-labs/awesome_repo.git" # git HTTPS URL
# use this format when accessing your repository via a classical personal access token
- git: "https://{{env_var('DBT_ENV_SECRET_GIT_CREDENTIAL')}}@github.com/dbt-labs/awesome_repo.git" # git HTTPS URL
# use this format when accessing your repository via a fine-grained personal access token (username sometimes required)
- git: "https://GITHUB_USERNAME:{{env_var('DBT_ENV_SECRET_GIT_CREDENTIAL')}}@github.com/dbt-labs/awesome_repo.git" # git HTTPS URL
Read more about creating a GitHub Personal Access token here. You can also use a GitHub App installation token.
In GitLab:
packages:
- git: "https://{{env_var('DBT_USER_NAME')}}:{{env_var('DBT_ENV_SECRET_DEPLOY_TOKEN')}}@gitlab.example.com/dbt-labs/awesome_project.git" # git HTTPS URL
Read more about creating a GitLab Deploy Token here and how to properly construct your HTTPS URL here. Deploy tokens can be managed by Maintainers only.
In Azure DevOps:
packages:
- git: "https://{{env_var('DBT_ENV_SECRET_PERSONAL_ACCESS_TOKEN')}}@dev.azure.com/dbt-labs/awesome_project/_git/awesome_repo" # git HTTPS URL
Read more about creating a Personal Access Token here.
In Bitbucket:
packages:
- git: "https://{{env_var('DBT_USER_NAME')}}:{{env_var('DBT_ENV_SECRET_PERSONAL_ACCESS_TOKEN')}}@bitbucketserver.com/scm/awesome_project/awesome_repo.git" # for Bitbucket Server
Read more about creating a Personal Access Token here.
Configure subdirectory for packaged projects
In general, dbt expects dbt_project.yml
to be located as a top-level file in a package. If the packaged project is instead nested in a subdirectory—perhaps within a much larger mono repo—you can optionally specify the folder path as subdirectory
. dbt will attempt a sparse checkout of just the files located within that subdirectory. Note that you must be using a recent version of git
(>=2.26.0
).
packages:
- git: "https://github.com/dbt-labs/dbt-labs-experimental-features" # git URL
subdirectory: "materialized-views" # name of subdirectory containing `dbt_project.yml`
Local packages
Packages that you have stored locally can be installed by specifying the path to the project, like so:
packages:
- local: /opt/dbt/redshift # use a local path
Local packages should only be used for specific situations, for example, when testing local changes to a package.
What packages are available?
Check out dbt Hub to see the library of published dbt packages!
Advanced package configuration
Updating a package
When you update a version or revision in your packages.yml
file, it isn't automatically updated in your dbt project. You should run dbt deps
to update the package. You may also need to run a full refresh of the models in this package.
Uninstalling a package
When you remove a package from your packages.yml
file, it isn't automatically deleted from your dbt project, as it still exists in your dbt_packages/
directory. If you want to completely uninstall a package, you should either:
- delete the package directory in
dbt_packages/
; or - run
dbt clean
to delete all packages (and any compiled models), followed bydbt deps
.
Configuring packages
You can configure the models and seeds in a package from the dbt_project.yml
file, like so:
vars:
snowplow:
'snowplow:timezone': 'America/New_York'
'snowplow:page_ping_frequency': 10
'snowplow:events': "{{ ref('sp_base_events') }}"
'snowplow:context:web_page': "{{ ref('sp_base_web_page_context') }}"
'snowplow:context:performance_timing': false
'snowplow:context:useragent': false
'snowplow:pass_through_columns': []
models:
snowplow:
+schema: snowplow
seeds:
snowplow:
+schema: snowplow_seeds
For example, when using a dataset specific package, you may need to configure variables for the names of the tables that contain your raw data.
Configurations made in your dbt_project.yml
file will override any configurations in a package (either in the dbt_project.yml
file of the package, or in config blocks).
Specifying unpinned Git packages
If your project specifies an "unpinned" Git package, you may see a warning like:
The git package "https://github.com/dbt-labs/dbt-utils.git" is not pinned.
This can introduce breaking changes into your project without warning!
This warning can be silenced by setting warn-unpinned: false
in the package specification. Note: This is not recommended.
packages:
- git: https://github.com/dbt-labs/dbt-utils.git
warn-unpinned: false
Setting two-part versions
In dbt v0.17.0 only, if the package version you want is only specified as major
.minor
, as opposed to major.minor.patch
, you may get an error that 1.0 is not of type 'string'
. In that case you will have to tell dbt that your version number is a string. This issue was resolved in v0.17.1 and all subsequent versions.
packages:
- git: https://github.com/dbt-labs/dbt-codegen.git
version: "{{ 1.0 | as_text }}"