dbt Cloud makes it easy to test every single change you make prior to deploying that code into production. Once you've connected your GitHub account or your GitLab account, you can configure jobs to run when new Pull Requests (referred to as Merge Requests in GitLab) are opened against your dbt repo. When these jobs complete, their statuses will be shown directly in the Pull Request. This makes it possible to deploy new code to production with confidence.
GitLab Webhooks Compatability and Availability
If you previously configured your dbt project by providing a GitLab git URL, you need to reconfigure the repository to connect through native GitLab authentication, as we cannot enable webhooks for your project through SSH.
GitLab Webhooks are available to only GitLab users who have a paid or self-hosted GitLab account.
The green checkmark means this PR is ready to be merged. The 'details' link shown here will navigate you to the relevant run in dbt Cloud.
The green checkmark means this MR is ready to be merged. The 'dbt Cloud' pop up shown here will navigate you to the relevant run in dbt Cloud.
Enabling Continuous Integration (CI)
To enable runs on Pull Requests, navigate to the Job Settings page for the relevant job. In the "Triggers" section of the page, switch to the "Webhooks" tab, and then click the checkbox next to
RUN ON PULL REQUESTS? as shown below.
Understanding CI in dbt Cloud
When Pull Request builds are enabled, dbt Cloud will listen for webhooks from GitHub or GitLab indicating that a new PR has been opened or updated with new commits. When one of these webhooks is received, dbt Cloud will enqueue a new run of the specified job. Crucially, this run will be configured to build into a special, temporary schema using the prefix
dbt_cloud. The name of these temporary schemas will be unique for each PR, and is shown in the Run Details page for the given run. This temporary schema will allow you to inspect the relations built by dbt Cloud, directly in your data warehouse.
When the run is complete, dbt Cloud will update the PR in GitHub or MR in GitLab with a status message indicating the results of the run. Once the PR is closed or merged, dbt Cloud will delete the temporary schema from your data warehouse.
With Slim CI, you don't have to rebuild and test all your models. You can instruct dbt Cloud to run jobs on only modified or new resources.
When creating or editing a job in dbt Cloud, you can set your execution settings to defer to a previous run state. Use the drop drop menu to select which production job you want to defer to.
Jobs that run on pull requests may select "self" or another job from the same project for deferral and comparison
When a job is selected, dbt Cloud will surface the artifacts from that job's most recent successful run. dbt will then use those artifacts to determine the set of new and modified resources. In your job commands, you can signal to dbt to run only on these modified resources and their children by including the
dbt seed --select state:modified+dbt run --models state:modified+dbt test --models state:modified+
Because dbt Cloud manages deferral and state environment variables, there is no need to specify
--state flags. Note: Both jobs need to be running dbt v0.18.0 or newer.
To learn more about state comparison and deferral in dbt, read the docs on state.
Reconnecting your dbt project to use a native GitHub or GitLab repo
If your dbt project relies on a Git URL and SSH to connect to your GitHub or GitLab repo, you need to disconnect your repo and reconnect it with native Github or GitLab auth in order to enable webhooks.
First, make sure you have native GitHub authentication or native GitLab authentication set up. After you have gone through those steps, head to
Account Settings ->
Projects and click on the project you'd like to reconnect through native GitHub or GitLab auth. Then click on the repository link.
Once you're in the repository page, you can click the
Edit button and then the
Disconnect Repository button at the bottom.
Confirm that you'd like to disconnect your repository. You should then see a new
Configure a repository link in your old repository's place. Click through to the configuration page, which will look like this:
GitLab tab and reselect your repository. That should complete the setup and enable you to use webhooks in your jobs configuration.