Back to supported technologies

dbt

dbt is a data transformation tool that enables data engineers to transform, test and document data in the cloud data warehouse.

What is dbt?

dbt™ is a SQL-first transformation workflow that lets teams quickly and collaboratively deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation. Now anyone on the data team can safely contribute to production-grade data pipelines.

dbt is popular amongst data engineers, thanks to its modular SQL capabilities and open-source community. The skills required are SQL, version control, and data modeling.

A modeling interface on top of dbt

Facing enterprise-level data integration challenges often involves joining numerous data models, including source data models and conceptual business-driven data models. The scale of this effort necessitates automation. This is where Data Vault excels as it provides both integration and automation capabilities (read why).

In a Data Vault context, it's crucial to start with a business perspective and incorporate the realities of various data sources (read how). The only way to handle the large volume and complexity of these models is to break down the integration workload into manageable parts and allow data modelers and engineers to work together effectively. This is why it's essential to have a visible data model, rather than having it hidden in code. A GUI that enables users to quickly understand the entities and their relationships is also crucial.

This is why we support dbt as a target for deploying Data Vault transformation code:

  • VaultSpeed encourages users to build their Data Vault model correctly and provides a comprehensible GUI for viewing and enriching source and target model metadata. It also automatically generates all necessary code based on the metadata, eliminating the need for manual coding.
  • dbt serves as a platform for storing, versioning, and running these generated modular SQL scripts. With features such as key definitions, data lineage, and dependency alerts, dbt is an ideal solution in a Data Vault context.

model your Data Vault in VaultSpeed

deploy Data Vault code into dbt models

How it works

VaultSpeed offers a modeling interface for dbt users to efficiently gather and enhance massive amounts of source metadata, transforming it into the desired integration model. The platform generates the necessary integration code in the form of dbt models and macros, and seamlessly integrates with Git to allow you to push the code to your desired development branch and retrieve it in your dbt environment. The Data Vault model is then delivered to the CDP as DDL statements.

To transfer data to Snowflake, VaultSpeed integrates with a majority of CDC tools. For file structures or streaming topics, Snowflake's Snowpipes can also be utilized.

Create a workflow schedule

Utilize VaultSpeed's Flow Management Control (FMC) add-on module to guarantee efficient execution of all data pipelines. Schedule your workflows through top-notch schedulers such as Apache Airflow or leverage the native scheduling capabilities offered by FMC and dbt.

This will ensure that your workflows are executed at the appropriate time and in the correct sequence.

VaultSpeed's FMC module generates a dbt aware plugin for Apache Airflow. It will offer two options to run your dbt models:

  1. dbt Cloud: Our plugin for Airflow seamlessly integrates with your dbt Cloud environment to run all necessary code trough API calls.
  2. dbt CLI: This allows you to run the dbt models locally with the dbt CLI run command.
Learn more

running dbt using airflow


Visit dbt.com

Watch the demo

"In Data Vault context, we have to deal with huge data models, and it’s challenging to spread all the metadata into YAML files. Putting VaultSpeed’s modeling GUI on top of dbt solves that problem for me. I can see the data models I’m working on in a central and managed view, and create my dbt models from there."
Michael Olschimke Data Vault 2.0 Author

Ready to get started?

Talk to our technical sales team to answer your questions.

Contact sales