dbt™ is a data transformation tool that enables data engineers to transform, test and document data in the cloud data warehouse.

What is dbt™?

dbt™ is a SQL-first transformation workflow that lets teams quickly and collaboratively deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation. Now anyone on the data team can safely contribute to production-grade data pipelines.

dbt™ is popular amongst data engineers, thanks to its modular SQL capabilities and open-source community. The skills required are SQL, version control, and data modeling.

Data Vault automation for dbt Cloud

Facing enterprise-level data integration challenges often involves joining numerous data models, including source data models and conceptual business-driven data models. The scale of this effort necessitates automation. This is where Data Vault excels as it provides both integration and automation capabilities (read why).

In a Data Vault context, it's crucial to start with a business perspective and incorporate the realities of various data sources (read how). The only way to handle the large volume and complexity of these models is to break down the integration workload into manageable parts and allow business data modelers, data vault engineers, analytics engineers and devops engineers to work together effectively. This is why it's essential to have a visible data model, rather than having it hidden in code. A GUI that enables users to quickly understand the entities and their relationships is also crucial.

This is why we support dbt Cloud™ as a target for deploying Data Vault transformation code:

  • VaultSpeed is a tool that helps users in creating and managing their Data Vault models. It provides a user-friendly graphical interface for users to view and edit the metadata of their source and target models. Moreover, it automatically generates the required code based on the metadata, which saves users from the tedious task of manual coding.
  • VaultSpeed's Data Vault layer can be considered as the foundational dbt Cloud project, which other domain-specific projects can depend on. The Data Vault project ensures that changes in source systems do not violate any data contracts towards other projects. This setup enables easy collaboration between Data Vault and Analytics engineers and aligns with the dbt Mesh philosophy.
  • dbt Cloud serves as a platform for storing, versioning, and running these generated modular SQL scripts. It's the perfect platform to deploy your Data Vault. With features such as key definitions, data lineage, and dependency alerts, dbt™ is an ideal solution in a Data Vault context.
Source data model in Vault Speed

model your Data Vault in VaultSpeed

Generated dbt model for a hub and corresponding lineage

deploy Data Vault code into dbt models

How it works

VaultSpeed offers a modeling interface for Data Vault engineers to efficiently gather and enhance massive amounts of source metadata, transforming it into the desired integration model. The platform generates the necessary integration code in the form of dbt™ models and macros, and seamlessly integrates with Git to allow you to push the code to your desired development branch and retrieve it in your dbt Cloud environment. The Data Vault model is then delivered to the CDP as DDL statements.

To transfer data to Snowflake or Databricks, VaultSpeed integrates with a majority of CDC tools. For file structures or streaming topics, Snowflake's Snowpipes or Databricks' Autoloader can also be used.

Dbt vaultspeed

Create a workflow schedule

Use VaultSpeed's Flow Management Control (FMC) to guarantee efficient execution. Schedule your workflows through native scheduling capabilities offered by FMC and dbt Cloud.

This will ensure that your workflows are executed at the appropriate time and in the correct sequence.

VaultSpeed's has built a dbt™ scheduler plugin to run your dbt™ models efficiently.

Learn more
Screenshot 2023 12 13 at 21 00 34

running dbt models

"In Data Vault context, we have to deal with huge data models, and it’s challenging to spread all the metadata into YAML files. Putting VaultSpeed’s modeling GUI on top of dbt solves that problem for me. I can see the data models I’m working on in a central and managed view, and create my dbt models from there."
Michael olschimke 1024x1024
Michael Olschimke Data Vault 2.0 Author

Ready to get started?

Talk to our technical sales team to answer your questions.