dbt™

Data Vault automation for dbt Cloud

Facing enterprise-level data integration challenges often involves joining numerous data models, including source data models and conceptual business-driven data models. The scale of this effort necessitates automation. This is where Data Vault excels as it provides both integration and automation capabilities (read why).

In a Data Vault context, it's crucial to start with a business perspective and incorporate the realities of various data sources (read how). The only way to handle the large volume and complexity of these models is to break down the integration workload into manageable parts and allow business data modelers, data vault engineers, analytics engineers and devops engineers to work together effectively. This is why it's essential to have a visible data model, rather than having it hidden in code. A GUI that enables users to quickly understand the entities and their relationships is also crucial.

This is why we support dbt Cloud™ as a target for deploying Data Vault transformation code:

• VaultSpeed is a tool that helps users in creating and managing their Data Vault models. It provides a user-friendly graphical interface for users to view and edit the metadata of their source and target models. Moreover, it automatically generates the required code based on the metadata, which saves users from the tedious task of manual coding.
• VaultSpeed's Data Vault layer can be considered as the foundational dbt Cloud project, which other domain-specific projects can depend on. The Data Vault project ensures that changes in source systems do not violate any data contracts towards other projects. This setup enables easy collaboration between Data Vault and Analytics engineers and aligns with the dbt Mesh philosophy.
• dbt Cloud serves as a platform for storing, versioning, and running these generated modular SQL scripts. It's the perfect platform to deploy your Data Vault. With features such as key definitions, data lineage, and dependency alerts, dbt™ is an ideal solution in a Data Vault context.

How it works

VaultSpeed offers a modeling interface for Data Vault engineers to efficiently gather and enhance massive amounts of source metadata, transforming it into the desired integration model. The platform generates the necessary integration code in the form of dbt™ models and macros, and seamlessly integrates with Git to allow you to push the code to your desired development branch and retrieve it in your dbt Cloud environment. The Data Vault model is then delivered to the CDP as DDL statements.

To transfer data to Snowflake or Databricks, VaultSpeed integrates with a majority of CDC tools. For file structures or streaming topics, Snowflake's Snowpipes or Databricks' Autoloader can also be used.

Create a workflow schedule

Use VaultSpeed's Flow Management Control (FMC) to guarantee efficient execution. Schedule your workflows through native scheduling capabilities offered by FMC and dbt Cloud.

This will ensure that your workflows are executed at the appropriate time and in the correct sequence.

VaultSpeed's has built a dbt™ scheduler plugin to run your dbt™ models efficiently.