dbt™

A modeling interface on top of dbt™

Facing enterprise-level data integration challenges often involves joining numerous data models, including source data models and conceptual business-driven data models. The scale of this effort necessitates automation. This is where Data Vault excels as it provides both integration and automation capabilities (read why).

In a Data Vault context, it's crucial to start with a business perspective and incorporate the realities of various data sources (read how). The only way to handle the large volume and complexity of these models is to break down the integration workload into manageable parts and allow data modelers and engineers to work together effectively. This is why it's essential to have a visible data model, rather than having it hidden in code. A GUI that enables users to quickly understand the entities and their relationships is also crucial.

This is why we support dbt™ as a target for deploying Data Vault transformation code:

• VaultSpeed encourages users to build their Data Vault model correctly and provides a comprehensible GUI for viewing and enriching source and target model metadata. It also automatically generates all necessary code based on the metadata, eliminating the need for manual coding.
• dbt™ serves as a platform for storing, versioning, and running these generated modular SQL scripts. With features such as key definitions, data lineage, and dependency alerts, dbt™ is an ideal solution in a Data Vault context.

How it works

VaultSpeed offers a modeling interface for dbt™ users to efficiently gather and enhance massive amounts of source metadata, transforming it into the desired integration model. The platform generates the necessary integration code in the form of dbt™ models and macros, and seamlessly integrates with Git to allow you to push the code to your desired development branch and retrieve it in your dbt™ environment. The Data Vault model is then delivered to the CDP as DDL statements.

To transfer data to Snowflake, VaultSpeed integrates with a majority of CDC tools. For file structures or streaming topics, Snowflake's Snowpipes can also be utilized.

Create a workflow schedule

Utilize VaultSpeed's Flow Management Control (FMC) add-on module to guarantee efficient execution of all data pipelines. Schedule your workflows through top-notch schedulers such as Apache Airflow or leverage the native scheduling capabilities offered by FMC and dbt™.

This will ensure that your workflows are executed at the appropriate time and in the correct sequence.

VaultSpeed's FMC module generates a dbt™ aware plugin for Apache Airflow. It will offer two options to run your dbt™ models:

1. dbt™ Cloud: Our plugin for Airflow seamlessly integrates with your dbt™ Cloud environment to run all necessary code trough API calls.
2. dbt™ Core: This allows you to run the dbt models locally with the dbt™ CLI run command.