Flow management control

Workflow management plugin

Use VaultSpeed’s flow management control (FMC) add-on module to generate workflow schedules.

FMC helps you ensure that all data pipelines are executed at the right time, in the right order. Whether you’re running your data warehouse in batch mode or using micro-batches and CDC tools.

Best-of-breed schedulers

For our workflow solution we stay true to our strategy and core offer: automation. So instead of building our own workflow scheduling tool, we ensured that the VaultSpeed FMC module works with the best workflow schedulers available:

Apache Airflow
— An open-source platform built to programmatically author, schedule, and monitor workflows

AzureData Factory
— The go-to platform to create, schedule and manage data pipelines in the Azure cloud

Generic FMC — Generate platform agnostic workflows that fit in any scheduler like the ones in dbt Cloud, Matillion, Snowflake, Databricks and so many others.

FMC Airflow

FMC deployed in Apache Airflow

Easy to configure

VaultSpeed’s FMC plugin generates a well-defined execution order for data pipeline orchestration. It’s easy to configure to execute extraction and loading tasks more efficiently.

Features

  • Incremental and initial loads
  • Parallel loading
  • Task grouping
  • Custom loading intervals
FMC Azure Data Factory

FMC deployed in Azure Data Factory

Platform agnostic workflows

What if you’re using a different scheduler, or you want to use your target platform’s native capabilities to orchestrate your data loads?

Generic FMC will allow you to generate generic workflows that can be implemented into any scheduler or workflow management solution.

FMC will generate JSON files containing all the data needed to build your own FMC implementation. All the logic to calculate and keep track of the loading windows is handled by procedures generated by VaultSpeed. The only thing you need to set up is a process that executes these procedures in the right order. You‘ll also need to handle metadata such as the load date and the load cycle id.

Find more details in the Docs portal. And get inspired by this community blog post on how to build a workflow solution natively on Snowflake using our generic FMC module. Check this GitHub repository for scheduler integrations with dbt cloud, Snowflake and others.

More add-on modules

Template Studio

Build your own custom, company-specific templates

Learn more

Spark Structured Streaming

Stream data into your Data Vault

Learn more

Agent Extensions

Read metadata directly from any source that supports JDBC or Kafka.

Learn more

Automate repetitive data warehousing tasks