More add-on modules
Spark Structured Streaming
Stream data into your Data Vault
Flow management control
Workflow management plugin
Use VaultSpeed’s flow management control (FMC) add-on module to generate workflow schedules.
FMC helps you ensure that all data pipelines are executed at the right time, in the right order. Whether you’re running your data warehouse in batch mode or using micro-batches and CDC tools.
Best-of-breed schedulers
For our workflow solution we stay true to our strategy and core offer: automation. So instead of building our own workflow scheduling tool, we ensured that the VaultSpeed FMC module works with the best workflow schedulers available:
Apache Airflow — An open-source platform built to programmatically author, schedule, and monitor workflows
AzureData Factory — The go-to platform to create, schedule and manage data pipelines in the Azure cloud
Generic FMC — Generate platform agnostic workflows that fit in any scheduler like the ones in dbt Cloud, Matillion, Snowflake, Databricks and so many others.
FMC deployed in Apache Airflow
Easy to configure
VaultSpeed’s FMC plugin generates a well-defined execution order for data pipeline orchestration. It’s easy to configure to execute extraction and loading tasks more efficiently.
Features
- Incremental and initial loads
- Parallel loading
- Task grouping
- Custom loading intervals
FMC deployed in Azure Data Factory
Platform agnostic workflows
What if you’re using a different scheduler, or you want to use your target platform’s native capabilities to orchestrate your data loads?
Generic FMC will allow you to generate generic workflows that can be implemented into any scheduler or workflow management solution.
FMC will generate JSON files containing all the data needed to build your own FMC implementation. All the logic to calculate and keep track of the loading windows is handled by procedures generated by VaultSpeed. The only thing you need to set up is a process that executes these procedures in the right order. You‘ll also need to handle metadata such as the load date and the load cycle id.
Find more details in the Docs portal. And get inspired by this community blog post on how to build a workflow solution natively on Snowflake using our generic FMC module. Check this GitHub repository for scheduler integrations with dbt cloud, Snowflake and others.