Workflow management plugin

Best-of-breed schedulers

For our workflow solution we stay true to our strategy and core offer: automation. So instead of building our own workflow scheduling tool, we ensured that the VaultSpeed FMC module works with the best workflow schedulers available:

Apache Airflow — An open-source platform built to programmatically author, schedule, and monitor workflows

AzureData Factory — The go-to platform to create, schedule and manage data pipelines in the Azure cloud

Generic FMC — Generate platform agnostic workflows that fit in any scheduler like the ones in dbt Cloud, Matillion, Snowflake, Databricks and so many others.

Easy to configure

VaultSpeed’s FMC plugin generates a well-defined execution order for data pipeline orchestration. It’s easy to configure to execute extraction and loading tasks more efficiently.

Features

• Incremental and initial loads
• Parallel loading
• Task grouping
• Custom loading intervals

Platform agnostic workflows

What if you’re using a different scheduler, or you want to use your target platform’s native capabilities to orchestrate your data loads?

Generic FMC will allow you to generate generic workflows that can be implemented into any scheduler or workflow management solution.

FMC will generate JSON files containing all the data needed to build your own FMC implementation. All the logic to calculate and keep track of the loading windows is handled by procedures generated by VaultSpeed. The only thing you need to set up is a process that executes these procedures in the right order. You‘ll also need to handle metadata such as the load date and the load cycle id.

Find more details in the Docs portal. And get inspired by this community blog post on how to build a workflow solution natively on Snowflake using our generic FMC module. Check this GitHub repository for scheduler integrations with dbt cloud, Snowflake and others.