Snowflake Procedures (Release 4.1.17)

Snowflake Procedures (Release 4.1.17)

In the latest VaultSpeed release we included important improvements for Snowflake customers ❄️.
We also used valuable customer feedback to build some improvements for VaultSpeed Studio. Finally, we partly redesigned ELT generation for Talend to increase generation speed and robustness.


Snowflake is one of the most popular data platforms around these days. The success of their recent IPO emphasizes that. We support Snowflake as a target and starting now, the integration just got better.
Following up on exciting developments in Snowflake, VaultSpeed now generates ELT code for Snowflake that is wrapped in Javascript procedures. From now on you can actually deploy and store your procedures inside Snowflake. One of the main advantages of this is that our workflow solution in Airflow can call these procedures instead of executing saved SQL files.
We also enabled auto deploy for Snowflake: you can now deploy DDL and ETL to the target using VaultSpeed’s Agent. Both changes together make it possible to start loading your data warehouse without any manual interaction by your developers.


Snowflake stored procedures

VaultSpeed Studio

In a previous release we announced VaultSpeed studio in closed alpha version. In the past few weeks we went to work with initial customer feedback. First thing we improved is the integrated template code editor. It includes smoother navigation options, changing and saving a template is more solid and you now have the option to compare your changes to the pre-edit version.

Template code editor

We added a view all option to the target definition where you can see all attributes of the template target. Previously, both existing and newly created target attributes were shown in separate windows.

In a previous post we explained the need for signature fields when doing automation. We did a complete redesign of the signature definition screen. You can create and select signature attributes on top of the screen and assign them to attributes in a list below. This list can be filtered before assigning a certain signature to a filtered set of fields.

Signature attributes

VaultSpeed studio will move to open alpha in one of the next releases. From then on, all clients can start a one month free trial period with VS Studio.


Talend is one of the first ELT tools for which we supported automation. This is done by generating jobscript that can be deployed to generate ELT mappings inside Talend Studio’s repository.

Due to improvements and changes in their product, the need for a major update of our template compiler for Talend became apparent. The result is that the generation of Talend jobscript is much more robust. We also took a giant leap in terms of speed: jobscript generation is up to 3 times faster compared to the previous release.

Quality of life changes

Some smaller changes will certainly improve user experience:

  • We added a download button to the automatic deployment screen. This allows users to download generated code through the browser instead of having to obtain it from the agent.



  • We improved the description of the attribute based SAT split.
  • You can directly upload CSV files to update short and abbreviated names in the source trough your browser.
  • We moved the automatic deployment menu out of the generation settings since it didn't really belong there.
  • Users will experience improved loading performance of parameter screens.
  • We added an extra tab to the DDL settings page where you can get an overview of all applied settings per layer.
  • Business views are checked for naming conflicts with other objects. We also provided users with a button to disable/enable generation for all business views (to use in combination with filters).
  • Releases in the delta generation tab are now sorted such that the latest one is always shown first.
  • We added subscription updates as data points to the generations graph on the dashboard. It will also show the next subscription reset date.
    Based on the number of jobs granted in your subscription, we set a warning when the mapping counter is above 80% or at 100% of the subscription limit.
  • And a lot more that did not make this post... 🤷‍♂️


Our next release is coming up quite fast. Would you like to stay tuned on future VaultSpeed releases? Fill in this form 👇


VaultSpeed Studio (Release 4.1.16)

VaultSpeed Studio (Release 4.1.16)

VaultSpeed STUDIO

Yesterday, VaultSpeed launched release 4.1.16. We are proud to announce a brand new module to our automation tool: VaultSpeed Studio.


The introduction of VaultSpeed Studio is an important milestone for VaultSpeed and our Customers. It opens up the powerful VaultSpeed template language and allows you to build, test and put to work your own custom templates to generate any logic required.

VaultSpeed Studio offers a solution for those clients looking to put automation to work in other parts of their data warehouse. The raw Data vault consists of highly repeatable patterns. On the contrary, pre-staging area's or presentation area's require more specific logic. Studio enables customers to automate these patterns as well.

This first version is released in closed alpha (i.e. only accessible on request). It allows Customers and Partners to build templates for the creation and loading of objects in the Business Vault Area. The following types of objects can, for example, be created: calculated satellites, effectivity satellites, end date satellites (for insert only), custom PITs, custom Bridges and much more.

Repeatability versus customization

VaultSpeed covers almost any object in the Raw Data Vault. We also cover standard PITs and Bridges tables in the Business Vault area. These objects are highly repeatable as they look the same in every organisation.

Pre-staging area's, presentation area's and access layers and certain business vault objects require a different approach. While still being a good target for automation, they require more customization and creativity. They do not necessarily look the same across organisations.

We situate VaultSpeed Studio in the area where objects become a less repeatable and require more customization.

The Interface

VaultSpeed Studio comes as a separate module. In this release we added the studio module for the Business Vault. Once you select a Business Vault release it will appear in the top menu.

VS Studio template overview

After opening Studio, a main window will appear where you can see a list of saved templates. When starting a new template or when changing an existing one, you can use populair code editors like Visual Studio Code (for which we deliver a code plugin) or you can view, edit and test the code in the editor screen.

VaultSpeed Studio template example

Template Language

Our template language was built to distinguish logic from implementation details. By result the language is quite easy to learn.

Signature attributes

When building templates based on metadata it is necessary to make a categorization in fields. In VaultSpeed, we do this by defining signature attributes. A signature attribute from a source model can be a Business Key, Relation Key, or other. Next to the standard signature objects you can now define new signature attrubutes on which generic logic can be applied (repeatable pattern).

The more generic a signature attribute, the more repeatable the patterns that you can create with them. VaultSpeed's standard templates use these highly generic signature attribute for the generation of Raw Data Vault objects and Business Vault objects like PITs and Bridges.

However, when you are building custom templates, you must be able to create custom signature attribute. So when you would need to perform a certain operation on a set of custom fields, you will be able to define your own signature attributes in VaultSpeed Studio.

Apply a custom signature to a certain set of fields

Template language

The VSS template language has a hierarchical structure. Each level in the hierarchy starts with a certain keyword and continues until the next one or the end of the file. Each of these levels can be conditioned based on certain parameters and variables, this allows you to combine the logic for different types of objects into 1 template.


VS Studio will be released in open alpha in one of the following releases. By then it will include a limited 30 day trial period of VSS for everyone, full usage of VSS will require an additional subscription (more details about pricing coming soon).

While VSS is probably the most important change in VS 4.1.16, there is some other new stuff included:

Foreign keys to REF tables

We added the ability to define foreign keys to reference tables. You can create relations between an object and a reference table and this will result in an additional ref_fk attribute in the satellite, these foreign key attributes will be coalesced with null values. This, combined with the addition of ghost records to the REF objects allows for inner joins between Satellites and Reference tables.

Multi Active Satellites

You can now create multi active satellites with a sub-sequence attribute in the source. An attribute can be set to be the sub-sequence attribute in the source graphical editor, this attribute in combination with the business key should be unique.

setting a multi-active SAT with a subsequence

Other features

We added a status indication for new and deleted objects in the grouping and DV overview screens. This change also introduces some performance improvements for these screens.

Selecting a reset option (i.e. not “keep everything”) when creating a new source release will reset the release to the latest one linked to a DV production release. If no production date is set yet, then it will reset fully like before.

There is an additional attribute type available in the source graphical editor: modification date. When the cdc type is modification date then setting a column to this type will indicate that it is the cdc-timestamp metadata attribute for a certain object.

It is now possible to delete entire projects: this can be useful for deleting a sandbox project that had been created for testing purposes. In this way you can keep your account nice and clean from clutter.

We added the ability to define unique keys in the source graphical editor, they can be defined by selecting attributes while holding ctrl/command and then right clicking and selecting unique key (and optionally a name for the key). You can also set an attribute to UK from the attribute sidebar.


VaultSpeed invests heavily in making the complex process of enterprise wide data integration as easy as possible. With this in mind, we will keep improving towards intuitive graphical interfaces that should assist the user in the decision-making process by implementing smart suggestions and eliminating as much of the manual work as possible.

Not all layers of a data warehouse can match the repeatability of the raw data vault layer. Pre-staging layers, Business data vault objects or data marts demand a higher level of customization. Vaultspeed will extend its capabilities to add custom logic so that our customers can work towards tailor-made solutions for their data integration challenges.

“VaultSpeed aims to deliver economies of scale in all layers of the modern data warehouse. We realize that a trade-off exists between customization and repeatability. We also realize that the economies of scale decrease with increasing levels of customization.

By designing your data pipeline using repeatable patterns you can reduce the customizations and deliver high quality and high value through automation.

With VS Studio, we took another step towards these goals. Eventually, Studio will help customers to exploit the benefits of automation all the way from source to presentation layer.

Additionally we will work together with our partners to create industry specific repeatable patterns based data pipelines that will create value for our customers.”

Dirk Vermeiren - CTO VaultSpeed

So stay tuned... more features are coming soon!

Do you want to stay informed about updates? Become part of our mailing group:

Referential Integrity Error Handling (Release 4.1.15)

Referential Integrity Error Handling (Release 4.1.15)

VaultSpeed release 4.1.15 just went live! Here’s what’s new...

Referential Integrity

When integrating multiple source systems into one data vault, you might encounter some issues with referential integrity. This can have numerous reasons like issues with source batch processing, late data arrival, source mismatches or the possibility of early arriving data.

In any case it will cause you some headaches to fix these kind of problems. With VaultSpeed, you now have an extra set of tools to tackle these problems.

In the latest release we added referential integrity error handling logic. We added two new parameters: REFERENTIAL_INTEGRITY_VALIDATED and REFERENTIAL_INTEGRITY_FORCE_LINK_LOAD.

What happens when you activate the first parameter? VaultSpeed will store records in an error table if it cannot find the referenced key. These records will not be inserted in the link tables. Not loaded unless you activate forced link loading, it that case the tool will add a reference to the ‘unknown’ record.

In addition VaultSpeed will add error table records when processing subsequent loads. This allows us to check whether the reference can be resolved with new data available.

Other Changes

  • You can now create a self reference relationship which contains an attribute that references itself i.e. (c,b) → (a,b). You can do this by right clicking the attribute while in relation creation mode.
Self Reference options in VaultSpeed
  • We added the ability to update and save multiple records at the same time to the following screens: Source objects & attributes, DV name management, DV data types screens, Business vault views.
New Save buttons
  • You can now revert SAT splits.
  • We added subscription info to the dashboard. You can also see the actual number of concurrent users.
VaultSpeed Dashboard page
  • You can now switch the ODI_FOLDER_SPLIT parameter after having already deployed the project (for customers using Oracle Data Integrator).

SAT split editor improvements

Our developers also made some improvements to the SAT-split editors based upon customer feedback (we added a new SAT-split screen in a previous release):

    • We now have an option to change the number of visible attributes in a split (just click the button with 3 vertical dots and go to settings).
    • We added an option to sort the attributes in descending order in the context menu.
    • The editor now has the option to show only unused attributes to the context menu.
    • You can now remove attributes from a split or add them to another split by dragging and dropping.
    • Long table names are now displayed correctly.
    • Improved handling of multiple splits.
    • The split name will no longer disappear when you scroll down the attribute list of a split.

Stay tuned for more release news!

Data Lineage Export (Release 4.1.14)

Data Lineage Export (Release 4.1.14)

In the upcoming release, we incorporated some important improvements. First, we added a first version of our data lineage export functionality. Second, we added support for higher customization of DDL Options. We also added an alternative screen to handle satellite splitting. And finally, after introducing a lot of new objects in the business vault, we added business vault loading to our Flow Management solution in Apache Airflow.

At VaultSpeed, we invest a lot of time analyzing customer feedback and enhancement requests. At regular points in time, we try to align this feedback with our product roadmap in order to stay in line with customers needs.

Data Lineage Export

With VaultSpeed we gather source metadata. This metadata can be enhanced, maintained and fine-tuned by users. In the end we use all this metadata to generate a valid Data Vault 2.0 model. With this in mind, we have all necessary information to provide customers with proper data lineage. It is now possible to export this information to a CSV format.

Exported Data lineage in csv format

Starting at the Data Vault overview screen, users will see 2 new buttons. The first button is for object level data lineage, the second one you can use for attribute level lineage. When you click these buttons a CSV file concerning the entire data vaults object or attribute lineage will be downloaded trough your browser.

This is the first iteration of data lineage functionality in VaultSpeed, we are looking at alternative export formats and integrations with popular data lineage tools and possibly also some data lineage views inside VaultSpeed.

DDL Options for each table type

Using VaultSpeed, you were already able to define certain DDL settings. With these settings, you can create optional DDL layouts for the tables that will be created throughout all layers of your data warehouse. For example you are able to define a certain level of parallelism, or set a specific partitioning type or other, depending on the specificities of your target database.

We now made it possible to create these DDL settings at table type level. So you could for example create a different setting for a HUB compared to a satellite.

Enhanced SAT split screen

Our UX designers try to improve VaultSpeed's look and feel continuously. They came up with a different layout for our satellite splitting screen. This version should work a lot better when you are splitting tables into more than 2 sattelite splits. Also, it is now possible to show multiple satellites in one canvas. Finally, it is more aligned with other graphical screens in the tool. So users should recognise the menu's from other screens.

New SAT-split screen

All previous functionalities are still available in this screen. The old screen is still present for user that prefer to work in the old one. We will analyze customer feedback as to which screen gets the best reviews.

Business Vault loading added to FMC

The FMC module or Flow Management Center is our Apache Airflow based solution to orchestrate flows. This module can be purchased in addition to a standard licence. It offers a solution for customers that additionally require an orchestration tool. In recent releases, we added a lot of new functionality in the Business Vault layer. This new logic can now also be loaded using our FMC module. VaultSpeed will generate Airflow DAGs for loading of PITs, Bridges and Cross source links. Read more about FMC on our features page.

Our FMC module generates DAGs for Apache Airflow

Tanks for reading. That's all for this release, stay tuned!

Best regards,
The VaultSpeed Team

Subscribe below and become part of VaultSpeed's ambassador community. Receive regular updates about new releases and share them within your network.

Cross Source Links (Release 4.1.12/13)

With release 4.1.12 and 4.1.13 we brought some interesting new features.

First of all, we included support for cross source exploration links. The main idea behind building an enterprise wide data warehouse is to be able to integrate data from multiple sources. This requirement is now delivered out of the box in VaultSpeed.

Vaultspeed already provided HUB Groups to integrate between sources at business key level. With our new Cross Source Link objects, you can create links between multiple sources. This adds an additional way to integrate and explore between sources.

In the source graphical editor an option has been added to select entities from other sources and their specific releases. By selecting and adding the object to the canvas, a relationship can be defined between objects from different sources. Notice that foreign objects will be shown in a different color and that the source name is mentioned in the object name.

Import object from another source and add a relationship

Once the object and the relationship have been added, new data vault and business vault releases can be generated. The resulting relationship will add a link object between both sources.

New link is added to the data vault model

We integrate cross source link objects at the Business Vault level. There are several reasons why this is the best place for the integration:

First of all, cross source links create dependencies between sources, we do not want inter source dependencies between sources in the raw data vault because it would complicate the loading process.

Second, we have to make sure that all the data has arrived in the raw data vault to process our cross source links. With data coming in from different sources at different times, there might be issues with “missing links”. VaultSpeed takes care of this issue.

Finally, by placing the cross source link in the business vault layer, we make it less dependent from underlying source changes. This facilitates version management and release management of these structures.

Also included in this release is a new type of delete logic, INSERT_ON_DELETE: when this parameter is enabled, deletes will insert a new record in the satellite instead of executing an update, these delete records will be end dated just like regular records. This parameter has no effect when using INSERT_ONLY_LOGIC.

We also improved performance and look and feel for the source graphical editor. The objects will now also be displayed in a tree like structure to give a better overview of the source.

Tree like structure to show source objects in the graphical editor

More new stuff coming soon!

Bridge tables (Release 4.1.11)

The start of a new year welcomes a new release. VaultSpeed kicked of 2020 with some brand new features in version 4.1.11. We added support for bridge tables in a new graphical editor and added the possibility to import and export business names in the business vault. By adding bridge tables, VaultSpeed now supports the 5 main objects in data vault 2.0: Hubs, Links and Satellites in the Raw Data Vault. But also PITs and Bridges in the Business Data Vault. The graphical bridge editor enables users to generate bridge tables based upon raw data vault objects. All available Hubs can be dragged and dropped to the editor.



Once a certain hub is selected on the canvas, the adjacent hub tab becomes active. It contains all adjacent hubs for the selected hub. Each adjacent hub can be dragged into the editor. Hubs can have multiple links to the same adjacent hub. It is not possible to add more than one link connection for a starting / ending hub.



The Bridges tab contains all available bridges. It is possible to remove, edit or create a bridge. To edit a bridge, first load it by dragging it into the editor, then you can start modifying it. To save the changes, click the edit button which is above bridge list in order to complete the edit. To create a new bridge, build it in the editor and click the create new bridge button. Creating a new bridge and editing a bridge is only possible when the bridge combination is valid, otherwise a warning message will be displayed.




It is possible to let the app generate possible paths between two hubs. To do this, Right click and select draw shortest path relation, then drag a line between the 2 hubs you want to connect. Then you can change the number of possible paths it will generate by setting “max generation options”, as well as excluding hubs from the paths. If there is only one viable option, then it will be directly applied to the editor. However, if there is more than one option, then an extra tab becomes active and it is possible to preview, regenerate or apply options to the paths. To preview or regenerate a path, you need to selected a bridge and click one of the buttons above the shortest path options list. During preview editor becomes read-only. To apply the selected shortest path option while in preview mode, click the apply button, otherwise you can drag and drop the desired path option into the editor.