Running Data Vault on Snowflake with VaultSpeed

Snowflake

Snowflake is one of the most popular data platforms around these days. The success of their recent IPO emphasizes that. We support Snowflake as a target and starting now, the integration just got better.

Following up on exciting developments in Snowflake, VaultSpeed now generates EL T code for Snowflake that is wrapped in Javascript procedures. From now on you can actually deploy and store your procedures inside Snowflake. One of the main advantages of this is that our workflow solution in Airflow can call these procedures instead of executing saved SQL files.

We also enabled auto deploy for Snowflake : you can now deploy DDL and ETL to the target using VaultSpeed’s Agent. Both changes together make it possible to start loading your data warehouse without any manual interaction by your developers.

VaultSpeed Studio

In a previous release we announced VaultSpeed studio in closed alpha version. In the past few weeks we went to work with initial customer feedback. First thing we improved is the integrated template code editor. It includes smoother navigation options, changing and saving a template is more solid and you now have the option to compare your changes to the pre-edit version.

We added a view all option to the target definition where you can see all attributes of the template target. Previously, both existing and newly created target attributes were shown in separate windows.

In a previous post we explained the need for signature fields when doing automation. We did a complete redesign of the signature definition screen. You can create and select signature attributes on top of the screen and assign them to attributes in a list below. This list can be filtered before assigning a certain signature to a filtered set of fields.

VaultSpeed studio will move to open alpha in one of the next releases. From then on, all clients can start a one month free trial period with VS Studio.

Talend

Talend is one of the first ELT tools for which we supported automation. This is done by generating jobscript that can be deployed to generate ELT mappings inside Talend Studio’s repository.

Due to improvements and changes in their product, the need for a major update of our template compiler for Talend became apparent. The result is that the generation of Talend jobscript is much more robust. We also took a giant leap in terms of speed: jobscript generation is up to 3 times faster compared to the previous release.

Quality of life changes

Some smaller changes will certainly improve user experience:

• We added a download button to the automatic deployment screen. This allows users to download generated code through the browser instead of having to obtain it from the agent.

• We improved the description of the attribute based SAT split.
• You can directly upload CSV files to update short and abbreviated names in the source trough your browser.
  
• We moved the automatic deployment menu out of the generation settings since it didn’t really belong there.
  
• Users will experience improved loading performance of parameter screens.
• We added an extra tab to the DDL settings page where you can get an overview of all applied settings per layer.
• Business views are checked for naming conflicts with other objects. We also provided users with a button to disable/enable generation for all business views (to use in combination with filters).
• Releases in the delta generation tab are now sorted such that the latest one is always shown first.
• We added subscription updates as data points to the generations graph on the dashboard. It will also show the next subscription reset date.
  Based on the number of jobs granted in your subscription, we set a warning when the mapping counter is above 80% or at 100% of the subscription limit.
• And a lot more that did not make this post…