Presentation layer templates, Denodo virtualization and record tracking satellites (Release 5.2)

September 13th, 2022

DSC08181 2
Jonas De Keuster
Thumnail release 5 2

VaultSpeed has just significantly updated its offering.

This release includes many exciting features, such as the multi-layer Template Studio to help you build the presentation layer in your Data Vault. We implement support for virtualization in Denodo. Our secure connection setup now also supports Azure Key Vault and Java Key Store. We introduce Record Tracking Satellites and empty hub groups. And finally, our UX developers have worked the whole summer on improving VaultSpeed’s UI. Users will experience better navigation and better performance, a fresh look and feel and yes, dark mode has finally arrived.

Two-stage automation


The automation paradox indicates a trade-off between the repeatability of logic and the level of customization. That’s why we created two separate automation stages in our tool. The cut-off point between the two stages lies at the crossroads where the level of repeatability exceeds the boundaries of separate organizations.

More specifically, we place this cut-off point in the Business Data Vault area.

Most objects we use in this area and in preceding layers contain highly repeatable Data Vault 2.0 logic: hubs, links, satellites, PIT (Point In Time) and bridge tables.

These objects look the same everywhere around the world. That’s why we firmly believe in strictly following the Data Vault 2.0 standard in this area of the DWH to get the best results. Breaking the patterns and deviating from the standard will result in technical debt and costly rework.

Other objects in the Business Data Vault layer and subsequent layers, such as computed satellites, dimensions, facts, flat tables, etc., require more customization in code. This is where data engineers can excel.

Screenshot 2022 09 13 at 17 45 13

Stage 1 — Build your ideal Data Vault model using our graphical modeling interface. No need to write any code. VaultSpeed delivers no-code Data Vault 2.0 automation using out-of-the-box templates. Configuration is done using parameters that influence template variables and generated code.

Stage1

No-code Data Vault 2.0 automation through our data modeling graphical user interface. Parameters influence the resulting code.

Multi-layer Template Studio

We’ve vastly upgraded our Template Studio module to give engineers the keys to our automation engine in stage 2. They get the ability to code custom templates to build any structure they want on top of the Raw Vault.


Template Studio is now multi-layered. You can build templates and refer to the data structures you created in preceding layers.

Multilayer
Virtualmarts

Example of objects needed to build a virtual data mart on Data Vault 2.0. Using elements from both the Raw and Business Data Vault.

Signature schemas and objects

To allow users to structure their templates in multiple layers, we’ve added signature schemas and signature objects.

A signature schema will define a database schema to which you can deploy data structures. For example, the information mart schema or the Business Vault schema (the latter is the default). A signature object will define a particular object type. For example, a dimension, a custom pit, a computed satellite, or a fact. These signatures can be reused in subsequent layers to build one template on top of another.

Signatures

With these signatures, you can build repetitive patterns for the presentation layer faster. The example below shows how objects are built on top of each other in subsequent layers of the DWH. Computed satellites and PITs in the Business Vault are re-used by the dimensions and facts in the data marts.

Templatestudioexample

We’ve also altered the template language a bit so developers can use and reuse the new signatures. For example, to expose Business Vault calculated fields in a star schema dimension. You can reuse existing objects and attributes, create new calculated fields, and build custom and repeatable aggregates by writing only a few lines of template code.

Templateandsqlcode

Template code (left) translated into SQL (right)

The final step is to indicate which templates will be applied to which objects. For example, you want to create a computed age satellite on ALL Raw Vault satellites. Or you want to apply the dimension template on all hubs. ELT code will be automatically generated and deployed for all dependent objects.

More details about the Template Studio can be found on the docs portal.

Denodo integration

VaultSpeed Template Studio objects can be deployed as:

  • Tables (ETL and DDL template)
  • Database views (only ETL template)

With this release, we’ve added a third option: remote views. A remote view allows you to deploy views in third-party virtualization solutions such as Denodo.

VaultSpeed will generate 2 types of views that can be deployed automatically into Denodo:

  • wrapper views in Denodo which reference the Raw Vault objects in the database
  • new views based on Studio templates

More info

Denodo

Supernova view in Denodo, generated by VaultSpeed

An interesting example of building supernova views with VaultSpeed and Denodo can be found in our docs portal.

Safer storage for agent connections

Agent connections will be stored more securely. We’ve added support for storing the connections in Azure Key Vault, Java Key store, and in a directory with one file per connection.

For more information and setup instructions go to Storing connections.

Empty hub groups for intuitive modeling

VaultSpeed has been working on a new Data Vault interface that allows for more intuitive Data Vault 2.0 modeling. We want data modelers to be able to use third-party modeling tools to create conceptual and logical data models.


Using API integrations, we allow them to create hub groups based on a conceptual data model, before adding sources that will provide the objects that populate that hub.

An example of such an import from Ellie was posted on our community VaultSpeed Community (please register to see the page).

Record tracking satellites (RTS) support

VaultSpeed now supports record tracking satellites (RTS). Users get to enable a new object parameter ADD_RECORD_TRACKING_SATELLITE for No-CDC objects. This action will attach an RTS object to its hub, keeping track of which keys existed in each load. Business rules allow for the distinction between a record that was not loaded due to a lock in the source and a true delete.

New UI in beta

Light, dark or classic mode? Try our new UI, now in beta!

New UI

Besides new color schemes, the new UI has a bunch of improvements:

  • better performance
  • better pagination
  • better navigation
  • URLs contain more information, meaning that you can now have a URL pointing to the source editor of a specific release
  • editing is now easier with our new actions menu and replaces double-clicking.

Stay informed on product news and sign up for our newsletter.