VaultSpeed has just significantly updated its offering.
This release includes many exciting features, such as the multi-layer Template Studio to help you build the presentation layer in your Data Vault. We implement support for virtualization in Denodo. Our secure connection setup now also supports Azure Key Vault and Java Key Store. We introduce Record Tracking Satellites and empty hub groups. And finally, our UX developers have worked the whole summer on improving VaultSpeed’s UI. Users will experience better navigation and better performance, a fresh look and feel and yes, dark mode has finally arrived.
Presentation layer templates, Denodo virtualization and record tracking satellites (Release 5.2)
September 13th, 2022
Two-stage automation
The automation paradox indicates a trade-off between the repeatability of logic and the level of customization. That’s why we created two separate automation stages in our tool. The cut-off point between the two stages lies at the crossroads where the level of repeatability exceeds the boundaries of separate organizations.
More specifically, we place this cut-off point in the Business Data Vault area.
Most objects we use in this area and in preceding layers contain highly repeatable Data Vault 2.0 logic: hubs, links, satellites, PIT (Point In Time) and bridge tables.
These objects look the same everywhere around the world. That’s why we firmly believe in strictly following the Data Vault 2.0 standard in this area of the DWH to get the best results. Breaking the patterns and deviating from the standard will result in technical debt and costly rework.
Other objects in the Business Data Vault layer and subsequent layers, such as computed satellites, dimensions, facts, flat tables, etc., require more customization in code. This is where data engineers can excel.
Stage 1 — Build your ideal Data Vault model using our graphical modeling interface. No need to write any code. VaultSpeed delivers no-code Data Vault 2.0 automation using out-of-the-box templates. Configuration is done using parameters that influence template variables and generated code.
No-code Data Vault 2.0 automation through our data modeling graphical user interface. Parameters influence the resulting code.
Stage 2 — Build custom integration templates to automate layers beyond the Raw Data Vault. Build computed satellites, links, custom PITs, facts and dimensions, using the same template engine. Capitalize on the quality of the highly standardized structures generated in the Raw Data Vault but experience total freedom in developing templates for end-user data consumption.
Multi-layer Template Studio
We’ve vastly upgraded our Template Studio module to give engineers the keys to our automation engine in stage 2. They get the ability to code custom templates to build any structure they want on top of the Raw Vault.
Template Studio is now multi-layered. You can build templates and refer to the data structures you created in preceding layers.
Example of objects needed to build a virtual data mart on Data Vault 2.0. Using elements from both the Raw and Business Data Vault.
Signature schemas and objects
To allow users to structure their templates in multiple layers, we’ve added signature schemas and signature objects.
A signature schema will define a database schema to which you can deploy data structures. For example, the information mart schema or the Business Vault schema (the latter is the default). A signature object will define a particular object type. For example, a dimension, a custom pit, a computed satellite, or a fact. These signatures can be reused in subsequent layers to build one template on top of another.
With these signatures, you can build repetitive patterns for the presentation layer faster. The example below shows how objects are built on top of each other in subsequent layers of the DWH. Computed satellites and PITs in the Business Vault are re-used by the dimensions and facts in the data marts.
We’ve also altered the template language a bit so developers can use and reuse the new signatures. For example, to expose Business Vault calculated fields in a star schema dimension. You can reuse existing objects and attributes, create new calculated fields, and build custom and repeatable aggregates by writing only a few lines of template code.
Template code (left) translated into SQL (right)
The final step is to indicate which templates will be applied to which objects. For example, you want to create a computed age satellite on ALL Raw Vault satellites. Or you want to apply the dimension template on all hubs. ELT code will be automatically generated and deployed for all dependent objects.
More details about the Template Studio can be found on the docs portal.
Denodo integration
VaultSpeed Template Studio objects can be deployed as:
- Tables (ETL and DDL template)
- Database views (only ETL template)
With this release, we’ve added a third option: remote views. A remote view allows you to deploy views in third-party virtualization solutions such as Denodo.
VaultSpeed will generate 2 types of views that can be deployed automatically into Denodo:
- wrapper views in Denodo which reference the Raw Vault objects in the database
- new views based on Studio templates
Supernova view in Denodo, generated by VaultSpeed
An interesting example of building supernova views with VaultSpeed and Denodo can be found in our docs portal.
Safer storage for agent connections
Agent connections will be stored more securely. We’ve added support for storing the connections in Azure Key Vault, Java Key store, and in a directory with one file per connection.
For more information and setup instructions go to Storing connections.
Empty hub groups for intuitive modeling
VaultSpeed has been working on a new Data Vault interface that allows for more intuitive Data Vault 2.0 modeling. We want data modelers to be able to use third-party modeling tools to create conceptual and logical data models.
Using API integrations, we allow them to create hub groups based on a conceptual data model, before adding sources that will provide the objects that populate that hub.
An example of such an import from Ellie was posted on our community VaultSpeed Community (please register to see the page).
Record tracking satellites (RTS) support
VaultSpeed now supports record tracking satellites (RTS). Users get to enable a new object parameter ADD_RECORD_TRACKING_SATELLITE for No-CDC objects. This action will attach an RTS object to its hub, keeping track of which keys existed in each load. Business rules allow for the distinction between a record that was not loaded due to a lock in the source and a true delete.
