Source copy, Databricks and Apache 2.0 (Release 4.2.4)

Source copy, Databricks and Apache Airflow 2.0 (Release 4.2.4)

We’re back with a new release, and it is stuffed with new features.
We added support for Databricks, we updated our Flow Management connector to work with Apache Airflow 2.0. Also, VaultSpeed users can now copy an entire source configuration. These, and many more changes, come with VaultSpeed R4.2.4!

Databricks

Run your Data Vault in the Databricks data lakehouse!
You are now able to generate and deploy Spark code to Databricks and run it with Airflow. The deployment will create Spark SQL notebooks in Databricks for all your Data Vault mappings. Airflow will launch those jobs, running the Notebooks. Integration with Azure Data Factory is coming soon.
The target Database type is still Spark, but the ETL generation type has to be set to Databricks SQL.

 

Airflow 2.0

Apache Airflow 2.0 brings a truckload of great new features like a modernized user interface, the Airflow API, improved performance of the scheduler, the Taskflow API and others. VaultSpeed now supports Airflow 2.0. The VaultSpeed plugin for Airflow and all generated code have been reworked. All code will still work for previous Airflow versions. Just like before, once you’ve installed our plugin into your Airflow environment, Airflow becomes VaultSpeed aware. You’re able to generate and deploy workflows and run all the code needed to load your Data Vault.

 

Copy Sources

Users will also have the ability to copy existing sources. In some cases, an organization will need to integrate multiple sources that share a lot of similarities between them.

To give an example: Company ABC has the same version of their Sales CRM running in both Europe and the US. The only difference is that they have a few additional modules activated in the US.

Using the source copy functionality, they can now copy the entire source configuration from EU Sales to US Sales. All you need to do is identify and configure objects or settings that are specific only for the new source, but you can now skip all similar configuration you had already done for the EU source.

Using this functionality can obviously save a lot of time when integrating similar sources into your Data Vault model.

 

 

User Experience Improvements

The new release comes with a few other changes like a better screen to create a new data vault release. It has become a lot easier to indicate which version of which sources you would like to include in a specific data vault release. You can also choose to exclude certain sources from your release.

 

We also made it possible to mark objects in the source editor as completed, the completed objects will be highlighted in green. This status can be toggled by right-clicking on an object. The selection page can filter out completed objects, and there is also a button to remove all completed objects from the canvas. This allows you to track progress in your source modelling and get things organized.

 

Business Keys

We made it easier to change and re-order business keys in the Data Vault. We added a new screen where for each hub group the business keys of the grouped objects can be renamed and reordered, and the business keys of the hubs in the group can be reordered to match. So the keys in the different sources can now have different orders and names and still result in the same hash key calculation.

 

 

We added similar ability to reorder the linked hubs in many-to-many links ( and non historized links). In a separate screen, you can change the order of the HUB’s included in a many to many link or non-historical link.

Other changes

  • We renamed the “build flag” property to “ignored” everywhere in the application.
  • Added extra template variables for the custom deploy scripts in the agent, instead of only the zip name, you can now also get the generation id, the generation info, and the generation type, similar to the git commit message functionality. example:
    deploy.cmd = sh C:\Users\name\Documents\agent\deploy.sh {zipname} {code_type} ”{info}”
  • The compare functionality in the source graphical overview will now skip ignored releases. This means that it will compare with the last non ignored locked release before the current one.
  • We added support for overlapping loading windows to the Azure Data Factory FMC, this can be configured by using the following parameters: FMC_OVERLAPPING_LOADING_WINDOWS, FMC_WINDOW_OVERLAP_SIZE, FMC_WINDOW_OVERLAP_TYPE.
  • The metadata-export has been converted to a task, this is done to support exporting data for very large Data Vaults. Before the export would time out and not return a file if it takes too long.

More releases are coming!

Want to stay up to date? Subscribe to our newsletter!



Whitepaper: Accelerate the mapping of your business taxonomy with VaultSpeed


Learn how the VaultSpeed automation tool is designed to transfer any business taxonomy you might think of, into the raw Data Vault layer.
The Raw Data Vault (RDV) contains what Data Vault 2.0 calls the ‘Single Version of the Facts’. Facts are nothing more than the raw, historical, unfiltered data from the sources.

The Business Data Vault (BDV) aligns business keys/terms from the source system with the different business views in order to ensure compliance. Different viewpoints coexist and are all regarded by Data Vault 2.0 to be valid versions of the truth.

You’ll discover that these ‘versions of the Truth’ and the ‘Single version of Facts’ can truly blend.

 

Complete the form to receive the whitepaper



WWDVC Roadmap Video: Survival of the Fastest

We are living in an environment that continues to evolve. That’s why we’re happy to share a sneak peek of our Roadmap and how we see the world of data warehouse automation will evolve in the near and not so near future.

Complete the form to watch this video


WWDVC Keynote Video: Automation in the Real World

There are a lot of aspects that can make or break your data warehouse project.

Discover how to overcome challenges in cloud architecture, business requirements and time to market with case studies from Eurocontrol, Olympus and Bank de Groof Petercam.

Complete the form to watch this video



Meet EON Colletive: our new Integration Partner in North America

Meet EON Colletive: our new Integration Partner in North America

Important news from the partnership front! We have recently teamed up with EON Collective. EON is a group of highly experienced data professionals located in USA & Canada. EON will act as an integration partner for VaultSpeed in the region.

We’re delighted to announce EON Collective as our newest integrator partner in North America. EON have strong focus on automation and are very familiar with Data Vault 2.0. Their expertise in data warehousing and data integration is impressive and we’re happy to team up with such a strong player.

Piet De Windt - CEO VaultSpeed

Every EONite team lead has over 20 years of experience in their discipline. They all at one time or another have worked for one of the world's largest consulting firms and all understand that that real change doesn't have to cost an arm and a leg. With that in mind, EON Collective's team developed the tools that lower the hours needed to bring you real results. They help organizations gain validated business insights faster and with greater flexibility. And help companies ensure business value through proven methodology and automated tools.

They are partnering up with VaultSpeed as their preferred solution for data warehouse automation:

We are very excited about being Vaultspeeds North American integration partner. Automation is a key component of any successful Data Vault implementation and we feel Vaultspeeds automation strategy in combination with Adept methodology for Data Vault implementation is the perfect combination.

We are also looking forward to working with Vaultspeed as we start to integrate some of our Adept technology with the Vaultspeed solution."-

Robert Scott - CTO EON Collective

The power of EON is having the collective capability to work alongside their clients utilizing the ADEPT Managed Solution. EON ADEPT links process model analysis and data-oriented analysis. In fact, ADEPT is not limited to automated process discovery based on event data. It also answers a wide variety of clients performance and compliance questions based on the identified solution's operational metrics. ADEPT was built with the simple goal of greatly reducing the cost of consulting.

We should also mention that EON are joining us at the World Wide Data Vault Conference starting May 17th. Any questions on how to get started with VaultSpeed in their region and about the ADEPT integration can be addressed. We can highly recommend Keith Belanger’s keynote presentation “Is your Data Vault speaking your language?”.


VaultSpeed @ World Wide Data Vault Consortium 2021

VaultSpeed @ World Wide Data Vault Consortium 2021

You simply have to check out the annual World Wide Data Vault Consortium on May 17 2021.
This is where the worldwide user community comes to get in-depth knowledge presenters about data hubs, the role of A.I., and of course, automation.

At the conference, VaultSpeeds will host three events: A hands-on demo session, roadmap presentation and customer succes stories.
What’s more, you can get a special 20% reduction on the subscription from us.

Hands-on Session

Skip most of the data integration preparation with VaultSpeed

VaultSpeed’s data warehouse automation enables organizations to integrate data from numerous source platforms into one data vault. We harvest source metadata, users configure their source models and our engine delivers generated structures, ELT and workflows. Vaultspeed’s guided automation framework helps the user to combine and enrichmetadata from different sources in an intuitive way that corresponds to your target model.

Our out-of-the-box templates cover 90% of your implementation needs. They are 100% production-ready as VaultSpeed handles all the quality assurance and testing. We simplify the complex process of building a data vault by forcing the user to follow a pre-defined set of steps. This significantly reduces the chance of errors and ensuing rework.

VaultSpeed is quickly evolving. New functionalities are implemented every three to four weeks. Our cloud setup ensures our customers always run on the latest version.

We always try to help users eliminate time-consuming manual work and constantly work at developing new features by which they can reach even higher levels of automation.

Key takeaways:

- Integrate sources into your data warehouse quickly using Data Vault 2.0 and VaultSpeed

- Use VaultSpeed’s powerful source editor to tailor the Raw Data Vault and Business Vault towards your business taxonomy.

Customer Succes

There are a lot of aspects that can make or break your data warehouse project. We’d like to cover three of those using three cases from the real world: Time to market, cloud architecture and fulfilling business requirements.

Learn how VaultSpeed is speeding up the implementation process at Eurocontrol, Europes Organization for the Safety of Air Navigation, using its out-of-the-box templates. One year into the project, Eurocontrol conducted an internal ROI analysis.

We’re launching a huge project at Olympus, a global player in the MedTech market. As they are playing on a global level, they are moving their data platform to the cloud. VaultSpeeds cloud architecture fits right in.

Finally, no project succeeds without fulfilling business requirements and speaking their language. At Bank de Groof Petercam, VaultSpeed enabled developers to map their business taxonomy to the data vault model.

Roadmap Presentation

The VaultSpeed automation is tool is at the top of the evolutionary/acceleration ladder.

Curious to know how a decade of hands-on experience in data integration projects has resulted in a SaaS platform that provides faster data warehouse automation?

For us, all along, that was the key of our evolution. It was not about our intrinsic strength or intellectual ability, but rather the ability to understand the difficulties that our customers encounter and adapting and tweaking our platform to help them survive.

And no, it definitely was not a rollercoaster. Or to quote Charles Darwin himself “In the long history of humankind (and animal kind, too) those who learned to collaborate and improvise most effectively have prevailed.”

We are living in an environment that continues to evolve. That’s why we’re happy to share a sneak peek of our Roadmap and how we see the world of data warehouse automation will evolve in the near and not so near future.


VaultSpeed enters the official DataVault 2.0 Certification Program

The Data Vault Alliance proudly announced their brand new Certified Software Vendor Program. VaultSpeed is happy to enter this new program as a continuation of our earlier certification efforts.

 

Why we use Data Vault 2.0?

Vaultspeed has based its automation engine for the integration layer on Data Vault 2.0. We have made this decision with a few key constraints in mind: Flexibility, agility, support to have multiple versions of the truth, repeatability and the use of a standard. Data Vault provides us with the best answer to these constraints.

Data Vault is very flexible as you can just add new business elements to the model without affecting previous efforts. Data Vault can easily absorb those changes. For the same reasons, it is a perfect fit for an agile approach. You can chop the entire workload in small, manageable sprints.

While there may be such a thing as the “single version of the truth”, we believe that it is almost impossible to obtain. Not everybody has the same point of view and this view may also change over time. This means that you will always have more versions of the truth. To achieve this, Data Vault starts from having a single version of the facts, this is the stable factor you need to be able to deliver multiple versions of the truth and still manage the data integration effort over time.

Data Vault is also perfect for automation. You can define a clear relation between source metadata and the the target model, and you can do so by using a limited set of repeatable patterns.

Why did VaultSpeed choose to be certified?

Vaultspeed values the Data Vault Standard for all the benefits it brings like resilience to change and repeatable patterns. Data Vault provides the foundation for automation. Being able to work with a well defined standard that is documented, used across the world, updated and safeguarded over time is key. In fact, this enables everyone to speak the same language. This emphasizes the importance of the Data Vault Alliance, led by founding father Dan Linstedt, as the organisation that sets the Data Vault standard. For these reasons we also want to have VaultSpeed Data Vault 2.0 certified by the DVA in order to prove that VaultSpeed provides the means to work by that very same standard.

Certified Data Vault 2.0

Starting in 2019 we started a track to get our tool certified togheter with Empowered Holdings and Scalefree.

Empowered Holdings, LLC and Scalefree teamed up in 2019 to work with VaultSpeed to get their Data Vault automation tool certified to Data Vault 2.0 standards. We are happy to announce that as of 2020, they have passed the tool certification process.

Certified Software Vendor Program

As of January 2021, Empowered Holdings LLC merged its Data Vault practices with DataVaultAlliance Holdings LLC. Going forward the DVA is currently developing a world-wide Vendor Tool Certification Program. This program and it’s details will be available to any software or hardware vendor interested in participating. The program will list a set of standards that the tool needs to meet, in order to have the components that automate Data Vault, be certified.

Read more about this program on the Data Vault Alliance’s website

 

 


Vaultspeed raises €3.6 Million Series A

Vaultspeed raises €3.6 Million Series A


Vaultspeed raises €3.6 Million Series A to accelerate growth and bring its best of breed data warehouse automation solution to the global market

PRESS RELEASE 17 March 2021 - Leuven, Belgium - Vaultspeed, the Belgium-based SaaS company specialized in data warehouse automation solutions, has closed a €3.6 million Series A round led by Fortino Capital Partners. The company was founded 2 years ago by Piet De Windt and Dirk Vermeiren with support of The Cronos Group, who remains on board through its seed investment fund the CoFoundry.

Vaultspeed’s data automation software serves data managers by accelerating and automating the entire lifecycle (design, build and maintain) of their Data Vault. Data Vault technology is an innovative approach to centralize enterprise data for business analysts who deliver the necessary real time insights that business leaders need to guide their decisions. This is where Vaultspeed comes into play. Dirk Vermeiren, CTO of Vaultspeed: “While ensuring quality and consistency, the tool automates the integration of data from multiple source systems into the Data Vault, making it available for further analysis throughout the enterprise. This is what agile business leaders need to accelerate their time to market, cut the complexity and reduce project risks.”

Vaultspeed’s customer base ranges from California (Department of Health of Santa Clara County) across Europe to Japan (Olympus). The rise of microservices driving the multiplication of distributed data sources, the move to the cloud, the increasing volume of data and degree of change and the scarcity of qualified talent stimulate organizations to respond faster and smarter, increasing the demand for automated data warehousing solutions.

Vaultspeed recently signed a global deal with the Japanese multinational Olympus who will be building regional as well as global data integration platforms in order to make faster and better use of their enterprise data and improve their customer-driven solutions for the medical, life sciences and industrial industries. By bringing Vaultspeed into the process, Olympus can now create faster and better integrated insights on their SAP data and non-SAP data which was not possible before.

Duco Sickinghe, Managing Partner at Fortino Capital: “We have seen a rapidly increasing traction for data vaults over the past years and are truly excited to support Piet De Windt (CEO) and Dirk Vermeiren (CTO) in accelerating their growth. Vaultspeeds’ data warehouse automation tool plays a crucial role in helping customers increase their agility, while responding to strong time-stamping, auditability and traceability requirements.”

Wim Bijnens, partner at CoFoundry: “We have seen Vaultspeed evolve from an idea into a prototype on to a proof of concept with some of our key customers in Belgium. Today, Vaultspeed is ready to scale up and deliver value to business leaders all over the world. With enthusiasm and belief in a great future for Vaultspeed we are pleased with the support of Fortino Capital in this exciting scaling phase.”

With the additional funding, Vaultspeed will look to further scale its organization and invest in its best of breed product in order to serve and expand its international customer base. Piet De Windt, CEO of Vaultspeed: “Vaultspeed’s cloud-based product is platform agnostic and integrates with top-tier tools in the data integration ecosystem. We strive to bring the best value and technology to our customers leveraging our strong and growing partner ecosystem. We are happy to onboard Fortino Capital and look forward to entering Vaultspeed’s next development phase together.”

About Vaultspeed

Vaultspeed is a Belgium-based software company. Its data warehouse automation solution speeds up the process of data integration through a best in class tool built on the Data Vault 2.0 methodology. More and more companies worldwide rely on Vaultspeed to simply build and maintain their enterprise data hub. The tool connects with most popular ELT(ETL)-tools, source, target technologies and orchestration engines.

About Fortino Capital Partners

Fortino Capital Partners is a Benelux-focused B2B software investor with a pan European reach. Fortino Capital invests in both Venture Capital and Growth private equity assets. With offices in Antwerp and Amsterdam, Fortino Capital’s investment portfolio includes Teamleader, Insided, MobileXpense, Efficy CRM, iObeya and Oqton among others.

For more information, please visit https://fortinocapital.com/

About Cofoundry

With a passion for innovation, The CoFoundry helps entrepreneurs transform their ideas into sustainable companies by funding them in a seed stage and by coaching them in the growth process. Embedded in the ecosystem of The Cronos Group, The CoFoundry has access to a wide network of relevant technology players.

For more information, please visit http://www.thecofoundry.co/

Read more...

Belgium:

International:


External tables & Template Previews (Release 4.2.3)

External tables & Template Previews (Release 4.2.3)

We have released VaultSpeed 4.2.3! Part of our focus was on improving performance of code generation tasks, but we also included some novelties.

External tables

We have extended DDL settings with support for INI and CDC layers. You can now generate table definitions for external/foreign tables.
This enables you to define INI and CDC tables as external tables, directly connecting them to csv’s, xls’s, db exports and many others... in your data lake.

 

 

VaultSpeed Studio Code Preview

VaultSpeed studio, our templating module now features previews. You can actually write a template and run a preview on a designated object to see what code it will generate. Copy paste preview code and test in on your development environment. Using preview, it won’t take long until you write the perfect custom template!

 

Preview for a custom effectivity SAT template

Performance improvements

We improved performance for delta generations. Delta’s calculate the difference between two separate Data Vault versions. They generate all necessary code to move from one version to the next.

 

Delta code generation

 

Calculation times were drastically improved, and the differences are especially important when changes are located only in a limited set sources.

Additionally, VaultSpeed’s agent will now only harvest metadata for objects that are included in a release, instead of all objects in the schema, this can greatly improve metadata retrieval performance for large sources with only a limited number of objects being used for the data warehouse.

Source editor improvements

We also added some new stuff in our source editor.

From now on you can use object or attribute mass update from the source graphical editor. This enables you to set the object type, CDC type, comments, data length,… for all objects or attributes matching a certain pattern.

 

Object/Attribute Mass Update

 

We improved the layout of objects in the source graphical editor, and added the ability to switch between vertical and horizontal orientation.

Third, we added shortest path functionality between 2 objects on the source graphical editor. Another tool that can help you to better understand your source model.

 

Shortest path

 

These changes were the final part in the roadmap to cover all functionality that was previously available in the tabular source editor, the old editor had become outdated and is no longer available.

 

 

For more info on VaultSpeed you can always subscribe below ⬇️

 

 

 

You value your privacy? We share your concern. So please check our privacy policy.


Azure Data Factory meets VaultSpeed FMC

Azure Data Factory meets VaultSpeed FMC (Release 4.2.2)

Some of our developers just don’t know how to stop. They came up with something new during Christmas Holidays: Support for Azure Data Factory (ADF) in our Flow Management Control (FMC) solution.

ADF FMC

In a previous blogpost we introduced our FMC solution on top of Apache Airflow. From now on, we also offer our workflow solution on top of Azure Data Factory.
This solution fits ideally for Azure DB or Synapse customers. They can use VaultSpeed to generate DDL and ELT to integrate their sources into the data warehouse. VaultSpeed can now also generate the orchestration in the form of json, which you can automatically deploy to ADF.

 

 

The VaultSpeed FMC for ADF uses Azure PAAS components exclusively. Azure Data Factory and your Data warehouse Database (SQL server or Synapse).
The database contains procedures and load metadata tables, meanwhile
ADF FMC will use stored procedure activities to execute those procedures.

 

Choose your preferred FMC platform

 

ADF has visual presentation of the workflows and built-in monitoring. ADF provides seamless pipeline restart-ability and failure management. You can also create Azure Dashboards based on ADF metrics. It is also possible to export the metrics into an external reporting tool like Power BI, Grafana or other candidates.

 

ADF FMC allows you to optimize parallelism and Azure cloud costs. Code generation is fully metadata driven while it still allows for integration with existing ADF pipelines like pre-staging or post processing.

Other changes

Despite this being a smaller release, some other quality of life changes are included. Every page in Vaultspeed now contains a link to the relevant VaultSpeed docs (book icon). We also added all the subscription info to the dashboard, such as extra modules and support tiers.

Want to stay tuned about our releases, leave your info below 👇 and we’ll add you to our mailing list.