Data Lake as an agile and scalable enhancement to the Data Warehouse

Data Lake as an agile and scalable enhancement to the Data Warehouse

An increasing level of digitalization in the organization poses new data storage challenges as you discover that the enterprise data warehouse (DWH) you have built up over the years no longer meets your requirements? The specialized departments need the rapid implementation of analyses and reports in order to keep pace with the speed of the markets. But does the mapping in the data warehouse take too long? Or do your new data-driven business models require information and analysis in seconds to respond to customer needs? Is the amount of structured and unstructured data constantly increasing?

With a Data Lake, you can add or extend the existing DWH to meet the growing needs of the digitized enterprise for data handling. A Data Lake is flexible in storing any format, agile in responding to multiple queries, fast in real-time data processing, and scalable in processing power and storage capacity.

Make more out of your data with a structured data lake

The idea that with a Data Lake, all data literally disappears into a large lake and is "fished out" when needed is a thing of the past. Such a data lake "swamp up" too fast, and the initially praised advantage of the unstructured collection of all data becomes an ever-larger disadvantage.

It’s important to combine the benefits of a DWH with those of a data lake in what Trivadis calls a managed data foundation. A governed data lake is an important component of any managed data foundation. It adds structure to the data lake in the form of zones and an archive, yet the data lake continues to be a flexible repository for all enterprise data that by far surpasses the capabilities of a DWH in terms of complexity, size and scalability.

cp-Jackpot-dank-Data-Lake_ib

We're bringing all your company's data together at Data Lake. Create new connections. And make sure that you get sound analyses for your business or project in fractions of a second.

 

In addition to greater agility, the capability to handle larger volumes of data and the analysis options, a data lake as we understand it (with fast data and event hub) supplements the pure DWH with functions such as stream processing and real-time analysis, which are often needed today. On the one hand the data is collected and stored in near real time – from machines or vehicles which can easily generate many terabytes of data in a matter of hours. On the other hand, this kind of fast data also has to be processed and analyzed in a matter of seconds to derive any benefits. The near real-time processing of fast data is one side of the coin. The other is pre-filtering to sort the data in advance and ensure that only the data which is necessary for the analysis is made available, while the other data is stored for use at a later time.

The governed data lake is therefore the ideal basis for a data lab used by data scientists for exploratory data analysis. There is no limitation on the types of data formats that can be stored. The data lake supplements and replaces parts of the DWH, whereby the classic DWH still retains its traditional end-user functions.

One-stop data lake consulting, planning, implementation and operation

As data experts, we prefer to use tools such as Spark, Kafka, Streamsets and Hadoop or BlobStores for cloud data lakes, but we also have expertise in the use of any other tools you may have on your premises. We also prefer to use our proprietary biGENiUS tool to model and implement our mature data lakes with a high level of governance. Originally developed for DWH automation, today we take advantage of the tool’s strengths for data lakes.

Even though data lakes may initially seem to be predestined for the cloud, there are cases where an on-premises solution makes more sense overall. The cloud pays off above all when it can show off its advantages in terms of elasticity, i.e. when high compute performance is required but is not constantly accessed.

We at Trivadis have been dealing with data for over 25 years, so we have the experience and knowledge to assess whether a Data Lake is a useful addition to your existing DWH or even necessary. To find the right strategy for handling your data, we review the performance of your DWH and analyze how your company's requirements will evolve in the future, and whether a Data Lake makes sense. We see it as our task to generate added value for your business from your data.

Do you have questions or need help with your project?
We are here to help you.