Hybrid Data Management

What is Data Management and why is it required?

Data is the most important asset of an enterprise. It’s no secret that data is growing at a pace that is making enterprises not just take notice but also make a hard stop, re-strategize and change the game plan other-wise they will fall behind in the fast lane.

Growth in data volume is one thing but there are other aspects around it that become tricky and difficult to manage. These aspects include data formats, storage, security, consumption in the form of analytics, business insights.

Therefore, every enterprise needs a data management strategy to address the following:

• Data Governance
• Data Operations
• Data Delivery

Key benefits of a data management strategy include:

• Higher Data Quality
• Better Business Insights driving informed business decisions
• Improved Data Security

Why Hybrid Data Management?

To understand hybrid data management better, it is important to understand the differences between traditional and emerging data & analytics platforms.

Key differences between traditional & emerging data management approach include:

As it’s quite apparent from the above comparison, there are advantages and disadvantages for each approach.

This drives the need for embracing a hybrid data management approach which can combine the advantages of both the approaches. There are also some other factors like the following that require a hybrid approach:

• Mixed analytical work-loads
• Long-tail usage of traditional data platforms as change to emerging is not easy
• Time-spent on data preparation & movement rather than transformations and analytical processing

Hybrid approach can scale with the growing enterprise needs, increase agility, enable innovation, increase predictability, improve forecasting accuracy, detect new behavioral patterns and deliver analytical insights relevant to the business processes and applications.

Deploying hybrid data management can be started anytime based on the immediate business need or a challenge like growing data volume and limitation with a physical data center/platform setup. Some other considerations before deploying include:

How to deploy Hybrid Data Management?

• Impact on current technical landscape which includes impacts on on-prem infra, data availability, data access, data movement & processing
• Impact on business processes
• Investment considerations
• Technology choices & adoption

The following diagram illustrates a hybrid data management reference architecture for a data warehouse and a data lake platform however it is not limited to what’s in the diagram. This can be customized further based on the enterprise architecture:

The hybrid data management combines the features from both the traditional and emerging platforms.

For example, the following combinations can be incorporated as shown in the above diagram.

• On premises + Cloud
• Structured + Unstructured data
• Enterprise Data Sources + External Source APIs like Social Media, Weather, etc,
• SQL + No SQL
• Data warehouse + Data Lake
• Commodity + Open Source

In principle, it’s extremely important for the CTOs/CDOs to look at a hybrid data management strategy so as to enable business with a robust data & analytics platform that drives agility, quality of data and insights to run and grow the business.