Often, the data team needs different solutions to process unstructured data, enable business intelligence, and build machine learning models. But with the unified Databricks Lakehouse Platform, all these are unified. It also simplifies data processing, analysis, storage, governance, and serving, enabling data engineers, analysts, and data scientists to collaborate effectively.

For the cloud engineer, this is good news. Managing permissions, networking, and security becomes easier as they only have one platform to manage and monitor the security groups and identity and access management (IAM) permissions.

Challenges Faced by Cloud Engineers

Access to data, reliability, and quality, are key for businesses to be able to leverage the data and make instant and informed decisions. Often, though, businesses face the challenge of:

• No ACID transactions: As a result, updates, appends, and reads cannot be mixed
• No Schema Enforcement: Leads to data inconsistency and low quality.
• Integration with Data Catalog Not Possible: Absence of single source of truth and dark data.

Since object storage is used by data lakes, data is stored in immutable files that can lead to:

• Poor Partitioning: Ineffective partitioning leads to long development hours for improving read/write performance and the possibility of human errors.

• Challenges to Appending Data: As transactions are not supported, new data can be appended only by adding small files, which can lead to poor quality of query performance.

Databricks Advantages

Databricks helps overcome these problems with Delta Lake and Photon.

Delta Lake: A file-based, open-source storage format that runs on top of existing data lakes, it is compatible with Apache Spark and other processing engines and facilitates ACID transactions and handling of scalable metadata, unifying streaming and batch processing.

Delta Tables, based on Apache Parquet, is used by many organizations and is therefore interchangeable with other Parquet tables. Semi-structured and unstructured data can also be processed by Delta Tables, which makes data management easy by allowing versioning, reliability, time travel, and metadata management.

It ensures:

• ACID
• Handling of scalable data and metadata
• Audit history and time travel
• Enforcement and evolution of schema
• Supporting deletes, updates, and merges
• Unification of streaming and batch

Photon: The lakehouse paradigm is becoming de facto but creating the challenge of the underlying query execution engine unable to access and process structured and unstructured data. What is needed is an execution engine that has the performance of a data warehouse and is scalable like the data lakes.

Photon, the next-generation query engine on the Databricks Lakehouse Platform, fills this need. As it is compatible with Spark APIs, it provides a generic execution framework enabling efficient data processing. It lowers infrastructure costs while accelerating all use cases, including data ingestion, ETL, streaming, data science, and interactive queries. As it does not need code change or lock-in, just turn it on to get started.

Read more on how Indium can help you: Building Reliable Data Pipelines Using DataBricks’ Delta Live Tables

Databricks Architecture

The Databricks architecture facilitates cross-functional teams to collaborate securely by offering two main components: the control plane and the data plane. As a result, the data teams can run their processes on the data plane without worrying about the backend services, which are managed by the control plane component.

The control plane consists of backend services such as notebook commands and workspace-related configurations. These are encrypted at rest. The compute resources for notebooks, jobs, and classic SQL data warehouses reside on the data plane and are activated within the cloud environment.

For the cloud engineer, this architecture provides the following benefits:

Eliminate Data Silos

A unified approach eliminates the data silos and simplifies the modern data stack for a variety of uses. Being built on open source and open standards, it is flexible. Enabling a unified approach to data management, security, and governance improves efficiency and faster innovation.

Easy Adoption for A Variety of Use Cases

The only limit to using the Databricks architecture for different requirements of the team is whether the cluster in the private subnet has permission to access the destination. One way to enable it is using VPC peering between the VPCs or potentially using a transit gateway between the accounts.

Flexible Deployment

Databricks workspace deployment typically comes with two parts:

– The mandatory AWS resources

– The API that enables registering those resources in the control plane of Databricks

This empowers the cloud engineering team to deploy the AWS resources in a manner best suited to the business goals of the organization. The APIs facilitate access to the resources as needed.

Cloud Monitoring

The Databricks architecture also enables the extensive monitoring of the cloud resources. This helps cloud engineers track spending and network traffic from EC2 instances, register wrong API calls, monitor cloud performance, and maintain the integrity of the cloud environment. It also allows the use of popular tools such as Datadog and Amazon Cloudwatch for data monitoring.

Best Practices for Improved Databricks Management

Cloud engineers must plan the workspace layout well to optimize the use of the Lakehouse and enable scalability and manageability. Some of the best practices to improve performance include:

• Minimizing the number of top-level accounts and creating a workspace as needed to be compliant, enable isolation, or due to geographical constraints.
• The isolation strategy should ensure flexibility without being complex.
• Automate the cloud processes.
• Improve governance by creating a COE team.

Indium Software, a leading software solutions provider, can facilitate the implementation and management of Databricks Architecture in your organization based on your unique business needs. Our team has experience and expertise in Databricks technology as well as industry experience to customize solutions based on industry best practices.

FAQ

The post What Cloud Engineers Need to Know about Databricks Architecture and Workflows appeared first on Indium.

L.L. Bean turned to Google Cloud to fulfill its cloud requirements. By modernizing data on, it experienced faster page loads and it was able to access transaction histories more easily. It also focused on value addition instead of infrastructure management. And, it reduced release cycles and rapidly delivered cross-channel services. These collectively improved its overall delivery of agile, cutting-edge customer experience.

Data Modernization with Google Cloud for Success

Many businesses that rely on siloed data find it challenging to make fully informed business decisions, and in turn accelerate growth. They need a unified view of data to be able to draw actionable, meaningful insights that can help them make fact-based decisions that improve operational efficiency, deliver improved services, and identify growth opportunities. In fact, businesses don’t just need unified data. They need quality data that can be stored, managed, scaled and accessed easily.

Google Cloud Platform empowers businesses with flexible and scalable data storage solutions. Some of its tools and features that enable this include:

BigQuery

This is a cost-effective, serverless, and highly scalable multi-cloud data warehouse that provides businesses with agility.

Vertex AI

This enables businesses to build, deploy, and scale ML models on a unified AI platform using pre-trained and custom tooling.

Why should businesses modernize with Google Cloud?

It provides faster time to value with serverless analytics, it lowers TCO (Total Cost of Ownership) by up to 52%, and it ensures data is secure and compliant.

Read this informative post on Cloud Cost Optimization for Better ROI.

Google Cloud Features

Improved Data Management

BigQuery, the serverless data warehouse from Google Cloud Platform (GCP), makes managing, provisioning, and dimensioning infrastructure easier. This frees up resources to focus on the quality of decision-making, operations, products, and services.

Improved Scalability

Storage and computing are decoupled in BigQuery, which improves availability and scalability, and makes it cost-efficient.

Analytics and BI

GCP also improves website analytics by integrating with other GCP and Google products. This helps businesses get a better understanding of the customer’s behavior and journey. The BI Engine packaged with BigQuery provides users with several data visualization tools, speeds up responses to queries, simplifies architecture, and enables smart tuning.

Data Lakes and Data Marts

GCP’s enables ingestion of batch and stream/real-time data, change data capture, landing zone, and raw data to meet other data needs of businesses.

Data Pipelines

GCP tools such as Dataflow, Dataform, BigQuery Engine, Dataproc, DataFusion, and Dataprep help create and manage even complex data pipelines.

Discover how Indium assisted a manufacturing company with data migration and ERP data pipeline automation using Pyspark.

Data Orchestration

For data orchestration too, GCP’s managed or serverless tools minimize infrastructure, configuration, and operational overheads. Workflows is a popular tool for simple workloads while Cloud Composer can be used for more complex workloads.

Data Governance

Google enables data governance, security, and compliance with tools such as Data Catalog, that facilitates data discoverability, metadata management, and data class-level controls. This helps separate sensitive and other data within containers. Data Loss Prevention and Identity Access Management are some of the other trusted tools.

Data Visualization

Google Cloud Platform provides two fully managed tools for data visualization, Data Studio and Looker. Data Studio is free and transforms data into easy-to-read and share, informative, and customizable dashboards and reports. Looker is flexible and scalable and can handle large data and query volumes.

ML/AI

Google Cloud Platform leverages Google’s expertise in ML/AI and provides Managed APIs, BigQuery ML, and Vertex AI. Managed APIs enable solving common ML problems without having to train a new model or even having technical skills. Using BigQuery, models can be built and deployed based on SQL language. Vertex AI, as already seen, enables the management of the ML product lifecycle.

Indium to Modernize Your Data Platform With GCP

Indium Software is a recognized data and cloud solution provider with cross domain expertise and experience. Our range of services includes data and app modernization, data analytics, and digital transformation across the various cloud platforms such as Amazon Web Server, Azure, Google Cloud. We work closely with our customers to understand their modernization needs and align them with business goals to improve the outcomes for faster growth, better insights, and enhanced operational efficiency.

FAQs

The post Data Modernization with Google Cloud appeared first on Indium.

Today, business leaders are able to seamlessly integrate facts and intuition, to drive strategic and operational decisions.

The overall market size for big data analytics is expected to grow from USD 138.9 billion currently to USD 229.4 billion by 2025 at a Compound Annual Growth Rate (CAGR) of 10.6%, according to MarketsandMarkets.

But CXOs – across sectors – are realizing that the role of the CTO and CIO in designing optimal big data engineering and architecture is becoming increasingly important. It is no longer only about access and availability of data. The key is to design a big data workflow that has both depth and breadth, ensuring real-time insights are captured.

In this blog, we focus on one aspect of big data engineering, which is data lake architecture.

How to Use Data Better to Drive Analytics?

Businesses typically use data warehouses to run queries and generate reports and dashboards to capture trends, patterns, and insights.

A data warehouse is an optimized database storing data that has been cleaned, enriched, and transformed, providing a unified view of enterprise-wide data. It has a clearly defined schema and data structure for structured data that have been extracted from different lines of business or transactional systems. It is particularly useful for operational reporting and analysis that is SQL-driven.

But for businesses, limiting their analytics to structured-only data is an opportunity lost. Unstructured data too carries insights and provides a more accurate picture when combined with structured data. What businesses really need is a data lake that is a storehouse of both structured and unstructured data and therefore provides a wider view with deeper insights.

Data Lake for Depth and Breadth

A data lake is like a superset with structured and unstructured data ingested from a variety of sources such as IoT devices, mobile apps, social media in addition to business applications. Due to the absence of a schema during data capture, it does not have a design or any specific purpose. Therefore, it can be used for a variety of analytics such as big data, search, log, real-time, machine learning, and so on.

For it to be meaningful, data lakes need the right storage, architecture, data governance, and security model.

• Integrate Disparate Data – If properly architected, data lakes enable the collection and retention of all types of data, which may include videos, images, binary files, streaming data and more.
• Unlimited Data Import – Being unstructured, you can import any volume of data in real-time into a data lake from different sources and in different formats. This enables quick scaling up too.
• Secure Storage and Cataloging – You can store relational data from sources such as the operational databases as well as data from line of business applications. You can also store non-relational data such as the one from IoT devices, mobile apps, and social media. It lets you crawl, catalog, and index data for better understanding of the data as well as make it secure as per your data governance policies. It is scalable to enable you to handle an increase in the volume of the Big Data.
• Unhindered Analytics – Business analysts, data scientists, and data developers can analyze the data without having to move it to a different analytics system and use any tool and or framework of their choice, be it open-source frameworks such as Apache Hadoop, Apache Spark, and Presto; or commercial tools.
• Powering Data Science and Machine Learning: Data lakes help transform raw data into structured data that is ready for data science, SQL analytics and, also, machine learning with low latency. What’s more, raw data can be retained at low cost for use in analytics and machine learning.

Architecture Matters

At Indium Software, a specialized data engineering service provider, we believe that the right architecture is essential to derive value from your data lake.

In our reckoning, the best fit data lake for your data analytics needs would be one that:

• Ensures data richness through storing all kinds of structured and unstructured data from a variety of sources and in multiple formats such as XML, JSON, text, image, audio, video, etc.
• Enables the conversion of unstructured data to structured data for easy use
• Is secure
• Facilitates the use of open source tools to lower costs and allow scalability
• Integrates data strategy to protect existing investments by enabling existing data warehouses to work together
• Is expandable and allows for a variety of use cases for greater and deeper insights using SQL, NoSQL, Excel etc.

We work with analytical tools based on the customer needs including:

• Azure Data Lake Analytics from Microsoft, a distributed, YARN-based cloud data processing architecture with batch processing capabilities
• AWS Cloud-based Analytics offering an integrated suite of services for the quick and secure  building and managing of data lake for analytics with self-service capabilities
• Apache Spark’s Delta Lake functionality, an open-source storage layer that runs on top of an existing data lake, ensures high data integrity with ACID transactions (Atomicity, Consistency, Isolation and Durability) and uses SQL queries on real-time data

Specific Use Cases for Data Lake Architecture

Customer Relationship Management

A Data Lake can integrate with the data from the organizational CRM platform as well as social media analytics to gain a deeper understanding of user preferences and behaviour.

Improved Innovation

Research and development teams can understand the impact of their hypothesis and fine-tune assumptions to improve outcomes by capturing insights from unstructured data

Increase Operational Efficiency

An optimally engineered data lake architecture is critical to garner insights from data generated from IoT Devices, NLP-based models, etc. Overall, it is critical to plan for a data lake, especially in scenarios where unstructured data can make a key difference in your decision-making process.

Indium Software, with more than two decades of experience in cutting edge technologies, has the right team and the experience to be able to study the needs of our customers and design the right architecture for garnering meaningful insights. If you would like to leverage our strengths for your benefit, please contact us here: https://www.indiumsoftware.com/inquire-now/

The post Best Fit Data Lake Architecture for Optimum Analytics appeared first on Indium.

However, not all companies today have adopted the cloud data warehouse approach. This article aims to help answer why you should move your data warehouse to the cloud. We will be covering two main topics in this article:

Question 1 – A data warehouse in the cloud – Do you need it?

Most definitely, yes! Just see how quickly your data warehouse is growing or even look at the number of project requests that are increasing for new data warehouses, new data lakes, new discovery sandboxes, faster query times, and more. Every IT department looks for a silver lining to meet the growing demands of business access that are routed through their business units. That silver lining is the cloud!

Question 2: What is expected from a cloud data warehouse, and how can you benefit from it? The list goes on, but I have narrowed it down to 6 key benefits:

Fast and Easy Deployment:

IT teams in the past, had to take into account how much storage and compute power they would require, at times even 2 years in advance. In the event of getting this information wrong, it would mean that extra hardware that was not needed was purchased or facing complaints regarding a dearth of storage.

However, this rather complicated and exhaustive planning & estimation process is not required in today’s day and age. By leveraging the cloud, users can build their own sandboxes, data warehouses or datamarts in minutes at any time of the day. One key advantage of having the data warehouse on cloud is that organizations only for the required resources when they are required.

Much needed Elasticity at significantly Lower Costs:

The major reason as to why people move to a data warehouse in the cloud is cost. A really expensive task is storing data in your own data center on premise. While expanding your data footprint, it becomes very tough to support your continuous analytical needs.

The immediate question that arises is Why? In the case of an on-premise data warehouse, you can’t quickly or easily scale, compute and store. In the event of you needing more storage, the compute will come along with it and the costs for both with have to be born.

Additionally, the need to purchase the requisite compute at peak times will also arise. Take the case of a retail company that needs to predict how much compute would be required to handle a Black Friday sale, they are stuck with the same compute for the rest of the year.

With a cloud data warehouse, that doesn’t need to be the case.

With the ideal cloud data warehouse in place, your system will be able to scale instantly and flexibly to deliver the required amount of compute that is necessary. As compute and storage are separate, only the essentialities will need to be purchased. The add on costs such as server rooms, hardware, networking , etc is also eliminated.

Data Security Concerns:

On-premise data warehouses were always considered to be the more secure option. Just like how the trust in digital copies increased when compared to physical paper copies, people are now starting to see as to why cloud data warehouses are more secure than on-premise data warehouses.

However, this as well entirely depends on the database possessed by the company.

Capabilities growing tremendously:

The overall value of your data warehouse tremendously improves when your data warehouse in the cloud. With better availability, performance and scalability, business intelligence and other applications can seamlessly deliver quicker and better insights.

The entire spectrum of data warehousing solutions such as data integration, business analytics, IoT and much more with the capability of being a fully integrated solution is possible with your data warehouse being in the cloud.

Self-Service Data Warehousing:

A self-driving data warehouse is what makes self-service entirely possible. An autonomous, self-driving data warehouse its own set of benefits. Management of the data warehouse does not become a worry for you anymore.

This means that you can reap the benefits of complete automation of upgrades, patching, and management. Using a data warehouse in the cloud means all you need to do is log into the cloud and allocate a new data warehouse in minutes – you do not need to depend on IT for this like before.

Availability and accessibility of data is more than ever.

IT teams now have the liberty to concentrate their resources and attention on providing business value from a strategic stand-point. This does not imply that database analysts (DBAs) are out of a job. Managing of applications connecting to a data warehouse and how developers use in database functions within their application code will still be done by DBAs. 

The Cloud in itself:

Database that is self-driving makes life a lot easier because it handles the monotonous yet essential work that executives do not want to do. The much-needed ability and capability in the cloud can be gained with the self-serving database.

Cloud data warehouse adoption may just be one step out of a multi-step journey for many organizations. The cloud provider must ensure complete PaaS, IaaS & SaaS solutions.

IT infrastructure can be simplified, and capital investments can be minimized by leveraging your cloud’s services for data management, business intelligence, applications, and infrastructure.

Whilst choosing a cloud, it is imperative that you pick one that allows for flexible deployment models. This will enable you to migrate on-premise workloads to your cloud data centers and vice-versa seamlessly.

Conclusion

Having a data warehouse in the cloud has plenty of benefits. But going beyond that is what we need to strive for. Assess how a self-driving data warehouse can be of help and make maximum use of it.

The post 6 Main Benefits of having a Cloud data Warehouse in Place! appeared first on Indium.

A repository for unstructured, structured and semi-structured data. These lakes permit data to rest in their most natural form without having to be transformed and analysed initially. In this aspect they are very different from data warehouses.

In more understandable terms, the different types of data generated by machines and humans can be loaded into a data lake for analysis and classification at a later time.

Properly structured data is required in a data warehouse before any work can be done on the data.

To understand properly as to why data lakes are the ideal candidates to house big data, it is very crucial to understand why how they are different from data warehouses.

The Difference Between a Data Warehouse and a Data Lake:

Probably the only similarity between a data warehouse and a data lake solution is the fact that they are both data repositories. Let’s now have a look at some of the key differences:

• In most cases, data warehouses make use of highly structured data whereas data lakes are designed in such a way that they support all types of data.
• All the data that may be analyzed at a future date is stored in data lakes. Due to limited storage being an issue, irrelevant data is eliminated in a data warehouse.
• With reference to the above discussed points, it is evident that the scale between a data warehouse and a data lake is vastly different. A data lake needs to highly scalable because it supports all types of data and stores it even if it’s not for immediate use.
• Metadata (data about data) being available allows users who work with data lakes to gain basic insights about the data really quickly. In the case of data warehouses, a member of the development team is required to access the data which in turn can create a bottleneck.
• Another key difference is that the intense data management required for data warehouses implies that they’re very expensive to maintain when compared to data lakes.

The Use of Data Lakes!

The advantage with data lakes is that advanced analytics tools and mining software take the raw data and turn them into useful insights. Structured and clean data is what data warehouses depend on, whereas data lakes let data rest in its raw and natural form.

Now that you know the importance of Data lakes, let’s look how most of the businesses implement Big Data which helps to increase their revenue.

Big Data Analytics

In order to uncover patterns, customer preferences and market trends with the objective to help business make informed decisions faster, big data analytics makes use of the data in a data lake. This is achieved through 4 different types of analysis:

Descriptive Analysis

Retrospection is the nature of descriptive analysis. A look at “where” the problem may have occurred. Big data analytics today is actually descriptive in nature because analytics can be generated quickly.

Diagnostic Analysis

Retrospective in nature again. Diagnostic analysis looks at “why” the specific problem occurred in the first place. This is more detailed than descriptive analytics.

Predictive Analysis

Analysis can provide an organization with models which are predictive in nature of when an event might occur next when AI and machine learning models are applied. Predictive analytics models are now widely adopted because of the amazing insights they generate.

Prescriptive Analysis

This is the future of big data analytics as it does not only assist with decision making but also provides a set of concrete answers. A high level of machine learning usage is involved in this analysis.

Architecture Of The Data Lake!

The question arises, how can data lakes store such massive and diverse amounts of data? For these massive repositories, what is the underlying architecture?

The data model that data lakes are built on is the schema-on-read model. A schema is essentially like a blueprint – the structure of the database outlining its model and how the data is structured within it.

When you can load your data in the lake without having to worry about structure, it is a schema-on-read data model. This model allows for a lot more flexibility.

On the other hand, data warehouses comprise of schema-on- write data models. This is a rather traditional method adopted for databases.

All sets of data with their relationship and index must be clearly pre-defined. This in turn, limits flexibility, especially when new data sets are added, or new features are added which may potentially create gaps in the database.

The backbone of a data lake is the schema-on-read data model. However, the processing framework is how the data actually is loaded into one.

The processing frameworks that ingest data into data lakes are explained below:

Stream Processing

Small batches of data processed in real-time. For businesses that harness real-time analytics stream processing is the very valuable.

Batch Processing

Processing many million blocks of data over long periods of time. In order to process big data, this is the least time sensitive method.

Stream vs Batch Processing

Apache Spark, Apache Storm and Hadoop are some of the commonly used big data processing tools which are capable of Stream and Batch processing.

Processing of unstructured data such as internet clickstream data, social media posts, sensor activity etc can be done only by a certain set of tools. Other tools on the market make use of machine learning programs to prioritize processing of speed and usefulness.

After data processing, once it is ingested in the data lake, it is time to make use of it.

Data lake challenges

Advantages of a data lake are that they are scalable, quick to load and flexible. However, they come at a cost.

• Unstructured data ingestion requires a lack of data governance and processes to make sure the right data is being looked at. For many businesses – especially those that haven’t adopted big data – possessing uncleaned and unorganized data isn’t an option.
• You could end up in a data swamp if the metadata or processes are misused to keep the data lake in check.
• Data security is always an issue which needs to be kept in check.
• Data lakes are being widely used in IT today. A few tools are still working out the security kinks. One major kink is to ensure that only the right people have access to the data which is sensitive in nature which is loaded into the lake.

These issues get resolved with time with any new technology. But like any new technology, these issues will resolve with time.

Big Data – The Role Data Lakes Play!

Even though data lakes have a few challenges, it is no secret that 80 percent of the data is unstructured in the world. As more businesses start adopting big data, the applications of data lakes are bound to rise.

Looking for an organized data transmission solution? Inquire Now about our Data Lake Services.

Data warehouses are strong in security and structure, but big data needs to be unconfined so that it can flow into data lakes freely.

The post The Importance of Data Lakes and What they Mean to Big Data! appeared first on Indium.

This is where data-driven decision making also faces a problem. The problem arises where collation of all the various sources of data in one repository is required.

This is because all the data sources, systems and formats are disparate. What this means is that the need to organize all of this data in one repository for analysis is extremely important. This is where the data warehouse comes to our aid.

The post How to Select the Best Data Warehouse For Your Needs (Infographic) appeared first on Indium.

Let’s start with what a data warehouse is – Integrated historical & current data in a central repository! This data repository is derived from external data sources and operational systems.

A data warehouse, being a central component of business intelligence allows enterprises to cover a rather wide range of business decisions.

These decisions may include – business expansion, production method improvements, product pricing so on and so forth.

Apart from the huge role that a data warehouse plays in analysis and reporting, a data warehouse provides the following benefits to an organization:

• It allows you to keep data analysis separate from production systems. Complex analytical queries cannot be run by operational databases used by organizations every day. This is where a data warehouse lets the organization to run such queries without there being any ramifications on the production systems.
• Data warehouses bring consistency to disparate data sources.
• Data warehouses have an optimized design for analytical queries.

The popularity of Data Warehouse-as-a-service has increased tremendously over the past five years. This is primarily because of the impact that cloud computing has had on big data architecture. Let’s now have a look at the major differences between cloud-based data warehouses and traditional data warehouses.

A Traditional Data Warehouse

The traditional on-premise data warehouse requires on-premise IT resources like software and servers for the functions of the data warehouse to be delivered. Infrastructure needs to be maintained effectively when organizations run their own on-premise data warehouse.

The 3 tier structure of a traditional data warehouse:

• The data warehouse server is what occupies the bottom tier. This contains data pulled from various sources and is integrated in a sole repository.
• The OLAP servers occupy the middle tier. This allows the data to be more accessible for the different types of queries that will be used on it.
• The front-end BI tools occupy the top layer. These tools are primarily used for querying, reporting and analytics.

In order to pull data into the data warehouse, ETL tools are usually used. These tools obtain data from various sources, process it and apply the relevant business rules to get the data into the right format based on the data model.

After this, the data is finally loaded into the data warehouse.

Bill Inmon and Ralph Kimball, two computer science pioneers have contrasting opinions when it comes to traditional warehouse design –

Bill Inmon suggested a top-down approach which meant that all enterprise data is stored in the data warehouse which is the central repository.

From this data warehouse, dimensional data marts which serve particular lines of business are created.

On the other hand, the bottom-up approach according to Ralph Kimball suggests that the result of the combining data marts is the data warehouse.

Cloud Data Warehouse

The concept of the cloud-based data warehouse approach stems from leveraging data warehouse services provided by the public cloud providers like Google BigQuery or Amazon Redshift or Azure SQL DW.

With data warehousing services accessible over the internet, public cloud providers allow companies to cut down heavily on their initial set up costs required for a traditional on-premise data warehouse.

Adding to that, the cloud data warehouses are fully managed by the providers. Hence, the service providers manage and assume entire responsibility of the required data warehouse functionalities. This includes updates and patches to the system.

In comparison – Traditional vs Cloud

The traditional data warehouse approaches differ from the cloud architectures. Cloud Architectures are somewhat different from traditional Data Warehouse approaches.

Take the case of Amazon Redshift – The operations of Redshift are designed where you are required to provision a cluster of cloud-based computing nodes.

A few of these nodes compile queries whereas a few of them execute these queries. Google on the other hand provides a serverless service.

This means that the allocation of machine resources is managed by Google dynamically.

These decisions are taken by Google thereby freeing up the user’s bandwidth. In the case of Azure, it is a solution that is relatively cheaper with an ability to scale and compute storage.

In Azure, you have the advantage of pausing and resuming your databases in minutes.

When it comes to cloud data warehouse, the level of optimization it offers is very tough to match by the traditional on-premise setup.

Another advantage of cloud over on-premise is columnar storage. This is when table values are stored by column and not rows.

This allows for faster aggregate queries in line with the type of queries you need to run in a data warehouse. Another feature that drastically improves query speeds is massively parallel processing.

This is done by using many machines to coordinate query processing for large datasets.

When it comes to scalability, in the cloud, it is just as simple as provisioning additional resources from the cloud provider.

On-premise scalability is expensive and time consuming as the need to purchase more hardware arises.

The tricky aspect of having a cloud data warehouse is security – transmitting terabytes of data over the internet brings about a security concern which includes compliance concerns as well.

This is because the data may carry sensitive information. An on-premise setup holds the edge here as these security concerns are totally avoided because the organization controls everything.

Summing it up

For medium and small-sized companies, the cloud makes data warehousing more accessible than before due to the low barriers to entry.

Cloud data warehouses entice even the biggest enterprise due to their lower costs – reduction in infrastructure management costs and easy scalability.

Putting things in perspective, the cloud does have its issues when it comes to security. However, the benefits clearly outweigh the negatives. Legacy on-premise setups are not entirely obsolete.

However, the volume and velocity of data is growing at the rate of knots today and cloud services are designed to handle this sort of data.

As it stands today, more and more workload is moving to the cloud and more and more companies have started providing cloud-based data warehousing services.

This trend tells us that the cloud is the future of data warehousing!

The post Data Warehousing – Traditional vs Cloud! appeared first on Indium.