What Cloud Engineers Need to Know about Databricks Architecture and Workflows

Databricks Lakehouse Platform creates a unified approach to the modern data stack by combining the best of data lakes and data warehouses with greater reliability, governance, and improved performance of data warehouses. It is also open and flexible.

Often, the data team needs different solutions to process unstructured data, enable business intelligence, and build machine learning models. But with the unified Databricks Lakehouse Platform, all these are unified. It also simplifies data processing, analysis, storage, governance, and serving, enabling data engineers, analysts, and data scientists to collaborate effectively.

For the cloud engineer, this is good news. Managing permissions, networking, and security becomes easier as they only have one platform to manage and monitor the security groups and identity and access management (IAM) permissions.

Challenges Faced by Cloud Engineers

Access to data, reliability, and quality, are key for businesses to be able to leverage the data and make instant and informed decisions. Often, though, businesses face the challenge of:

• No ACID transactions: As a result, updates, appends, and reads cannot be mixed
• No Schema Enforcement: Leads to data inconsistency and low quality.
• Integration with Data Catalog Not Possible: Absence of single source of truth and dark data.

Since object storage is used by data lakes, data is stored in immutable files that can lead to:

• Poor Partitioning: Ineffective partitioning leads to long development hours for improving read/write performance and the possibility of human errors.

• Challenges to Appending Data: As transactions are not supported, new data can be appended only by adding small files, which can lead to poor quality of query performance.

Databricks Advantages

Databricks helps overcome these problems with Delta Lake and Photon.

Delta Lake: A file-based, open-source storage format that runs on top of existing data lakes, it is compatible with Apache Spark and other processing engines and facilitates ACID transactions and handling of scalable metadata, unifying streaming and batch processing.

Delta Tables, based on Apache Parquet, is used by many organizations and is therefore interchangeable with other Parquet tables. Semi-structured and unstructured data can also be processed by Delta Tables, which makes data management easy by allowing versioning, reliability, time travel, and metadata management.

It ensures:

• ACID
• Handling of scalable data and metadata
• Audit history and time travel
• Enforcement and evolution of schema
• Supporting deletes, updates, and merges
• Unification of streaming and batch

Photon: The lakehouse paradigm is becoming de facto but creating the challenge of the underlying query execution engine unable to access and process structured and unstructured data. What is needed is an execution engine that has the performance of a data warehouse and is scalable like the data lakes.

Photon, the next-generation query engine on the Databricks Lakehouse Platform, fills this need. As it is compatible with Spark APIs, it provides a generic execution framework enabling efficient data processing. It lowers infrastructure costs while accelerating all use cases, including data ingestion, ETL, streaming, data science, and interactive queries. As it does not need code change or lock-in, just turn it on to get started.

Read more on how Indium can help you: Building Reliable Data Pipelines Using DataBricks’ Delta Live Tables

Databricks Architecture

The Databricks architecture facilitates cross-functional teams to collaborate securely by offering two main components: the control plane and the data plane. As a result, the data teams can run their processes on the data plane without worrying about the backend services, which are managed by the control plane component.

The control plane consists of backend services such as notebook commands and workspace-related configurations. These are encrypted at rest. The compute resources for notebooks, jobs, and classic SQL data warehouses reside on the data plane and are activated within the cloud environment.

For the cloud engineer, this architecture provides the following benefits:

Eliminate Data Silos

A unified approach eliminates the data silos and simplifies the modern data stack for a variety of uses. Being built on open source and open standards, it is flexible. Enabling a unified approach to data management, security, and governance improves efficiency and faster innovation.

Easy Adoption for A Variety of Use Cases

The only limit to using the Databricks architecture for different requirements of the team is whether the cluster in the private subnet has permission to access the destination. One way to enable it is using VPC peering between the VPCs or potentially using a transit gateway between the accounts.

Flexible Deployment

Databricks workspace deployment typically comes with two parts:

– The mandatory AWS resources

– The API that enables registering those resources in the control plane of Databricks

This empowers the cloud engineering team to deploy the AWS resources in a manner best suited to the business goals of the organization. The APIs facilitate access to the resources as needed.

Cloud Monitoring

The Databricks architecture also enables the extensive monitoring of the cloud resources. This helps cloud engineers track spending and network traffic from EC2 instances, register wrong API calls, monitor cloud performance, and maintain the integrity of the cloud environment. It also allows the use of popular tools such as Datadog and Amazon Cloudwatch for data monitoring.

Best Practices for Improved Databricks Management

Cloud engineers must plan the workspace layout well to optimize the use of the Lakehouse and enable scalability and manageability. Some of the best practices to improve performance include:

• Minimizing the number of top-level accounts and creating a workspace as needed to be compliant, enable isolation, or due to geographical constraints.
• The isolation strategy should ensure flexibility without being complex.
• Automate the cloud processes.
• Improve governance by creating a COE team.

Indium Software, a leading software solutions provider, can facilitate the implementation and management of Databricks Architecture in your organization based on your unique business needs. Our team has experience and expertise in Databricks technology as well as industry experience to customize solutions based on industry best practices.

FAQ