• Home  >  
  • Perspectives  >  
  • A Comprehensive Guide: Optimizing Azure Databricks Operations with Unity Catalog  
Blog June 21, 2023
5 min read

A Comprehensive Guide: Optimizing Azure Databricks Operations with Unity Catalog

Learn how Unity Catalog in Azure Databricks simplifies data management, enabling centralized metadata control, streamlined access management, and enhanced data governance for optimized operations.

For a Data engineer/Admins, making sure that all operations run smoothly is a priority.

That’s where Unity Catalog can help ensure that stored information is managed correctly, especially for those working with Azure Databricks. The Unity Catalog (UC) is a powerful metadata management system that is built into Delta Lake. It provides a centralized location to help users manage the metadata information on the data stored in Delta Lake. It also helps to simplify data management by providing a unified view of data across different data sources and formats.

Before Unity Catalog, every ADB (Azure Databricks) workspace had its own metastore, user management, and access controls, which led to duplication of efforts when maintaining consistency across all workspaces. To overcome these challenges, Databricks developed Unity Catalog, a unified governance solution for data and AI assets on the Lakehouse. Unity Catalog provides centralized access control, auditing, lineage, and data discovery capabilities across Azure Databricks workspaces.

At Tiger Analytics, we’ve worked to enable Unity Catalog for clients with new Databricks deployment and upgrading existing hive metastore to Unity Catalog to leverage all benefits Unity Catalog provides.

Databricks Unity Catalog

Making the Most of Unity Catalog’s Key Features

Centralized metadata and user management

Unity Catalog provides a centralized metadata layer to enable sharing data objects such as catalogs/ schema/ tables across multiple workspaces. It introduces two new built-in admin roles (Account Admins and Metastore Admins) to manage key features. 

  • Account Admin: manages account-level resources like metastore, assigns metastore to workspaces, and assigns principals to the workspace.
  • Metastore Admin: manages metastore objects and grants identities access to securable objects (catalog/ schema/ tables/ views).

Centralized data access controls
Unity Catalog permits the use of Standard SQL-based commands to provide access to data objects.

GRANT USE CATALOG ON CATALOG < catalog_name > TO < group_name >;

GRANT USE SCHEMA ON SCHEMA < catalog_name >.< schema_name >

TO < group_name >;

GRANT

SELECT

ON < catalog_name >.< schema_name >.< table_name >

TO < group_name >;

Data lineage Data access auditing

Unity Catalog automatically captures user-level audit logs that record access to user data. It also captures lineage data that tracks how data assets are created and used across all languages and personas.

Data search and discovery

Unity Catalog lets you tag and document data assets and provides a search interface to help data consumers find data.

Delta Sharing

Unity Catalog allows users in Databricks to share data securely outside the organization, which can be managed, governed, audited, and tracked.

Unity Catalog Delta Sharing

Unity Catalog Metastore

Managing Users and Access Control

  • Account admins can sync users (groups) to workspaces from Azure Active Directory (Azure AD) tenant to Azure Databricks account using a SCIM provisioning connector.
  • Azure Databricks recommends using account-level SCIM provisioning to create, update, and delete all users (groups) from the account.

Unity Catalog Managing Users and Access Control

Unity Catalog Objects

Unity Catalog Objects

Metastore 

A metastore is the top-level container of objects in the Unity Catalog. It stores data assets (tables and views) and the permissions that govern access. UC metastore is mapped to an ADLS container, this container stores the Unity Catalog metastore’s metadata and managed tables. You can only create one UC metastore per region. Each workspace can only be attached to one UC metastore at any point in time. Unity Catalog has a 3-tier structure (catalog.schema.table/view) for referencing objects.

External Location and Storage Credential

  • Storage credential created either as managed identity or service principal provides access to the underlying ADLS path.
  • Storage credentials (managed identity/ service principal) should be authorized to that external storage account location by providing IAM role at storage account level.
  • External Location is an object that combines a cloud storage path with storage credentials to authorize access to the cloud storage path.
  • Each cloud storage path can be associated with only one external location. If you attempt to create a second external location that references the same path, the command fails.

External Location and Storage Credential

Managed and External Tables

  • Unity Catalog manages the lifecycle of managed tables. This means that if you drop managed tables, both metadata and data are dropped.
  • By default, UC metastore ADLS container (the root storage location) will store the managed tables’ data as well, but you can override this default location at the catalog or schema level. Managed tables are in Delta format only.
  • External tables are tables whose data is stored outside of the managed storage location specified for the metastore, catalog, or schema. Dropping them will only delete the metadata of the table.

How to Create a UC Metastore and Link Workspaces

This flow diagram explains the sub-tasks needed to create a Metastore.

How to Create a UC Metastore and Link Workspaces

Step 1: Create an ADLS storage account and container

This Storage account container will store Unity Catalog metastore’s metadata and managed tables.

Step 2: Create an access connector for Databricks 

Create Access Connector for Azure Databricks, and when deployment is done, make a note of Resource ID.

Step 3: Provide RBAC to access the connector

Add role assignment: Storage Blob Data Contributor to the managed identity (access connector) in Step#2

Step 4: Create metastore and assign workspaces

Unity Catalog Metastore

Once a UC metastore has been attached to a workspace, this will be visible  under the workspace data tab:

Unity Catalog Workspace

If Unity Catalog is enabled for any existing workspace which had tables stored under hive_metastore catalog, those existing tables can be upgraded using SYNC command or UI, or they can be accessed using hive_metastore.<schema_name>.<table_name> 

Enabling Unity Catalog as part of lakehouse architecture helps in achieving a centralized metadata layer for more enterprise-level governance without sacrificing the ability to manage and share data effectively. It helps in planning workspace deployments with limits in mind. This helps eliminate the risks of not being able to share the data and govern the project. With Unity Catalog, we can overcome the limitations and constraints of the existing Hive metastore, enabling us to better collaborate and leverage the power of data according to specific business needs.

Shivam Banga

Explore more blogs

4 min read
August 10, 2022
How to integrate Tableau and Python and why it's necessary
Readshp-arrow-topright-large
6 min read
August 29, 2022
How to Design your own Data Lake Framework in AWS
Readshp-arrow-topright-large
4 min read
September 15, 2022
Harnessing own Retail’s Ring of Power with GCP
Readshp-arrow-topright-large
5 min read
November 17, 2022
Enabling Fivetran Transformations for dbt - a step-by-step approach
Readshp-arrow-topright-large
Copyright © 2024 Tiger Analytics | All Rights Reserved