Designing Enterprise Data Governance with Unity Catalog

Unity Catalog makes access control, lineage and audit a property of the platform rather than of each workspace. That only pays off if the design underneath survives growth: catalogs that follow how the organisation is actually divided, isolation that does not depend on everyone granting correctly, and a privilege model built on groups and inheritance rather than thousands of individual grants.

This article sets out such a design. It covers the object model and how to map it to environments and domains, isolation through workspace binding and storage, identity and a privilege model that scales, row filters, column masks and attribute-based access control, lineage and audit through system tables, grants as code, and a migration path from the legacy Hive metastore. Examples use generic names throughout.

The object model

A metastore is the top-level container, one per region, shared by every workspace in that region that is attached to it. Below it, data and AI assets use a three-level namespace, catalog.schema.object, where objects are tables, views, volumes, functions and registered models. Tables and volumes are either managed, with Unity Catalog owning governance and the storage lifecycle, or external, with Unity Catalog owning governance only. Storage credentials, external locations, connections for Lakehouse Federation, and shares and recipients for OpenSharing (formerly Delta Sharing) sit directly under the metastore.

Two consequences shape everything else. The metastore gives regional isolation, but Databricks does not intend it as the default unit of data isolation; that is the catalog’s job. And because one metastore spans a region’s workspaces, separation between environments has to be designed through catalogs, bindings and grants rather than assumed from separate workspaces.

Mapping catalogs to environments and domains

The catalog layout is the decision that is hardest to change later. Three layouts are common:

• Catalog per environment (dev, test, prod) with schemas per domain. Few catalogs and simple bindings, but domain ownership exists only at schema level and every domain shares one catalog owner.
• Catalog per environment and domain (sales_dev, sales_prod, finance_prod). Each domain owns its catalogs, storage and grants, and each environment binds cleanly to its workspaces. More catalogs, so automation becomes mandatory.
• Catalog per business unit with environments inside (sales.orders_dev). Avoid it: development and production share one isolation boundary that bindings cannot split.

Medallion layers usually work best as schemas inside a domain catalog (sales_prod.raw, sales_prod.curated). Keep the environment out of table names and pass the catalog as a parameter, so the same code deploys everywhere. And note one default: all users have USE CATALOG on the main catalog. That suits a sandbox, not governed production data.

Isolation: workspace binding and storage

By default, every catalog is reachable from every workspace attached to the metastore. Workspace-catalog binding restricts a catalog to named workspaces, optionally read-only, and access from an unbound workspace is denied even for a user who holds grants. Grants decide who; bindings decide from where. Bind production catalogs to the production workspaces, where network controls, compute policies and service principals are configured for production, and give analytics workspaces read-only bindings to what they consume. External locations, storage credentials and service credentials can be bound the same way.

Storage follows the same boundaries. A storage credential wraps a cloud identity, such as an IAM role on AWS; an external location pairs a storage path with a credential. Managed storage can be set at metastore, catalog or schema level, and data lands at the lowest level that has one. Databricks recommends catalog-level managed storage as the main unit of isolation; metastore-level storage is no longer required. Never reuse a bucket that is or was a DBFS root, and do not let anything outside Unity Catalog read managed storage directly, because that bypasses access control and audit.

Treat external locations as infrastructure: few, broad, and created by administrators. Do not grant READ FILES or WRITE FILES to end users. Register volumes or external tables at sub-paths instead, never at the root of an external location, and use volumes for path-based file access.

Identity: account-level groups

Unity Catalog grants privileges only to principals defined at the account level. Provision users, groups and service principals from your identity provider into the account, through automatic identity management where supported and account-level SCIM otherwise, and turn off workspace-level SCIM. Workspace-local groups cannot be used in Unity Catalog grants, so recreate them as account groups, ideally in the identity provider, before you migrate anything.

Design groups around access:

• readers and engineers per domain and environment, such as sales-prod-readers;
• an owner group per domain that holds ownership and MANAGE, such as sales-platform-admins;
• steward groups that classify data, such as sales-data-stewards;
• writer groups containing the service principals that run production jobs, such as sales-prod-writers.

Run production pipelines as service principals and reserve direct MODIFY on production tables for them; a person with write access to production can overwrite it by accident. In filters, masks and dynamic views, test membership with is_account_group_member() rather than is_member(), which evaluates workspace-level group membership.

A privilege model that scales

Privileges inherit downward: a grant on a catalog or schema applies to every current and future object inside it. Reading a table requires USE CATALOG on its catalog, USE SCHEMA on its schema and SELECT on the table, directly or inherited. The usage privileges are also a control point: only catalog and schema owners, or holders of MANAGE, can grant them, so a table owner cannot share a table with people the schema owner has not admitted.

Build on those properties. Grant to groups, at the highest level that is correct, which is usually the schema. Grant BROWSE on catalogs to all account users: it exposes metadata, not data, and lets people find assets and request access, provided you configure access request destinations. Make groups the owners of production catalogs and schemas; whoever runs CREATE becomes the first owner, so reassign ownership as part of creation. Delegate administration with MANAGE rather than ownership, and use ALL PRIVILEGES sparingly; it excludes MANAGE, READ METADATA, EXTERNAL USE SCHEMA and EXTERNAL USE LOCATION.

-- Groups own production objects, not the engineer who ran CREATE.
ALTER CATALOG sales_prod OWNER TO `sales-platform-admins`;
ALTER SCHEMA sales_prod.curated OWNER TO `sales-platform-admins`;

-- Discovery for everyone: metadata only, no data.
GRANT BROWSE ON CATALOG sales_prod TO `account users`;

-- Readers see the curated schema. USE CATALOG alone exposes nothing.
GRANT USE CATALOG ON CATALOG sales_prod TO `sales-prod-readers`;
GRANT USE SCHEMA, SELECT ON SCHEMA sales_prod.curated TO `sales-prod-readers`;

-- Production writes come from a group of pipeline service principals.
GRANT USE CATALOG ON CATALOG sales_prod TO `sales-prod-writers`;
GRANT USE SCHEMA, SELECT, MODIFY, CREATE TABLE ON SCHEMA sales_prod.curated
  TO `sales-prod-writers`;

-- Stewards tag data; APPLY TAG does not let them read it.
GRANT USE CATALOG, USE SCHEMA, APPLY TAG ON CATALOG sales_prod
  TO `sales-data-stewards`;

SHOW GRANTS ON SCHEMA sales_prod.curated;

Finer write privileges (INSERT, UPDATE and DELETE as children of MODIFY) are in Beta at the time of writing; check the current documentation before designing around them.

Fine-grained access

Row filters and column masks

A row filter is a SQL UDF that returns a boolean, attached to a table; a column mask is a UDF that returns the value or a masked form of it. A table has at most one row filter. Filters run with the definer’s rights, except identity functions such as is_account_group_member(), which evaluate the querying user, so a mapping table used inside a filter does not need to be readable by everyone.

-- Maintained by the data owner: which group may see which region.
CREATE TABLE IF NOT EXISTS sales_prod.governance.region_access (
  group_name STRING NOT NULL,
  region     STRING NOT NULL
);

CREATE OR REPLACE FUNCTION sales_prod.governance.region_filter(row_region STRING)
RETURNS BOOLEAN
RETURN is_account_group_member('sales-global-readers')
  OR EXISTS (
    SELECT 1 FROM sales_prod.governance.region_access AS a
    WHERE a.region = row_region AND is_account_group_member(a.group_name)
  );

ALTER TABLE sales_prod.curated.orders
  SET ROW FILTER sales_prod.governance.region_filter ON (region);

CREATE OR REPLACE FUNCTION sales_prod.governance.mask_email(email STRING)
RETURNS STRING
RETURN CASE
  WHEN is_account_group_member('sales-pii-readers') THEN email
  ELSE concat('***@', substring_index(email, '@', -1))
END;

ALTER TABLE sales_prod.curated.customers
  ALTER COLUMN email SET MASK sales_prod.governance.mask_email;

Readers need a SQL warehouse, standard access mode on Databricks Runtime 12.2 LTS or above, or dedicated access mode on 15.4 LTS or above. On dedicated compute the filtering runs on serverless compute, which must be enabled for the workspace and may be billed. Drop a filter or mask from its table before dropping the function, or the table becomes inaccessible.

Attribute-based access control

Table-level filters and masks do not scale to hundreds of tables, and table owners can remove them. ABAC policies attach to a catalog, schema or table and select their targets through governed tags, so one policy covers every matching table in scope, including tables created later, and owners below cannot override it. Governed tags are account-level tags with allowed values and permissions on who may assign them. A policy names the principals it applies to with TO and exempts others with EXCEPT.

-- Replaces the table-level mask above: two different masks on one column
-- for the same user make Databricks block the query.
ALTER TABLE sales_prod.curated.customers ALTER COLUMN email DROP MASK;

-- Stewards tag the column; the governed tag `pii` defines the allowed values.
SET TAG ON COLUMN sales_prod.curated.customers.email pii = email;

-- The UDF only masks. Who is affected is the policy's job.
CREATE OR REPLACE FUNCTION sales_prod.governance.redact_email(email STRING)
RETURNS STRING
RETURN concat('***@', substring_index(email, '@', -1));

CREATE POLICY mask_pii_email
ON CATALOG sales_prod
COMMENT 'Mask columns tagged pii = email outside the approved group'
COLUMN MASK sales_prod.governance.redact_email
TO `account users`
EXCEPT `sales-pii-readers`
FOR TABLES
MATCH COLUMNS has_tag_value('pii', 'email') AS email
ON COLUMN email;

Creating and reading ABAC-protected tables requires Databricks Runtime 16.4 or above, or serverless compute; older runtimes cannot read them at all, which matters for any legacy job you plan to leave untouched. Metastore-level policies, policies on views and DENY policies are in Beta at the time of writing, so check the current documentation for release status before committing a design to them. Dynamic views, with current_user() or is_account_group_member() in the definition, remain useful when access logic spans several tables or reshapes data for sharing.

Lineage, audit and system tables

Unity Catalog captures lineage automatically for queries run on Databricks, down to column level, and aggregates it across all workspaces attached to the metastore. The lineage system tables, system.access.table_lineage and system.access.column_lineage, keep a rolling 395-day window by default, while Catalog Explorer retains lineage captured since September 2024. Two limitations affect design: lineage is not preserved when catalogs, schemas, tables, views or columns are renamed, so settle names before production; and path-based references weaken column-level lineage in the UI.

Audit events land in system.access.audit. Unity Catalog actions are logged under the unityCatalog service, permission changes as updatePermissions, and queries should filter on event_date. System table schemas must be enabled before you can query them.

-- Permission changes in the last 30 days.
SELECT
  event_time,
  user_identity.email                   AS changed_by,
  request_params['securable_type']      AS securable_type,
  request_params['securable_full_name'] AS securable,
  request_params['changes']             AS changes
FROM system.access.audit
WHERE service_name = 'unityCatalog'
  AND action_name = 'updatePermissions'
  AND event_date >= date_sub(current_date(), 30)
ORDER BY event_time DESC;

-- What reads from a table, before you change it.
SELECT DISTINCT target_table_full_name, entity_type
FROM system.access.table_lineage
WHERE source_table_full_name = 'sales_prod.curated.orders'
  AND target_table_full_name IS NOT NULL
  AND event_date >= date_sub(current_date(), 90);

For current state rather than history, information_schema views such as table_privileges show grants with an inherited_from column. One caveat at the time of writing: holders of MANAGE see only their own grants there, so use SHOW GRANTS for a complete answer.

Grants as code

Governance managed by hand in the UI drifts. Manage catalogs, schemas, bindings and grants with the Terraform Databricks provider or versioned SQL, reviewed like any other change.

variable "prod_workspace_id" { type = number }
variable "bi_workspace_id" { type = number }

resource "databricks_catalog" "sales_prod" {
  name    = "sales_prod"
  owner   = "sales-platform-admins"
  comment = "Sales domain, production. Managed by Terraform."

  # Must fall inside an external location.
  storage_root = "s3://example-sales-prod/managed"

  # Also binds the catalog to the workspace this provider is connected to.
  isolation_mode = "ISOLATED"
}

resource "databricks_workspace_binding" "sales_prod_prod" {
  securable_name = databricks_catalog.sales_prod.name
  workspace_id   = var.prod_workspace_id
}

resource "databricks_workspace_binding" "sales_prod_bi" {
  securable_name = databricks_catalog.sales_prod.name
  workspace_id   = var.bi_workspace_id
  binding_type   = "BINDING_TYPE_READ_ONLY"
}

resource "databricks_schema" "curated" {
  catalog_name = databricks_catalog.sales_prod.name
  name         = "curated"
  owner        = "sales-platform-admins"
}

# Authoritative: grants on this catalog that are not listed here are revoked.
resource "databricks_grants" "sales_prod" {
  catalog = databricks_catalog.sales_prod.name
  grant {
    principal  = "account users"
    privileges = ["BROWSE"]
  }
  grant {
    principal  = "sales-prod-readers"
    privileges = ["USE_CATALOG"]
  }
  grant {
    principal  = "sales-prod-writers"
    privileges = ["USE_CATALOG"]
  }
  grant {
    principal  = "sales-data-stewards"
    privileges = ["USE_CATALOG", "USE_SCHEMA", "APPLY_TAG"]
  }
}

resource "databricks_grants" "curated" {
  schema = databricks_schema.curated.id
  grant {
    principal  = "sales-prod-readers"
    privileges = ["USE_SCHEMA", "SELECT"]
  }
  grant {
    principal  = "sales-prod-writers"
    privileges = ["USE_SCHEMA", "SELECT", "MODIFY", "CREATE_TABLE"]
  }
}

Know the semantics before the first apply. databricks_grants is authoritative for its securable: any grant on that catalog or schema missing from the configuration is removed, including grants made by hand. databricks_grant, singular, is authoritative for one principal on one securable, which suits shared catalogs where teams own their own grants. Privileges are written with underscores. If the identity running Terraform relies on MANAGE, declare that grant too, or Terraform removes it. ABAC policies can be managed the same way with databricks_policy_info.

Migrating from the Hive metastore

The upgrade follows a fixed order: provision identities at the account level and turn off workspace-level provisioning; convert workspace-local groups to account groups; attach the workspace to the regional metastore; upgrade tables; grant access to account principals; repoint queries and jobs to three-level names; move files off DBFS; upgrade compute to supported runtimes; and finally disable access to legacy features, so nothing new lands in the Hive metastore.

For tables, Databricks recommends federating first. Hive metastore federation exposes the existing metastore, whether internal, external or AWS Glue, as a foreign catalog governed by Unity Catalog, so access control, audit and lineage apply while workloads keep running. Federated internal metastores allow reads and writes; external ones, including Glue, are read-only. Tables are then upgraded in place without moving data, with SET MANAGED MOVE for managed tables or SET EXTERNAL. Alternatively, SYNC upgrades Hive external tables, and Hive managed tables stored outside the DBFS root, to Unity Catalog external tables; managed tables in the DBFS root need CREATE TABLE ... CLONE or CREATE TABLE AS SELECT.

-- Path 1: after federation, upgrade a foreign table in place to a managed table.
ALTER TABLE hms_sales.orders.order_lines SET MANAGED MOVE;

-- Path 2: register Hive external tables as Unity Catalog external tables.
SYNC SCHEMA sales_prod.legacy_orders FROM hive_metastore.orders
  SET OWNER `sales-platform-admins` DRY RUN;

For large estates, UCX, a Databricks Labs project, provides assessment, group migration and table migration workflows; like other Labs projects it is provided as-is, without Databricks support SLAs. Translate legacy table ACLs into intent (which group should read which domain) and grant at schema level, instead of replaying object-by-object grants. And plan the work by domain, as you would an ETL migration: the companion article on migrating legacy ETL pipelines to Databricks covers waves, dual-running and reconciliation.

Design summary

• One metastore per region; catalogs per environment, or per environment and domain where ownership differs; layers as schemas; catalog names passed as parameters.
• Production catalogs bound to production workspaces; read-only bindings for consumers.
• Catalog-level managed storage in dedicated buckets; few external locations; no READ FILES for end users; managed tables by default.
• Account-level groups from the identity provider; groups own production objects; service principals write to production.
• Grants to groups at schema level; BROWSE for discovery; MANAGE delegated sparingly.
• ABAC with governed tags for rules that generalise, with tag coverage monitored; table-level filters and masks for exceptions.
• Audit and lineage system tables enabled and queried; names settled before production.
• Grants in Terraform or versioned SQL, one mechanism per securable, with drift detection.
• Hive metastore federated first, upgraded by domain, legacy access disabled at the end.

Unity Catalog design is part of ByteLoom’s Databricks and lakehouse engineering work. If your governance model needs a review or a rebuild, discuss your project.

References

• Unity Catalog best practices
• Unity Catalog permissions model concepts: inheritance, ownership and MANAGE
• Unity Catalog privileges reference
• Workspace-catalog binding
• Manually apply row filters and column masks
• Create and manage ABAC row filter and column mask policies
• When to use ABAC or table-level row filters and column masks
• Lineage system tables reference
• Upgrade a Databricks workspace to Unity Catalog
• Terraform Databricks provider: databricks_grants resource