Optimizing Databricks Compute Costs and Spark Workloads
Attribute Databricks spend with system tables, choose compute per workload, tune Spark and Delta layout, and hold the savings with policies and reviews.
Accelerator · Platform assessment
ByteLoom Lakehouse Health Check
A structured, read-only review of a Databricks environment, with findings ranked by impact and effort.
In development
This accelerator is in development and not yet available. This page describes its planned scope and design, which may change as it is built.
- Status
- In development
- Platform
- Databricks
- Access
- Read-only, metadata only
- Output
- Prioritized findings report
- Format
- Assessment and review session
Overview
Why a health check
Databricks environments rarely degrade in one step. Settings chosen for a pilot become production defaults. Clusters multiply faster than the policies meant to govern them. Tables stay in the legacy Hive metastore long after a Unity Catalog rollout, and scheduled jobs keep running as the engineer who first wrote them.
None of this is a failure on its own. Together it adds cost and risk that are hard to see from inside a single team, and harder to explain to the people who pay for the platform.
The Lakehouse Health Check is being designed to make that picture visible in a structured, repeatable way: the same checks, run against configuration and usage metadata, producing findings that can be verified and compared over time.
Design principles
- Read-only by design. It is planned never to change a setting or read the contents of a table.
- Evidence for every finding. Each finding would cite the configuration value or usage record behind it, so it can be checked rather than taken on trust.
- Documented, versioned rules. Every check would carry its rationale and a reference to public Databricks documentation.
- Engineering judgment on top. Automated checks find patterns. A review with your engineers decides which of them matter in your context.
Scope
What it evaluates
The areas and checks below reflect the current design. They will be refined as it is built.
-
Workspace configuration
How each workspace is set up and administered, compared with Databricks' documented recommendations.
- Identity setup: account-level groups, identity provider sync, and workspace-local groups still in use
- Who holds workspace admin rights and the unrestricted cluster creation entitlement
- Compute policy coverage, and how much compute is created outside a policy
- Personal access token use, token lifetimes, and OAuth for automation
- Secrets kept in secret scopes rather than in notebook code or job parameters
-
Job and compute efficiency
Whether workloads run on the right type of compute, sized for the work they actually do.
- Scheduled workloads running on all-purpose compute instead of jobs compute or serverless
- Auto-termination on all-purpose compute and auto-stop on SQL warehouses
- Node utilization from the compute system tables, to find idle or oversized clusters
- Databricks Runtime versions, including compute on runtimes past end of support
- Spark settings carried over from older runtimes that override current defaults, such as disabling adaptive query execution
-
Governance configuration
How access to data is granted, enforced and audited, centered on Unity Catalog.
- Unity Catalog adoption: workspaces attached to a metastore, and tables still in the legacy Hive metastore
- Privilege model: grants to groups rather than individuals, object ownership, and broad grants such as ALL PRIVILEGES on a catalog
- External locations and storage credentials: who can create and use them, and direct cloud storage access that bypasses Unity Catalog
- Compute access modes, including legacy no-isolation shared clusters that cannot enforce Unity Catalog permissions
- Audit configuration: whether the audit log system table or audit log delivery is enabled, and who reviews it
-
Operational reliability
How well production workloads withstand failure, and how quickly someone finds out when they do not.
- Run failures and their termination causes, from the jobs system tables
- Retries, timeouts and duration warnings on production tasks
- Failure notifications routed to a destination someone owns
- Production jobs that run as individual users instead of service principals
- Jobs and pipelines deployed from source control, for example with Declarative Automation Bundles (formerly Databricks Asset Bundles) or Terraform
-
Cost optimization opportunities
Where Databricks spend goes, whether it can be attributed, and which changes are likely to reduce it.
- Spend by workspace, SKU, workload and owner from the billing system tables
- Tag coverage for cost attribution, enforced through compute policies and serverless usage policies
- Account budgets and spend alerts
- Compute choices with a direct cost effect: instance types, spot capacity for workers, Photon and serverless
- Storage overhead from tables that are never vacuumed, and predictive optimization status for managed tables
-
Architecture improvement recommendations
Structural changes that would make the platform simpler to run and easier to extend, drawn from patterns across the other five areas.
- Workspace and catalog topology: separation of development, test and production, and catalog layout by environment or domain
- Table design: managed versus external tables, partitioning versus liquid clustering, and small-file patterns
- Ingestion and pipeline patterns, including where Auto Loader or declarative pipelines would replace custom code
- Infrastructure as Code coverage for workspaces, permissions and jobs
- Platform capabilities that are available but unused, such as system tables and predictive optimization
Approach
How it is designed to work
The intended process, step by step. It describes the design, not a finished service.
-
Read-only collection
With access agreed in advance, a collector would read configuration through the Databricks REST APIs and usage from system tables such as billing, compute, jobs and audit. It would not read the contents of any table. Where a setting can only be read by an administrator, the client could run the collector and share its output instead.
-
Analysis against a rule set
Each check would be a documented rule with a rationale, the evidence it relies on and a link to the relevant public Databricks documentation. The rule set would be versioned, so a later assessment can be compared with the first.
-
Scored findings
Each finding would be rated for severity and estimated effort, so quick wins are separated from structural work. Scores would rank findings within one assessment. They are not a benchmark against other organizations.
-
A prioritized report
Findings would be grouped by area and ordered by priority. Each would state the evidence, the risk or cost involved and a specific recommendation, with a machine-readable export for tracking in the client's own backlog.
-
A review session
A working session with the platform team would walk through the findings, add context that metadata cannot show, and agree what to address first.
Future demo area
Demo preview
An interactive demo is planned for this space. Until it exists, the mock-up below shows the intended structure of the output. Every item in it is a generic example, not a result from a real environment.
Interactive demo planned
Illustrative mock-up. Not real data. Interactive demo planned.
Lakehouse health check report
Example workspace, generic sample findings
- High
- 2
- Medium
- 2
- Low
- 2
- Workspace configuration 1 example finding
- Job and compute efficiency 1 example finding
- Governance configuration 1 example finding
- Operational reliability 1 example finding
- Cost optimization opportunities 1 example finding
- Architecture improvement recommendations 1 example finding
- Severity: High
Scheduled jobs run on all-purpose compute
Evidence: Job compute settings and the billing system tables
Recommendation: Move scheduled workloads to jobs compute or serverless. All-purpose compute is billed at a higher rate and can stay running between runs.
- Severity: High
Privileges granted directly to individual users
Evidence: Unity Catalog grants on catalogs and schemas
Recommendation: Grant privileges to account-level groups, and manage membership through your identity provider.
- Severity: Medium
Production jobs run as individual users
Evidence: The Run as setting on each job
Recommendation: Run production jobs as a service principal so they keep working when people change roles or leave.
- Severity: Medium
Compute without cost-attribution tags
Evidence: Compute configuration and system.billing.usage
Recommendation: Enforce custom tags through compute policies, then report spend by tag from the billing system tables.
- Severity: Low
Long-lived personal access tokens used for automation
Evidence: Token management settings
Recommendation: Move automation to OAuth with service principals, and set a shorter maximum lifetime for new tokens.
- Severity: Low
External tables with many small files
Evidence: Table details and file counts
Recommendation: Schedule OPTIMIZE and VACUUM for these tables, or move them to Unity Catalog managed tables so predictive optimization can maintain them.
Benefits
What it is designed to change
The outcomes it is being designed to deliver for a client.
-
A baseline you can measure against
An evidence-based picture of how the platform is configured and used today, which can be repeated later to show what has changed.
-
A short, ordered list of work
Findings ranked by severity and effort, so the team knows what to fix first instead of facing a long list of observations.
-
Spend you can explain
Databricks cost attributed to workspaces, workloads and owners, with specific changes that are likely to reduce it.
-
Risk found early
Gaps in access control, failure handling and ownership surfaced before they become an incident or an audit finding.
Fit and scope
Who it is for, and where it stops
Who it is for
- Heads of data platform who want an independent view of their Databricks estate before a budget cycle, a contract renewal or a significant scale-up.
- Teams that inherited a Databricks environment, or grew one quickly, and need a clear account of what is running and why.
- Organizations planning a Unity Catalog migration or a governance review, who need to know their starting point.
- Engineering and finance leads who need Databricks spend attributed to teams and workloads.
What you would receive
- Findings report
- Every finding with its area, severity, effort, evidence and recommendation.
- Executive summary
- A short overview of the main risks and opportunities, written for leadership.
- Findings export
- A machine-readable file of the findings, for tracking in your own backlog.
- Review session
- A working session with your platform team to agree priorities and next steps.
What it is not
- Not a compliance certification or audit opinion. It does not attest to SOC 2, ISO 27001 or any other standard.
- Not a Databricks product, and not reviewed, certified or endorsed by Databricks.
- Not a substitute for a security review or penetration test.
- Not a review of your data. It reads configuration and usage metadata, never table contents.
- Not a guarantee of savings. Cost findings are estimates for your team to validate.
Development status In development
In development, not yet available
This accelerator is in development and not yet available. This page describes its planned scope and design, which may change as it is built. This page will be updated as it moves through each stage.
- Concept (earlier stage) Problem and scope defined.
- Design (earlier stage) Checks, outputs and method specified.
- In development Current stage Being built and tested. Not yet offered.
- Early access (not yet reached) Offered to a small number of early-access clients.
- Available (not yet reached) Offered as a standard engagement.
Early access
Register interest in early access.
Tell us about your Databricks environment and what you would want assessed. Early conversations help decide which checks are built first, with no commitment on either side.