Databricks 1: Platform Architecture and Compute

Databricks interviews start with “how does the platform work?” and quickly move to “which compute would you use for this workload, and what will it cost?”. This module gives you the architecture vocabulary and the decision rules.

1. The architecture: control plane and compute plane

flowchart TB
    subgraph ACC["Databricks account (one per organisation)"]
        IAM[Identity: users, groups, service principals<br/>SCIM from Entra ID / Okta]
        UC[Unity Catalog metastore<br/>one per region]
        WS1[Workspace: prod]
        WS2[Workspace: dev]
    end
    subgraph CP["Control plane (Databricks-managed)"]
        WEB[Web app, notebooks, jobs scheduler,<br/>cluster manager, REST APIs, query history]
    end
    subgraph CMP["Compute plane"]
        CLASSIC["Classic compute<br/>VMs in YOUR cloud account / VNet"]
        SLS["Serverless compute<br/>in Databricks' account, isolated per workspace"]
    end
    STOR[("Cloud object storage in your account<br/>S3 / ADLS / GCS: Delta tables, volumes")]
    WS1 --> CP
    CP --> CLASSIC & SLS
    CLASSIC --> STOR
    SLS --> STOR
    UC -. governs .-> STOR

• Control plane: run by Databricks. Hosts the UI, notebooks’ source, job definitions, cluster management and APIs. Your data isn’t stored here (beyond metadata such as query text and results caching settings).
• Compute plane: where data is processed.
  • Classic: clusters run as VMs in your cloud subscription and network (you control VNet/VPC, NAT, firewalls; you pay the cloud provider for VMs plus Databricks for DBUs).
  • Serverless: compute runs in Databricks-managed infrastructure, starts in seconds, scales automatically, and is billed in DBUs only. Isolation between customers is enforced by Databricks.
• Storage: your data lives in your object storage (or in Unity Catalog managed storage you configure), as Delta/Parquet files. Databricks compute accesses it with credentials governed by Unity Catalog.
• Account vs workspace: the account holds identities, Unity Catalog metastores, billing and networking configs. Workspaces are environments (prod, dev, per business unit) attached to a regional metastore, which lets several workspaces share governed data.

Why it matters in interviews: security questions (“does data leave our account?”), networking (private link, no public IPs), and the classic vs serverless trade-off all follow from this split.

2. Compute types and when to use each

Access modes (Unity Catalog era):

• Standard (formerly shared): multi-user, with user isolation and fine-grained access control enforced; supports Python/SQL/Scala on recent runtimes.
• Dedicated (formerly single user): assigned to one user or group; needed for some workloads (ML runtimes, RDD APIs, certain libraries).
• No isolation: legacy, not governed by Unity Catalog. Avoid.

3. Databricks Runtime and Photon

• Databricks Runtime (DBR): a versioned bundle of Spark plus Databricks optimisations, Delta Lake and libraries. Use LTS versions for production stability; the ML runtime adds ML libraries.
• Photon: a vectorised query engine written in C++ that executes Spark SQL/DataFrame operations on columnar batches. It speeds up scans, joins, aggregations and writes (especially Delta MERGE, wide tables and SQL-heavy workloads). It’s on by default for SQL warehouses and serverless.
  • Best for SQL/DataFrame-heavy work; no benefit for Python UDFs or RDD code (those still run outside Photon).
  • Photon compute has a higher DBU rate, so the speed-up must exceed the price difference. It usually does for heavy SQL; measure for your jobs.
  • The query profile shows which operators ran in Photon.

4. Autoscaling, spot and cluster policies

• Autoscaling adds and removes workers based on pending tasks. Classic autoscaling suits batch; enhanced autoscaling (pipelines, serverless) also considers streaming backlog and scales down more aggressively.
• Auto-termination on all-purpose clusters (e.g. 30-60 minutes idle) is the single easiest cost saving.
• Spot/preemptible workers for fault-tolerant batch: keep the driver on-demand, use spot with fallback for workers. Losing a spot node causes task retries (and lost shuffle files), so avoid spot for long stateful streaming jobs.
• Cluster policies constrain what users can create (instance types, max workers, mandatory tags, auto-termination, allowed runtimes, spot ratio). They’re the main governance lever for classic compute cost and security.
• Tags on compute flow through to cloud billing and to the billing system tables for chargeback.

5. How cost works: DBUs

• A DBU (Databricks Unit) is a normalised unit of processing per hour. Each compute type has a DBU consumption rate per instance and a price per DBU that depends on the product (jobs, all-purpose, SQL, serverless, pipelines), tier and cloud.
• Classic compute cost = cloud VM cost + DBU cost. Serverless cost = DBUs only (higher DBU price, but no idle VM time and no separate cloud bill for compute).
• Rule of thumb: jobs compute is much cheaper per DBU than all-purpose, so running production on all-purpose clusters is a classic cost mistake.
• Visibility: the system.billing.usage and system.billing.list_prices system tables give usage by workspace, SKU, job, warehouse and custom tag. Build cost dashboards and budgets from them.

Cost levers interviewers expect: jobs instead of all-purpose compute, serverless for spiky/short workloads (no idle), auto-termination, right-sizing (memory vs compute-optimised), spot for batch, Photon where it pays back, liquid clustering and predictive optimisation to reduce scanned data, SQL warehouse auto-stop and sizing, and cluster policies to prevent oversized clusters.

6. Choosing compute: worked decisions

Interview questions