Databricks Interview Questions

The control plane is Databricks-managed: UI, notebooks, job scheduler, cluster manager, APIs. The compute plane processes data: classic clusters run in your cloud account and network; serverless compute runs in Databricks-managed, workspace-isolated infrastructure. Table data stays in your cloud storage (or UC managed storage) governed by Unity Catalog.

All-purpose: interactive, shared, higher DBU rate, for development. Job cluster: created per job run and terminated after, cheaper DBU rate, isolated and reproducible, for production on classic compute. Serverless: instant start, autoscaling, no infrastructure to manage, billed in DBUs only, for most jobs and notebooks when libraries and configs are supported.

Databricks’ vectorised C++ query engine that executes Spark SQL/DataFrame operations on columnar batches, speeding up scans, joins, aggregations, writes and MERGE. It doesn’t accelerate Python UDFs or RDD code and has a higher DBU rate, so it pays off on SQL/DataFrame-heavy workloads.

Classic cost = cloud VMs + DBUs (rate depends on compute type and tier); serverless = DBUs only. Levers: production on jobs compute or serverless (not all-purpose), auto-termination, cluster policies (instance types, max workers, tags), spot workers for batch, SQL warehouse auto-stop and right-sizing, Photon where it pays back, liquid clustering and predictive optimisation to scan less, incremental instead of full refreshes, and billing system tables for per-team dashboards and budgets.

They constrain compute creation: allowed instance types and runtimes, max workers, mandatory tags, auto-termination, spot settings, access mode. They enforce cost control, security and consistency, and give teams self-service within guardrails.

An ordered transaction log of JSON commits (with periodic Parquet checkpoints) records which immutable Parquet files make up each version. A commit atomically creates the next log entry; readers reconstruct consistent snapshots from the log; writers use optimistic concurrency with conflict detection; schema enforcement and constraints are checked before commit.

Both try to write the next log version; one wins. The loser checks whether the winning commits logically conflict with what it read and wrote. Disjoint appends succeed on retry; overlapping updates/deletes/merges on the same files fail with a concurrent modification error. Reduce conflicts with disjoint partitions/clustering, explicit predicates, row-level concurrency (deletion vectors + liquid clustering), or serialising writers.

OPTIMIZE compacts small files. Z-order rewrites data sorted on a space-filling curve over several columns for better data skipping, but it’s a full, non-incremental rewrite. Liquid clustering is incremental, its keys can change without rewriting the table, it supports row-level concurrency, and it’s the recommended default for new tables (or CLUSTER BY AUTO). It can’t be combined with partitioning or Z-order.

It permanently deletes data files no longer referenced by the current table version and older than the retention threshold (default 7 days). Older versions referencing them can no longer be time-travelled, and long-running readers of old snapshots can fail. It’s also required to physically remove deleted data for GDPR.

Bitmap files that mark rows as deleted without rewriting the Parquet file (merge-on-read). They make DELETE, UPDATE and MERGE much faster; readers apply the bitmaps, and files are rewritten later by OPTIMIZE/REORG. Physically purging deleted data requires REORG TABLE … APPLY (PURGE) followed by VACUUM.

Deduplicate the source per key, add pruning predicates on clustering/partition columns, cluster the target on the merge key, enable deletion vectors and Photon, keep files well-sized, and check DESCRIBE HISTORY metrics (files rewritten, time) to confirm the effect.

A table property that records row-level changes per commit (insert, update pre/post image, delete, with version and timestamp), so downstream jobs read only changes between versions (readChangeFeed). It’s the standard way to propagate incremental changes from silver to gold or to external systems.

Databricks’ account-level governance layer for data and AI assets: the three-level namespace (catalog.schema.object), centralised permissions inherited down the hierarchy, row filters, column masks and tag-based ABAC, governed volumes, functions and models, automatic table/column lineage, audit logs and system tables, credential management via storage credentials and external locations, plus Delta Sharing and Lakehouse Federation.

Managed: UC controls the storage location and lifecycle; DROP removes data (with an UNDROP window); eligible for predictive optimisation and automatic clustering. External: data at a path you specify in an external location; DROP removes metadata only. Default to managed.

Catalogs per environment (and often per domain: prod_sales, dev_sales) with separate managed storage; schemas per layer (bronze/silver/gold) or per data product; workspace-catalog binding so prod catalogs are only reachable from prod workspaces; grants to groups only; pipelines run as service principals with MODIFY on their schemas; consumers get SELECT on gold.

Attach SQL UDF row filters (ALTER TABLE … SET ROW FILTER) and column masks (ALTER COLUMN … SET MASK) that use current_user() / is_account_group_member(), or at scale define ABAC policies on governed tags (e.g. pii) at catalog/schema level so tagged columns are masked everywhere automatically. Dynamic views are the legacy alternative.

Unity Catalog objects that govern non-tabular files (raw CSV/JSON, images, PDFs, ML artifacts) under catalog.schema.volume with the same permission model, accessed at /Volumes/.... They replace ungoverned DBFS mounts.

Delta Sharing shares live, read-only tables/views/volumes/models with other Databricks accounts or external tools without copying, audited and revocable. Lakehouse Federation goes the other way: it queries external databases (Postgres, Snowflake, SQL Server…) through UC connections and foreign catalogs without ingesting, which is good for exploration and migration but not for heavy repeated workloads.

A Structured Streaming source (cloudFiles) that incrementally ingests new files from cloud storage, using directory listing or cloud file notifications for discovery, and tracking processed files in its checkpoint (RocksDB) for exactly-once ingestion. It supports schema inference, evolution modes and a rescued data column; run it continuously or with availableNow as a scheduled batch.

addNewColumns (default: the stream fails once on a new column, updates the schema, and continues after restart, which is automatic in jobs with retries), rescue (never evolve; unknown fields go to _rescued_data), failOnNewColumns (fail until the schema is updated manually), none (ignore new columns; rescued data only if configured).

A declarative framework for batch and streaming ETL: you define streaming tables, materialized views and views in SQL or Python; the engine builds the dependency graph and manages incremental processing, checkpoints, retries, compute (including serverless with enhanced autoscaling), data quality expectations and an event log. AUTO CDC handles CDC and SCD1/SCD2.

Declarative data quality constraints: expect records violations (warn), expect_or_drop drops violating rows, expect_or_fail stops the update. Metrics go to the pipeline event log for monitoring.

It orders changes per key by the sequence_by column (e.g. LSN or timestamp) and applies only the latest; late-arriving older changes don’t overwrite newer state. Deletes come from apply_as_deletes, and SCD2 history is maintained with start/end columns automatically.

Re-running only the failed tasks and their downstream dependents of a failed job run, keeping successful tasks’ results. Combined with idempotent tasks, it makes recovery fast and cheap.

Schedule (cron), file arrival, table update (UC tables changed), and continuous. Data-aware triggers run work when upstream data is actually ready, avoiding “schedule and hope” timing that processes incomplete data or wastes time waiting.

It processes all data available at start (in multiple batches, respecting rate limits), then stops. It gives batch-like scheduled runs with streaming’s incremental bookkeeping (checkpoints, exactly-once), and is often the most cost-efficient choice for hourly/daily SLAs.

Use idempotent logic in the batch function (MERGE on keys), or Delta’s idempotent write options txnAppId (a stable id for the query) and txnVersion (the batch id), so a replayed batch is skipped. The plain Delta streaming sink already records batch ids in the transaction log.

For Unity Catalog managed tables, Databricks automatically decides when to run OPTIMIZE, VACUUM and statistics collection based on usage, removing manual maintenance jobs and improving query performance and storage cost.

The disk cache automatically stores copies of remote Parquet/Delta data on local SSDs in a fast format and stays consistent with table changes, with no code needed; it’s great for repeated reads of tables. Spark cache() stores a DataFrame’s computed result in memory/disk for reuse within one application and must be managed manually.

YAML-based project definitions of jobs, pipelines and other resources with per-environment targets, deployed via the Databricks CLI. They enable code review, CI/CD and consistent promotion from dev to prod, running as service principals.

Secure cluster connectivity (no public IPs), customer-managed VNet/VPC with egress controls, Private Link for front-end/back-end, IP access lists, serverless egress policies, customer-managed keys, and SSO/SCIM with service principals for automation.

Databricks-managed Delta tables in the system catalog: audit logs, lineage, billing, compute, jobs/pipelines, query history. Uses: cost dashboards and budgets per team, access reviews and anomaly detection, SLA monitoring of job durations, and finding unused tables to delete.