Lakehouse and Medallion Architecture

1. How we got here: warehouse → lake → lakehouse

One-line definition for interviews:

“A lakehouse puts warehouse capabilities (ACID transactions, schema enforcement, governance, fast SQL) on top of open file formats in cheap object storage. BI, data science and streaming then share one copy of the data instead of copying it between a lake and a warehouse.”

What makes it possible: an open table format writes a transaction log (Delta _delta_log/*.json, Iceberg metadata/manifest tree) that records which Parquet files make up each table version. Readers see a consistent snapshot; writers commit atomically with optimistic concurrency. See Storage & table formats.

2. Medallion architecture

A convention for layering data by quality and purpose, not a technology. Bronze → Silver → Gold.

flowchart LR
    SRC[Sources] -->|"as-is, append"| B[("Bronze<br/>raw")]
    B -->|"clean, dedupe, conform,<br/>MERGE / SCD2"| S[("Silver<br/>enterprise entities")]
    S -->|"model, aggregate,<br/>business logic"| G[("Gold<br/>marts, KPIs, features")]
    G --> BI[BI]
    G --> ML[ML]
    G --> APP[Apps / APIs]
    S -.->|"data scientists<br/>explore here"| DS[Notebooks]

Layer contracts (what each layer promises)

Metadata columns to add in bronze

SELECT
  *,
  current_timestamp()          AS _ingested_at,
  _metadata.file_path          AS _source_file,      -- AutoLoader / file sources
  'crm_postgres.orders'        AS _source_system,
  :pipeline_run_id             AS _run_id
FROM ...

For Kafka sources also keep topic, partition, offset, timestamp: they’re your dedup and replay keys.

3. Design decisions interviewers ask about

”Why not go straight from source to gold?”

• Replayability: bronze lets you rebuild silver/gold when logic changes, without re-extracting from sources (which may not keep history).
• Debuggability: you can always answer “what did the source actually send?”
• Decoupling: ingestion (one team, many sources) is separated from modelling (many domains).
• Cost: storage is cheap; re-extraction from production databases is expensive and risky.

”Isn’t three copies of the data wasteful?”

Storage is ~$23/TB-month; compute and engineer time cost far more. Mitigate with retention policies (bronze → cold tier after N days), VACUUM, and views instead of tables for thin gold layers.

”How many layers?”

Medallion is a starting point. Real platforms often have more, e.g. a six-layer enterprise layout:

flowchart LR
    L0["0 · Landing<br/>files as delivered"] --> L1["1 · Bronze<br/>raw Delta"]
    L1 --> L2["2 · Silver<br/>conformed entities"]
    L2 --> L3["3 · GDPR / privacy layer<br/>pseudonymised, consent-filtered"]
    L3 --> L4["4 · Gold<br/>domain marts"]
    L4 --> L5["5 · Secure / ABAC serving layer<br/>row filters, column masks per persona"]

Each extra layer must earn its place with a distinct contract (e.g. “nothing downstream of layer 3 can contain direct identifiers”). Adding layers just to look organised is an anti-pattern.

”Batch or streaming between layers?”

Both work. Delta tables are both a batch table and a streaming source/sink, so bronze → silver can be readStream with trigger(availableNow=True) (incremental batch) today and a continuous stream tomorrow, with the same code.

(spark.readStream.table("bronze.orders")
   .withWatermark("event_ts", "1 hour")
   .dropDuplicatesWithinWatermark(["order_id"])
   .writeStream
   .foreachBatch(merge_into_silver)            # idempotent MERGE
   .option("checkpointLocation", "/chk/silver_orders")
   .trigger(availableNow=True)                 # or processingTime="1 minute"
   .start())

4. Anti-patterns

5. Lakehouse platform components (vendor-neutral map)

6. Interview questions