Medallion Architecture: What Each Layer Must Guarantee

A finance dashboard shows revenue 4% higher than the ERP for one week. The cause is ordinary: a source system re-sent two days of order files, the silver table appended them, and every gold aggregate counted those orders twice. The fix should be a code change and a rebuild, but bronze was overwritten on every load, so the clean history the rebuild needs no longer exists. The team had built a medallion architecture with three named layers and none of the guarantees that make the layers worth having. Bronze, silver and gold are not storage tiers. Each one is a promise to the layer above it, and a layer that cannot state its promise is just another copy of the data with its own bill. What follows is what each layer has to guarantee, the two operations that test those guarantees in production (deleting data and replaying it), and when fewer layers are the better design.

What Is Medallion Architecture?

Medallion architecture is a data design pattern that organizes a lakehouse into three layers of increasing quality: bronze holds data as the source sent it, silver holds validated, deduplicated records, and gold holds business-ready tables modeled for reporting and machine learning. Databricks, which popularized the name, calls it "a recommended best practice but not a requirement", and it is a logical pattern rather than a product: it works the same on Delta Lake or Apache Iceberg tables, on Databricks, Microsoft Fabric or plain Spark. The code and procedures below are Delta Lake; on Iceberg the maintenance steps map to expire_snapshots, rewrite_data_files and remove_orphan_files.

If you are still choosing between a lakehouse and a warehouse, start with our comparison of data lakehouse vs data warehouse for mid-size teams. The medallion question comes after that one: how many refinement steps does your data need, and what does each step promise?

Three Layers, Three Guarantees

Every layer exists to give the next one something it can rely on without re-checking. Stated as contracts, the three layers look like this.

Layer	Guarantee to the next layer	Who reads it	What breaks without the guarantee
Bronze	Every record the source sent, unaltered, append-only, with where and when it arrived	Pipelines that build silver, audit	A silver bug cannot be fixed retroactively
Silver	One validated, typed row per business key, at its latest known state	Engineers, analysts who need detail, data scientists	Duplicates and type errors reach every report
Gold	Each business metric defined once, in a model the business recognizes	BI, executives, ML features, applications	Two dashboards disagree on the same number

Bronze: A Replayable Record of What the Source Sent

Bronze guarantees that every record the source delivered is kept as received, appended and never rewritten, together with the file or offset it came from and the time it was ingested. That is what makes every later mistake recoverable: if silver logic is wrong, you fix it and rebuild from bronze instead of asking the source to resend history it may no longer have. For the same reason, Databricks recommends storing most bronze fields as string, VARIANT or binary so an unexpected schema change at the source does not reject records at ingest.

from pyspark.sql import functions as F

bronze = (spark.readStream.format("text")          # one JSON order per line, kept as a raw string
    .load("s3://acme-landing/orders/")
    .select(
        F.col("value").alias("raw_payload"),
        F.col("_metadata.file_path").alias("source_file"),
        F.col("_metadata.file_modification_time").alias("source_file_modified_at"),
        F.current_timestamp().alias("ingested_at")))

(bronze.writeStream.format("delta")
    .outputMode("append")
    .option("checkpointLocation", "s3://acme-lake/_checkpoints/bronze_orders")
    .toTable("bronze.orders_raw"))

A separate bronze copy is not worth paying for when the source is already a durable, replayable store you control. If the files already sit in object storage with a retention policy, and that store can also delete one customer's records on request, a second copy only adds storage and a pipeline to monitor (a Kafka topic fails the second test, so it cannot stand in for bronze when the data is personal); Fabric's guidance likewise recommends a shortcut instead of a copy for files already in OneLake, ADLS Gen2 or S3.

The common mistake is treating bronze as a staging area: typing columns on the way in, or overwriting the table on each load because "the source has the history". The consequence shows up months later. A source adds a field or changes a type, the typed ingest silently drops or nulls those records, and when someone notices, the only copy of the original payload has been overwritten.

Silver: One Validated Row per Business Key

Silver guarantees that each business entity or event appears exactly once, keyed, typed and validated, at its latest known state. Databricks phrases the minimum as "at least one validated, non-aggregated representation of each record". This is the layer where deduplication, type casting, null handling and late data are resolved once, so no gold table has to repeat that logic and drift from the others.

The common mistake is an upsert that assumes the incoming batch is already unique. A Delta MERGE fails when several source rows match the same target row, and the documentation states the new data "needs to be deduplicated within itself". If two copies of an existing order arrive in one batch, the MERGE fails and the stream stops; if two copies of a new order arrive, both are inserted and silver silently holds a duplicate. The batch below is deduplicated first, keeps rejected records instead of dropping them, and only updates a row when the incoming version is newer, which also makes a retried micro-batch harmless.

ORDER_SCHEMA = ("order_id STRING, customer_id STRING, amount DECIMAL(18,2), currency STRING, "
                "status STRING, placed_at TIMESTAMP, updated_at TIMESTAMP")
REQUIRED = ("order_id IS NOT NULL AND amount IS NOT NULL AND updated_at IS NOT NULL "
            "AND status IS NOT NULL")      # gold filters on status; a NULL would vanish there
REJECTS_APP_ID = "silver_orders_rejected_v1"   # change it whenever the checkpoint is reset

def upsert_orders(batch_df, batch_id):
    parsed = (batch_df
        .withColumn("o", F.from_json("raw_payload", ORDER_SCHEMA))   # malformed JSON yields nulls
        .select("o.*", "raw_payload", "ingested_at", "source_file"))

    (parsed.where(f"NOT ({REQUIRED})")                  # keep rejects, never drop them silently
        .write.format("delta").mode("append")
        .option("txnAppId", REJECTS_APP_ID)             # a retried batch_id is not appended twice
        .option("txnVersion", batch_id)
        .saveAsTable("silver.orders_rejected"))
    parsed.where(REQUIRED).createOrReplaceTempView("orders_batch")

    batch_df.sparkSession.sql("""
        MERGE INTO silver.orders AS t
        USING (
          SELECT order_id, customer_id, amount, currency, status, placed_at, updated_at, ingested_at
          FROM (SELECT *, ROW_NUMBER() OVER (
                  PARTITION BY order_id
                  ORDER BY updated_at DESC, ingested_at DESC, source_file DESC) AS rn
                FROM orders_batch)
          WHERE rn = 1
        ) AS s
        ON t.order_id = s.order_id
        WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE SET
          customer_id = s.customer_id, amount = s.amount, currency = s.currency,
          status = s.status, placed_at = s.placed_at, updated_at = s.updated_at,
          ingested_at = s.ingested_at
        WHEN NOT MATCHED THEN INSERT *
    """)

(spark.readStream.option("skipChangeCommits", "true")   # see the deletes section below
    .table("bronze.orders_raw")
    .writeStream.foreachBatch(upsert_orders)
    .option("checkpointLocation", "s3://acme-lake/_checkpoints/silver_orders")
    .start())

A separate silver layer is not the right choice when there is one clean source and one consumer: a single well-typed table feeding a single dashboard gains nothing from a validated copy of itself. Skipping it where it is needed produces the opening scenario. Databricks also advises against writing silver directly from ingestion, because it couples parsing failures to the table everyone downstream trusts.

Gold: Each Business Number Defined Once

Gold guarantees that a business metric has one definition, in a model the business recognizes: dimensional tables, conformed dimensions and pre-aggregated measures that BI tools and ML features read directly. Its value is agreement, not speed. When "net revenue" excludes cancelled and test orders in one gold table, every dashboard that reads it gets the same answer.

CREATE OR REPLACE TABLE gold.daily_net_revenue AS
SELECT CAST(placed_at AS DATE) AS order_date,
       currency,
       SUM(amount)              AS net_revenue,
       COUNT(*)                 AS orders
FROM silver.orders
WHERE status NOT IN ('cancelled', 'test')
GROUP BY CAST(placed_at AS DATE), currency;

Materializing a gold table is not the right choice for every metric when a governed semantic model (a Power BI semantic model, dbt's semantic layer) already defines the measures over silver or a small set of gold facts. The common mistake is one gold table per dashboard request, each with its own filter for what counts as revenue. The operational consequence is metric drift: two reports disagree in a board meeting, and the team spends a week proving which filter was right. Who owns each definition is a governance question; our data governance framework for engineering teams covers how to assign it.

Deletes and Replays: Where Medallion Guarantees Get Tested

Two routine operations run against the grain of the design: deleting records that must disappear, and rebuilding a layer after a logic bug.

Deleting From an Append-Only Bronze

An erasure request collides with the bronze guarantee, because bronze promises never to rewrite history, and getting it wrong is a compliance failure rather than a data-quality one: the customer disappears from the dashboard while their raw records survive in three other places. Delta's streaming documentation is explicit: a stream reading a table that receives an UPDATE, DELETE, MERGE or overwrite "throws an exception" unless you restart it or set an option. The recommended option, skipChangeCommits, is opt-in per stream and skips the whole commit, so the delete never reaches silver. That is why Databricks' GDPR guidance deletes in bronze first and then in every downstream table explicitly. Its documented alternative is streaming silver from bronze's change data feed and handling deletes in code, which pays off when erasures are frequent; the explicit procedure below suits a few requests a month. None of this is needed if personal data never lands in the lake in clear form: when the source pseudonymizes customers and the key mapping lives outside the lakehouse, an erasure is one deleted mapping row.

-- Delta Lake. 0. Upstream first: landing files and any queue still holding the records.

-- 1. Bronze, then every downstream copy, including the quarantine table
DELETE FROM bronze.orders_raw
  WHERE get_json_object(raw_payload, '$.customer_id') = 'c-1042';
DELETE FROM silver.orders_rejected
  WHERE get_json_object(raw_payload, '$.customer_id') = 'c-1042';
DELETE FROM silver.orders WHERE customer_id = 'c-1042';
DELETE FROM gold.customer_lifetime_value WHERE customer_id = 'c-1042';

-- 2. Rebuild aggregates that summed the customer (gold.daily_net_revenue)

-- 3. Tables with deletion vectors: DELETE only marked the rows, so rewrite the files
REORG TABLE bronze.orders_raw APPLY (PURGE);
REORG TABLE silver.orders_rejected APPLY (PURGE);
REORG TABLE silver.orders APPLY (PURGE);
REORG TABLE gold.customer_lifetime_value APPLY (PURGE);

-- 4. Remove the old files once the retention period has passed
VACUUM bronze.orders_raw;
VACUUM silver.orders_rejected;
VACUUM silver.orders;
VACUUM gold.customer_lifetime_value;

Steps 0 and 3 are the ones teams skip. Landing files left in place mean the next full replay re-ingests the erased customer. With deletion vectors enabled (check the table property delta.enableDeletionVectors; a Databricks workspace setting can enable it for new tables), a DELETE only marks rows as removed inside the existing files and VACUUM alone never removes them. Without deletion vectors, DELETE rewrites the files, but the old ones stay readable through time travel until VACUUM, which by default skips files removed within the last seven days (delta.deletedFileRetentionDuration). The consequence of stopping after step 1 is an erasure reported as complete while the customer's records are still readable through time travel or a replay. skipChangeCommits has a cost too: it skips every non-append commit, including an accidental UPDATE on bronze, so changes to bronze outside this procedure need a review step.

Replaying Silver After a Logic Bug

Replay is what bronze is paid for. Delta's documented recovery for a stream that must reprocess its source is to delete the output and the checkpoint and restart from the beginning. In medallion terms: stop the silver stream, fix the logic, rebuild into new tables (silver.orders_v2 and silver.orders_rejected_v2) with a new checkpoint and a new REJECTS_APP_ID, compare row counts and totals against the old tables, swap consumers, then rebuild the gold tables that read them. Both tables need a new version: writing the replay's rejects into the existing quarantine table appends every historical reject a second time, and reusing the app ID makes Delta ignore the rejects write entirely, because batch IDs restart at 0.

A full replay is not the right tool when the bug only touched recent data. If the bad logic shipped last Tuesday, delete the affected keys from silver and that window from the rejects table, then call upsert_orders once as a batch job over bronze rows filtered on ingested_at, with a one-off REJECTS_APP_ID. The delete matters: the update condition skips rows whose updated_at has not changed, so re-merging without it changes nothing.

The mistake is deleting only the checkpoint and replaying into the existing table. The update guard skips every row whose updated_at is unchanged, so the replay corrects no existing row and removes none that the fixed logic now rejects, such as test orders that should never have reached silver. The consequence is a rebuild that looks successful and still carries the bug. A full replay also scans all of bronze, so its cost grows with retention, which must still cover the longest period you might need to reprocess.

When Medallion Architecture Is the Wrong Shape

Three layers is a default, not a law, and each extra hop costs a copy of the data, a pipeline to monitor and added latency. Use fewer layers, or keep a path outside them, in these situations:

1. A small estate with a handful of clean sources: two layers (raw and curated) carry the same guarantees with one fewer pipeline.
2. Decisions that must be made in seconds, such as fraud checks: serve them from the stream, and let the medallion layers feed reporting on the same events afterwards. Our article on event streaming architecture covers that side.
3. A warehouse team already using dbt's staging, intermediate and marts structure: that is the same refinement idea under other names; what it may lack is bronze, if the loader replaces raw tables on each sync.

The signal that a team has outgrown a two-layer setup is repeated cleaning logic: the same deduplication or currency conversion written in three reporting tables. That is the moment a silver layer pays for itself.

Medallion Architecture FAQ

Does medallion architecture replace dimensional modeling?

No. It decides how many refinement steps data passes through; dimensional modeling decides how the gold layer is shaped. Databricks recommends a dimensional model for gold.

Should each layer be a separate workspace?

In Fabric, Microsoft recommends a separate workspace per layer for access control, and allows gold to be a warehouse instead of a lakehouse. On Databricks the equivalent boundary is usually a catalog or schema per layer with its own permissions.

The Decision Rule

Write one sentence per layer stating what it guarantees to the next, and the operation that would break it. If you cannot write the sentence, merge the layer into its neighbour. If you can, test it before production: delete one customer end to end and rebuild one silver table from bronze. A setup that survives both is a medallion architecture; one that does not is three copies of the same problem. Designing and running these pipelines alongside the applications that feed them is part of our data and cloud engineering services.