Medallion Architecture 详解数据分析与工程领域的奖牌架构
Medallion Architecture（奖牌架构）是一种将数据按质量和成熟度分层的模式，数据从原始状态逐步精炼为可直接用于业务决策的高质量数据产品。Bronze、Silver、Gold 三层结构由 Databricks 提出，现已成为数据湖仓架构的事实标准。[1]
目录
一、什么是 Medallion Architecture
Medallion Architecture（奖牌架构）是一种数据生命周期与质量分层模式，将原始数据、清洗数据和精选数据组织为逐步提升价值的三层结构，以支持可靠的分析、机器学习和运营用途。[2]
该架构的核心思想可以类比为一个精炼厂：原始矿石从一端进入，经过逐步精炼，最终产出可用的精炼金属从另一端输出。Bronze、Silver、Gold 三个名称正是反映了这种渐进式的质量提升过程。[3]
核心设计原则
渐进式丰富：每一层依赖于前一层的输出，数据价值逐层递增。
可追溯性：元数据和血缘必须在层间保留。
契约收紧：随着数据上升，Schema 和语义定义越来越严格。
可复现性：Bronze 层应允许重新处理以重建更高层。
二、三层架构详解
2.1 Bronze 层：原始数据层
Bronze 层存储从数据源采集的原始数据，仅做最小限度的转换。它的核心目标是保留足够的原始信号以支持后续重放。典型数据包括 CDC 事件、API 返回值、日志、Kafka 主题、CSV 导出和供应商数据文件。[3]
Bronze 层典型表结构：
bronze.orders_raw
- source_system -- 数据来源系统
- ingestion_batch_id -- 采集批次ID
- ingestion_time -- 采集时间
- payload_json -- 原始 JSON 负载
- source_event_time -- 源事件时间
- source_file_path -- 源文件路径2.2 Silver 层：清洗规范化层
Silver 层将原始数据转化为可用的实体。这是进行 payload 解析、时间戳标准化、记录去重、类型检查、删除处理、参照数据关联以及构建一致性维度的地方。[3]
Silver 表应对理解领域的工程师和分析师是安全的。它们不一定是最终的业务指标，但应该是可信赖的构建基块。
Silver 层典型表结构：
silver.orders
- order_id -- 主键，已去重
- customer_id -- 客户ID，已关联
- order_status -- 订单状态（已标准化）
- order_total -- 订单总额（已类型化）
- currency -- 币种
- ordered_at -- 下单时间（事件时间）
- updated_at -- 更新时间
- is_deleted -- 软删除标记2.3 Gold 层：业务数据产品层
Gold 层发布回答真实业务问题的数据集。Gold 表驱动仪表板、财务报告、ML 特征、反向 ETL 任务和运营报表。它们应有明确的所有者、文档化语义、新鲜度期望和质量检查。[2]
Gold 层典型表结构：
gold.daily_revenue
- revenue_date -- 收入日期
- region -- 区域
- product_line -- 产品线
- gross_revenue -- 总收入
- refunds -- 退款金额
- net_revenue -- 净收入
- paying_customers -- 付费客户数三、工作原理与数据流
当数据从源系统进入 Medallion 架构时，会经历以下分层处理流程：
- 数据采集 → Bronze 采集器从源系统收集原始记录并写入 Bronze 层。数据以追加方式写入，保持不可变性。包含采集元数据（来源、批次ID、时间戳）。
- Bronze → Silver（转换） 批处理或流处理引擎解析 payload、标准化时间戳、去重、类型校验、处理删除和更新、关联参照数据，生成一致性实体表。
- Silver → Gold（聚合） 聚合和富化作业将 Silver 实体转化为业务指标。Gold 表编码业务语义，配备所有者、新鲜度 SLA 和访问策略。
- Gold → 消费者 BI 仪表板、ML 训练、财务报告、反向 ETL 和 API 直接消费 Gold 层数据产品。
- 重放与修复 如果发现 Bug 或业务逻辑变更，可从 Bronze 重新处理以重建 Silver 和 Gold，同时保持数据源真可见。
四、数据质量检查分层
质量检查应随着数据向上流动而逐层收紧。Bronze 检查证明采集成功且可追溯；Silver 检查证明实体正确性；Gold 检查证明业务含义和消费者安全。[3]
| 边界 | 有效检查项 | 失败处理 |
|---|---|---|
| 源 → Bronze | 文件数量、Schema 捕获、源时间戳、批次ID、重复投递率、畸形 payload 率 | 隔离坏 payload，告警采集负责人，保留原始数据用于重放 |
| Bronze → Silver | 主键唯一性、必填字段、类型转换、迟到数据、删除处理、参照完整性 | 阻止提升或仅发布部分数据并附带可见的新鲜度和质量状态 |
| Silver → Gold | 指标对账、语义定义、维度一致性、行级安全、新鲜度 SLA | 阻止仪表板刷新，通知数据产品负责人，保留上一可信版本 |
五、各层对比
| 维度 | Bronze | Silver | Gold |
|---|---|---|---|
| 核心问题 | 能否完整重放原始数据？ | 工程师能否信任这些实体？ | 业务方能否直接据此决策？ |
| 数据形态 | 原始文件/事件，追加写入 | 清洗、规范化、去重的实体表 | 聚合、业务级数据产品 |
| 数据粒度 | 原始事件级 | 实体级（单条订单/客户） | 聚合级（日收入/月留存） |
| 质量保证 | 完整性和可追溯性 | 正确性和一致性 | 语义稳定性和时效性 |
| 业务规则 | 无 | 不编码业务指标 | 编码业务语义和定义 |
| 典型负责人 | 数据平台/采集团队 | 数据工程/领域团队 | 分析/领域产品负责人 |
| 消费者 | 仅内部（用于重放） | 工程师、分析师 | 业务决策者、BI 仪表板、ML 模型 |
| 存储策略 | 低成本对象存储 | 分区表存储 | 物化视图、高性能查询 |
| 访问控制 | 最宽松 | RBAC 开始收紧 | 最严格，行级安全 |
| Schema 契约 | 捕获但不强制 | Schema-on-Write | 严格语义契约 |
| 可变性 | 不可变（追加） | 可更新（合并/SCD） | 可刷新（定时/触发） |
六、实际示例：订单管道
以一个订单服务为例，展示数据如何从 Bronze 流向 Gold：[3]
6.1 场景描述
一个电商平台的订单服务持续发出 CDC（变更数据捕获）事件。数据需要经过三层处理后，最终为财务团队提供每日收入报表，为运营团队提供履约指标。
6.2 Bronze 层：存储原始 CDC 事件
bronze.orders_cdc
raw_payload -- 原始 CDC payload
op -- 操作类型 (INSERT/UPDATE/DELETE)
source_lsn -- 源日志序列号
source_event_time -- 源事件时间
ingestion_time -- 采集时间6.3 Silver 层：重建订单状态
silver.orders_current
order_id -- 主键，已去重
customer_id -- 客户ID
status -- 订单状态 (pending/paid/shipped/delivered/refunded)
total_amount -- 订单总额
updated_at -- 最后更新时间
is_deleted -- 软删除标记
silver.order_events
order_id -- 订单ID
event_type -- 事件类型
event_time -- 事件时间
previous_status -- 前一状态
next_status -- 后一状态6.4 Gold 层：发布业务指标
gold.daily_revenue
revenue_date -- 收入日期
region -- 区域
channel -- 销售渠道
net_revenue -- 净收入
paid_orders -- 已付款订单数七、最佳实践
✅ 推荐做法
保留 Bronze 用于重放：Bronze 不可变性和保留策略是整个架构可复现性的基石。
Silver 保持事实中立：Silver 应是可复用的事实，不编码单一仪表板的业务逻辑。
Gold 聚焦业务消费：Gold 编码业务语义，配备所有者和新鲜度目标。
每层边界设质量检查：层间过渡需要数据质量检查，而不仅仅是 SQL 转换。
记录血缘和元数据：每层保留血缘，支持审计和可追溯性。
明确所有者：每个 Gold 表有明确负责人和告警路由。
分层回填：回填如同生产变更，需要范围、负责人、验证查询和回滚计划。
❌ 避免做法
过早删除 Bronze：没有重放能力，每次 Schema 或业务规则变更都将很痛苦。
业务指标放入 Silver：Silver 应是可复用事实；Gold 才编码业务含义。
所有内容复制到每层：层应增加价值，而非成倍增加存储和混乱。
层间无测试：每个层过渡都需要数据质量检查。
Gold 表无所有者：没有所有者的 Gold 表只是等待出故障的仪表板依赖。
忽略三种时间：事件时间、采集时间和处理时间必须区分，否则调试困难。
静默回填：静默改变 Gold 表即使新数字更正确也会破坏信任。
7.1 Gold 表发布前检查清单
☐ 谁拥有此表？质量或新鲜度失败时谁收到告警？
☐ 表的粒度是什么？消费者是否可能意外重复计数？
☐ 依赖哪些 Silver 表？这些依赖是否已文档化？
☐ 编码了哪些定义？是否与另一团队的指标定义不同？
☐ 能否从 Bronze 重建？发现 Bug 时是否可复现？
☐ 失败发布时是否有上一可信版本？消费者是否受影响？
八、常见误区
8.1 典型故障模式
| 故障 | 症状 | 可能原因 | 缓解措施 |
|---|---|---|---|
| Gold 数据过期 | 仪表板显示旧数据 | 下游作业失败 | 自动重试和告警 |
| Schema 漂移 | Silver 查询报错 | 上游 Schema 变更 | Schema 校验和软失败 |
| 重复记录 | 指标虚高 | 非幂等采集 | 幂等键和去重 |
| 分区不完整 | 聚合数据缺失 | Bronze 部分写入失败 | 原子写入和暂存 |
| 成本飙升 | 计算费用超预期 | Gold 表重建过于频繁 | 速率限制和成本告警 |
| 访问泄露 | 未授权读取 Gold | RBAC 或策略配置错误 | 策略自动化和审计 |
8.2 架构误用
九、与其他架构的关系
| 概念 | 与 Medallion 的区别 |
|---|---|
| Data Lake（数据湖） | 聚焦存储而非分层精炼；常被混淆为等同于 Bronze 层 |
| Data Warehouse（数据仓库） | 强调分析存储；通常位于 Gold 层位置 |
| Lakehouse（湖仓一体） | 结合数据湖和数据仓库特性；支持分层模式但非同义词 |
| ETL/ELT | 是转换方法而非分层定义；人们常混淆执行方式与分层架构 |
| CDC（变更数据捕获） | 是采集方法而非分层模型；通常作为 Bronze 层的输入源 |
| Delta Lake / Iceberg | 支持分层模式的存储格式；非分层所必需 |
| Data Mesh（数据网格） | 组织模式而非数据分层模型；与分层架构互补而非替代 |
| Semantic Layer（语义层） | 业务视图，通常构建在 Gold 之上 |
9.1 典型架构模式
1. 批处理 ELT on Data Lake
使用定时 Spark 作业将 Bronze 文件转化为 Silver 表，创建 Gold 物化视图。适用于成本敏感且无近实时需求的场景。
2. 流式优先管道
通过 Kafka 采集，使用流处理引擎近实时产出 Silver，聚合到 Gold 支持低延迟仪表板。适用于新鲜度关键的场景。
3. Lakehouse + ACID 存储
将 Bronze 和 Silver 存储为 Delta/Iceberg 格式（支持事务），使用 SQL 引擎创建 Gold。适用于需要原子性和重处理的场景。
4. 混合 CDC + 批处理
通过 CDC 捕获源数据库变更写入 Bronze，微批处理到 Silver，定时聚合到 Gold 用于分析。适用于事务系统集成。
Medallion Architecture ExplainedA Comprehensive Guide to Bronze, Silver, and Gold Data Layers
Medallion Architecture is a data quality stratification pattern that organizes raw, cleaned, and curated datasets into progressively higher-value stages. The three-tier Bronze, Silver, Gold structure was popularized by Databricks and is now the de facto standard for data lakehouse design.[1]
Contents
1. What is Medallion Architecture
Medallion Architecture is a data lifecycle and quality stratification pattern that organizes raw, cleaned, and curated datasets into progressively higher-value stages to support reliable analytics, ML, and operational use.[2]
The core idea can be compared to a refinery: raw ore enters one end, is progressively refined, and usable refined metal exits the other. The names Bronze, Silver, and Gold reflect exactly that progressive quality improvement.[3]
Core Design Principles
Progressive enrichment: Each layer depends on previous layer outputs.
Traceability: Metadata and lineage must persist between layers.
Tightening contracts: Schemas and semantic definitions tighten as data ascends.
Reproducibility: Bronze should allow reprocessing to rebuild higher layers.
2. The Three Layers Explained
2.1 Bronze Layer: Raw Data
The Bronze layer stores source data with minimal transformation. Its core goal is to preserve enough original signal to support replay. Typical data includes CDC events, API payloads, logs, Kafka topics, CSV exports, and vendor data files.[3]
Typical Bronze table structure:
bronze.orders_raw
- source_system -- source system identifier
- ingestion_batch_id -- ingestion batch ID
- ingestion_time -- when received
- payload_json -- raw JSON payload
- source_event_time -- when event occurred at source
- source_file_path -- original file path2.2 Silver Layer: Clean & Conformed Facts
The Silver layer turns raw data into usable entities. This is where you parse payloads, normalize timestamps, deduplicate records, apply type checks, handle deletes, join reference data, and create conformed dimensions.[3]
Silver tables should be safe for engineers and analysts who understand the domain. They are not necessarily final business metrics, but they should be dependable building blocks.
Typical Silver table structure:
silver.orders
- order_id -- primary key, deduplicated
- customer_id -- linked customer
- order_status -- standardized status
- order_total -- typed total amount
- currency -- currency code
- ordered_at -- event time
- updated_at -- last update time
- is_deleted -- soft delete flag2.3 Gold Layer: Business-Ready Data Products
The Gold layer publishes datasets that answer real business questions. Gold tables power dashboards, finance reports, ML features, reverse ETL jobs, and operational reporting. They should have strong ownership, documented semantics, freshness expectations, and quality checks.[2]
Typical Gold table structure:
gold.daily_revenue
- revenue_date -- revenue date
- region -- region
- product_line -- product line
- gross_revenue -- gross revenue
- refunds -- refund amount
- net_revenue -- net revenue
- paying_customers -- paying customer count3. How It Works: Data Flow
As data flows from source systems through the Medallion Architecture, it goes through the following staged process:
- Ingestion → Bronze Ingestors collect raw records from sources and write to Bronze. Data is appended immutably with ingestion metadata (source, batch ID, timestamps).
- Bronze → Silver (Transform) Batch or streaming engines parse payloads, normalize timestamps, deduplicate, type-check, handle deletes/updates, join reference data to produce conformed entity tables.
- Silver → Gold (Aggregate) Aggregation and enrichment jobs transform Silver entities into business metrics. Gold tables encode business semantics with owners, freshness SLAs, and access policies.
- Gold → Consumers BI dashboards, ML training, finance reports, reverse ETL, and APIs directly consume Gold layer data products.
- Replay & Repair If a bug is found or business logic changes, Silver and Gold can be rebuilt from Bronze deterministically while keeping the source of truth visible.
4. Data Quality Checks by Layer
Quality checks should become stricter as data moves upward. Bronze checks prove ingestion worked and is traceable. Silver checks prove entity correctness. Gold checks prove business meaning and consumer safety.[3]
| Boundary | Useful Checks | Failure Action |
|---|---|---|
| Source → Bronze | File count, schema capture, source timestamp, batch ID, duplicate delivery, malformed payload rate | Quarantine bad payloads, alert ingestion owner, keep raw data for replay |
| Bronze → Silver | Primary key uniqueness, required fields, type conversion, late arrivals, deletes, referential integrity | Stop promotion or publish partial data with visible freshness/quality status |
| Silver → Gold | Metric reconciliation, semantic definitions, accepted dimensions, row-level security, freshness SLA | Block dashboard refresh, notify data product owner, preserve previous trusted version |
5. Layer Comparison
| Dimension | Bronze | Silver | Gold |
|---|---|---|---|
| Core Question | Can we replay what arrived? | Can engineers trust these entities? | Can the business act on this? |
| Data Shape | Raw files/events, append-only | Cleaned, normalized, deduped entities | Aggregated, business-level products |
| Granularity | Raw event level | Entity level (single order/customer) | Aggregate level (daily revenue/monthly retention) |
| Quality Guarantee | Completeness & traceability | Correctness & conformance | Semantic stability & freshness |
| Business Rules | None | Does not encode metrics | Encodes business semantics |
| Typical Owner | Data platform / ingestion team | Data engineering / domain team | Analytics / domain product owner |
| Consumers | Internal only (for replay) | Engineers, analysts | Business decision-makers, BI, ML |
| Storage Strategy | Low-cost object storage | Partitioned table store | Materialized views, high-performance |
| Access Control | Most relaxed | RBAC tightening | Most strict, row-level security |
| Schema Contract | Captured but not enforced | Schema-on-Write | Strict semantic contract |
| Mutability | Immutable (append) | Updatable (merge/SCD) | Refreshable (scheduled/triggered) |
6. Example: Orders Pipeline
Using an e-commerce order service as an example, here’s how data flows from Bronze to Gold:[3]
6.1 Scenario
An e-commerce platform’s order service continuously emits CDC (Change Data Capture) events. Data needs to flow through three layers to ultimately provide daily revenue reports for finance and fulfillment metrics for operations.
6.2 Bronze: Store Raw CDC Events
bronze.orders_cdc
raw_payload -- raw CDC payload
op -- operation type (INSERT/UPDATE/DELETE)
source_lsn -- source log sequence number
source_event_time -- source event time
ingestion_time -- when received by platform6.3 Silver: Reconstruct Order State
silver.orders_current
order_id -- primary key, deduplicated
customer_id -- customer ID
status -- order status (pending/paid/shipped/delivered/refunded)
total_amount -- order total
updated_at -- last update time
is_deleted -- soft delete flag
silver.order_events
order_id -- order ID
event_type -- event type
event_time -- event time
previous_status -- previous status
next_status -- next status6.4 Gold: Publish Business Metrics
gold.daily_revenue
revenue_date -- revenue date
region -- region
channel -- sales channel
net_revenue -- net revenue
paid_orders -- paid order count7. Best Practices
✅ Recommended
Keep Bronze for replay: Bronze immutability and retention are the foundation of reproducibility.
Silver stays fact-neutral: Silver should be reusable facts, not encoding one dashboard’s logic.
Gold focuses on consumption: Gold encodes business semantics with owners and freshness targets.
Quality checks at each boundary: Layer transitions need data quality tests, not just SQL transforms.
Record lineage & metadata: Each layer preserves lineage for audit and traceability.
Clear ownership: Every Gold table has a named owner and alert routing.
Treat backfills as production changes: Scope, owner, validation, rollback plan, and consumer communication.
❌ Avoid
Deleting Bronze too early: Without replay, every schema or business-rule change becomes painful.
Business metrics in Silver: Silver should be reusable facts; Gold encodes business meaning.
Copying everything into every layer: Layers should add value, not multiply storage and confusion.
No tests at boundaries: Each layer transition needs quality checks.
Gold without an owner: A gold table without an owner is a dashboard dependency waiting to break.
Ignoring three timestamps: Event time, ingestion time, and processing time must be distinguished.
Silent backfills: Silently changing a Gold table breaks trust even if the new number is more correct.
7.1 Gold Table Publishing Checklist
☐ Who owns this table? Who gets alerted when freshness or quality fails?
☐ What is the grain? Can a consumer accidentally double-count?
☐ Which Silver tables feed it? Are dependencies documented?
☐ What definitions are encoded? Do they differ from another team’s metric?
☐ Can it be reproduced from Bronze? Is it reproducible if a bug is found?
☐ Is a previous trusted version available? Are consumers protected during failed publishes?
8. Common Mistakes
8.1 Typical Failure Modes
| Failure | Symptom | Likely Cause | Mitigation |
|---|---|---|---|
| Stale Gold | Dashboards show old data | Downstream job failure | Automated retries and alerting |
| Schema drift | Silver query errors | Upstream schema change | Schema validation and soft-fail |
| Duplicate records | Inflated metrics | Non-idempotent ingestion | Idempotency keys and dedupe |
| Partial partitions | Missing aggregates | Failed partial writes to Bronze | Atomic writes and staging |
| Cost spike | Unexpected compute bills | Overly frequent Gold rebuilds | Rate limits and cost alerts |
| Access leak | Unauthorized Gold reads | Weak RBAC or policy misconfig | Policy automation and audits |
8.2 Architecture Misuse
9. Relationship with Other Architectures
| Concept | How it differs from Medallion |
|---|---|
| Data Lake | Focuses on storage, not staged refinement; often confused as equivalent to Bronze |
| Data Warehouse | Emphasizes analytics storage; often sits at the Gold layer position |
| Lakehouse | Combines lake and warehouse features; supports layering but not synonymous |
| ETL/ELT | Transformation approach, not a layer definition; execution vs layering often conflated |
| CDC | Ingestion method, not a layering model; typically feeds Bronze |
| Delta Lake / Iceberg | Storage formats that support layering patterns; not required for layering |
| Data Mesh | Organizational pattern, not data staging model; complements rather than replaces layering |
| Semantic Layer | Business view typically built on top of Gold |
9.1 Typical Architecture Patterns
1. Batch ELT on Data Lake
Use scheduled Spark jobs to transform Bronze files into Silver tables, creating Gold materialized views. Use when cost-efficiency matters and near-real-time is not required.
2. Streaming-First Pipeline
Ingest via Kafka, use stream processors to produce Silver in near real-time, aggregate into Gold for low-latency dashboards. Use when freshness is critical.
3. Lakehouse with ACID Storage
Store Bronze and Silver as Delta/Iceberg (transactional), use SQL engine to create Gold. Use when atomicity and reprocessing are needed.
4. Hybrid CDC + Batch
Capture source DB changes to Bronze via CDC, micro-batch to Silver, scheduled aggregations to Gold. Use for transactional system integration.