Why Should Data Quality Be Fixed at Ingestion, Not in the Dashboard?

Data Quality

An enterprise data platform (EDP) can connect hundreds of source systems and still produce unreliable information. The problem often lies in what happens after those connections are established. Data enters the platform, gets copied across environments, and moves through transformation pipelines without a consistent framework for validation, standardization, or traceability.

This becomes particularly difficult in enterprises operating across legacy databases and an expanding ecosystem. Each application stores information differently and follows its own update process. As more applications connect, keeping information consistent becomes harder, affecting business reports and, hence, decisions. 

Data quality must be an ingestion and architecture decision, not a reporting correction. The platform needs to preserve source information, enforce quality rules as data progresses through its layers, and maintain lineage across every transformation. Without these controls, even a modern lakehouse can become difficult to govern and increasingly expensive to maintain.

The problem often stems from the way EDPs handle information from connected systems, particularly when ingestion lacks clear rules and controls. Let’s understand how. 

What Creates a Data Swamp in a Modern Enterprise?

A data swamp does not result simply from having too many data sources. It develops when systems lack clear rules for validating, organizing, and maintaining their data.

It develops gradually when every SaaS application, ERP module, POS system, CRM, marketing platform, and operational database is connected directly to reporting. 

The pattern is familiar:

  • Each source has its own identifiers, date formats, currencies, units, statuses, and definitions of a customer, order, claim, or location.
  • Integration teams build point-to-point extracts to meet an immediate reporting need.
  • Analysts patch gaps with spreadsheet logic, SQL overrides, and dashboard-level filters.
  • Nobody owns the canonical definition of a core metric.
  • A source-system change silently changes the meaning of a field downstream.

The result is not merely messy data. It is an estate where leadership cannot confidently answer a basic question: “Which number is correct, and where did it come from?”

So, what is the alternative to cleaning data later?

Give data a path to travel, where each stage has one job and one quality bar, starting with ingestion. That is the idea behind the medallion pattern.

Why Must Data Quality Start at Ingestion?

Ingestion is the first control point in the data lifecycle. Incoming records should be checked for structure, completeness, validity, and consistency before moving downstream.

Without these checks, source-level issues spread across pipelines, forcing engineering teams to repeat corrective work. Data contracts help prevent this by defining expected schemas, mandatory fields, data types, identifiers, and acceptable values for each source.

The checklist for ingestion-stage validation:

  • Schema validation: Detect missing fields, unexpected columns, and incompatible data types.
  • Completeness and uniqueness: Identify missing mandatory values and duplicate records.
  • Referential integrity: Check relationships between connected datasets.
  • Standardization: Normalize dates, currencies, and identifiers.
  • Reconciliation: Compare source and ingested record counts or control totals.
  • Exception management: Quarantine invalid records and track validation failures.

Monitoring should also capture failed records, exceptions, pipeline status, and data freshness. This prevents unreliable data from entering trusted analytical datasets without review or remediation.

Data engineering practices, supported by medallion architecture, help maintain data quality across every stage of the data lifecycle.

How Does Medallion Architecture Work? 

Medallion architecture provides a structured approach to organizing a lakehouse into progressive layers: bronze, silver, and gold. Each layer has a defined role in preparing data for reliable reporting, analytics, and AI applications. 

Microsoft Fabric supports this architecture through OneLake, its unified data lake, allowing data quality to improve progressively across all three layers. So, quality is enforced before a metric reaches a report, AI model, or executive dashboard.

1. Bronze Layer: Preserve Raw Data

The bronze layer is the immutable landing zone. It preserves source data as received: REST API payloads, database extracts, CDC streams, files, and SaaS exports.

This layer’s purpose is not presentation, but protection. Bronze retains the original structure, values, timestamps, and source metadata. When a vendor changes an API field, alters a status code, or introduces a new record type, the platform retains evidence of what changed. Teams can inspect the raw payload, identify the difference, and correct the ingestion or transformation logic without losing historical context.

For instance, in manufacturing, production environments generate a constant stream of sensor readings, machine events, and quality-inspection records. Bronze preserves these inputs exactly as received, so teams can investigate a downtime event, validate an equipment alert, or reconstruct a quality issue from the original source rather than a transformed version of it. 

2. Silver Layer: Clean and Standardize Data

The silver layer turns raw records into governed, conformed data. This is where the platform applies the rules that make data usable across the enterprise:

  • Standardize dates, currencies, units, and time zones.
  • Resolve entity keys across systems.
  • Remove duplicates and handle late-arriving records.
  • Apply referential integrity and schema validation.
  • Flag, quarantine, or reject records that fail quality thresholds.
  • Preserve lineage from source system to transformed record.

Some Industry-specific Use Cases: 

Healthcare

Healthcare organizations often manage patient information across EMR, claims, and engagement systems. The silver layer helps reconcile patient identifiers, standardize clinical and administrative records, and maintain consistency across datasets used for healthcare data and analytics. 

Retail & Digital commerce

Retailers often manage customer profiles, product catalogs, inventory, and sales transactions across retail and e-commerce platforms, stores, and loyalty systems. The silver layer standardizes product identifiers, consolidates customer records, and aligns sales channels, giving merchandising and demand-planning teams a consistent view of performance.

Insurance

Policy, claim, and contract information in insurance and warranty often resides in separate systems, making consistent reporting difficult. The silver layer aligns these records, standardizes policy identifiers, claim statuses, and coverage details, and prepares reliable datasets for claims-cycle analysis.

3. Gold Layer: Prepare Data for Business Use 

The gold layer contains business-ready datasets: conformed dimensions, curated facts, metrics, and semantic models. This is where a “net revenue,” “active policy,” “readmission,” or “food cost percentage” is defined once and reused consistently.

Gold is not a place to hide bad data. It is the controlled output of a governed refinement process. It contains curated datasets structured for specific business and analytical requirements.

Validated information is organized into reporting models, dimensional structures, and governed datasets. These can support Power BI dashboards, operational reporting, and advanced analytical workloads.

Medallion Architecture

Best Practices That Make Medallion Architecture Reliable

Implementing three data layers alone does not guarantee a reliable platform. The architecture must be supported by consistent engineering standards, governance policies, and operational controls. 

The following practices should guide enterprise lakehouse implementations:

  1. Define source-level data contracts: Document expected schemas, mandatory fields, identifiers, and validation requirements for every connected system.
  2. Preserve raw records: Retain source information and ingestion metadata to support historical analysis, audit, and replay.
  3. Establish quality gates: Define measurable conditions for promoting datasets between bronze, silver, and gold.
  4. Design for schema evolution: Account for API and source-system changes without allowing unnecessary downstream dependencies.
  5. Maintain end-to-end lineage: Document source connections, transformation logic, dataset dependencies, and reporting relationships.
  6. Centralize business definitions: Govern shared metrics and analytical models instead of duplicating calculations across reports.
  7. Monitor pipeline performance: Track failures, data freshness, processing duration, validation exceptions, and reconciliation results.
  8. Plan recovery mechanisms: Implement retry logic, historical backfills, and controlled replay procedures.

How Does Sundew Use Microsoft Fabric to Build Governed Data Platforms?

Microsoft Fabric provides an integrated foundation for building and managing enterprise data platforms. At Sundew, we use its capabilities alongside our data engineering expertise to address complex integration and governance requirements. Our focus remains on:

  • Data quality at every stage: Defined bronze, silver, and gold layers establish progressive validation rather than relying on downstream reporting corrections.
  • Resilience to source changes: Raw data preservation and controlled transformations help contain the impact of changing APIs and schemas.
  • Governed analytical consumption: Curated datasets, semantic models, and traceable lineage support consistent reporting and downstream AI workloads.

Our work with a multi-site catering and hospitality business demonstrates how these principles apply in practice. We built a unified lakehouse on Microsoft Fabric, integrating data from financial control, point-of-sale, workforce scheduling, purchasing, and sustainability systems through batch REST API ingestion. This created a common data foundation for consolidated reporting across sites and business units. 

What Should a Strong Enterprise Data Platform Deliver: A Checklist to Follow

As data engineering and architecture experts, we recommend evaluating a platform proposal beyond its technology choices. Based on our experience designing enterprise data platforms, we have curated the following checklist for decision-makers. It helps assess architectural strength, data governance, and long-term operational readiness before committing to modernization. 

Enterprise Data Platform

These considerations help distinguish a platform built for long-term enterprise use from one that simply connects data sources without addressing the complexity of managing them.

Closing Thoughts:

Data Officers Should Make Data Quality an Architectural Priority

What would change if every metric your leadership team reviewed could be traced back to its source without a manual investigation? That question should shape the next decision you make about your enterprise data platform.

At Sundew, we believe the answer begins before the dashboard. It begins with governed ingestion, protected raw data, validated transformations, and business-ready datasets that retain their meaning as they move through the platform.

We help enterprises make this possible through our data engineering capabilities across the Microsoft ecosystem. Using Microsoft Fabric, Azure, Power BI, and Microsoft Purview, we design and build data platforms that connect complex systems, support reliable data flows, and strengthen governance. We also ensure these platforms integrate with existing SAP environments, legacy databases, and SaaS applications. This enables modernization without disrupting your enterprise's existing data landscape.

Our goal is simple: Help enterprises build data platforms they can trust today and confidently build on as their business and AI ambitions grow.

Connect with Sundew to assess your current data estate and design a governed Microsoft Fabric platform built for the decisions ahead. 

Thank You!

Excellent!

Successfully subscribed to Sundew Solutions newsletter!

Acknowledged