Data Validation: Rules, Checks and Continuous Data Quality Monitoring
|
9
min read
What if a dataset passes a validity definition on paper but no one tests the rule before the data reaches a dashboard, model, or regulatory report? Data validation is the operational process of applying defined rules, constraints, formats, domains, and business conditions to determine whether data meets specified requirements. It turns an abstract quality expectation into an explicit test that can pass, fail, warn, reject, or quarantine a record.
Data Validity is a Data Quality dimension. Data Validation is the process used to test that dimension against defined requirements.
That distinction matters. Data Quality describes whether data is fit for its intended use, while validation supplies the executable mechanism that checks records against agreed conditions. This guide explains how data validation rules, data validation checks, measurement, automation, and continuous monitoring work together.
Table of Contents
What Data Validation Is and Why It Exists
Data validation tests whether records conform to predefined expectations about structure, content, relationships, and business meaning. A rule might require a customer identifier, restrict a country code to an approved domain, confirm that a date has the expected format, or ensure that an order amount meets a business condition.
Validation exists because errors become harder to isolate after they move through multiple systems. A malformed record can affect analytics, machine-learning pipelines, operational workflows, or regulatory reporting before someone notices the problem at the dashboard layer. Testing at ingestion or during transformation gives teams a chance to stop, flag, or isolate the record while its origin is still visible.
The distinction between a quality dimension and its operational test is also reflected in DAMA-DMBOK® 2.0 Revised Edition, commonly used as a reference point for data management practices. Validity describes conformity to defined formats, domains, and rules. Data validation is the activity that measures that conformity in a particular dataset and at a particular point in a pipeline.
Why one-time cleansing isn't enough
A cleansing project can repair known defects, but it doesn't protect the next load. Continuous data quality monitoring measures quality over time and applies controls so that data continues to conform to business expectations. The persistent feedback loop helps teams identify drift, degradation, and process breakage before downstream consumers use unreliable values, as described in the research on continuous data quality monitoring.
Gartner's data-quality guidance references an average annual cost of $12.9 million associated with poor data quality, making systematic validation and monitoring a business control as well as a technical practice (Gartner data quality guidance). The practical response isn't unlimited testing. It's choosing the rules that protect the most important data and connecting every failure to an owner and an action.
How Data Validation Rules Work
A data validation rule has three essential parts: condition, scope, and action. The condition defines the logical test, the scope identifies where the test applies, and the action determines what the pipeline should do when the test fails.
Consider a customer_orders table:
order_total > 0customer_emailmatches the approved email patterncountry_codebelongs to an allowed set
The same condition may produce different outcomes depending on severity. A missing regulatory identifier might block a load, while an unusual but reviewable value might generate a warning. A malformed record could move to quarantine rather than disappear, preserving evidence for correction and replay.
Component | Role | Example ( |
|---|---|---|
Condition | Defines the logical test |
|
Scope | Identifies the object and stage being checked |
|
Action | Specifies the response to failure | Quarantine the record and notify the data owner |
Rules can be declarative, such as SQL constraints or YAML configurations, or procedural, such as tests implemented with dbt or Great Expectations. Enterprise integrations may also expose checks through an API, including digna's REST API for data validation.
A useful rule should be reusable and parameterized. For example, a single range-check template can accept different minimum and maximum values for different monetary fields. Store the rule definition, severity, owner, and version beside the data model it protects. That makes changes reviewable when a schema or business policy changes.
Types of Data Validation Checks
No single check type catches every failure. A mature data validation framework layers structural tests with content, relationship, and behavioral checks.
Check Type | Purpose | Customer Dataset Example |
|---|---|---|
Schema | Confirms columns, data types, and nullability |
|
Domain | Restricts values to an approved set |
|
Format | Tests a required pattern | Email follows the accepted structure |
Range | Applies numeric or date boundaries | Quantity isn't negative |
Uniqueness | Detects duplicate identifiers |
|
Referential integrity | Confirms relationships across datasets | Every order references an existing customer |
Statistical or distribution | Finds unusual aggregate behavior | Null rates or row counts shift unexpectedly |
Structural and content checks
Schema checks catch a missing column, an unexpected type, or a changed nullable flag before downstream transformations fail. Domain checks detect values that are syntactically acceptable but not approved, such as an unknown country code or an unsupported customer status.
Format validation handles patterns for email addresses, dates, postal codes, and account numbers. Range validation checks values such as age, quantity, percentages, and monetary amounts against defined boundaries.
Nullability validation separates required fields from optional attributes. A customer ID may be mandatory, while a secondary phone number may be optional. Treating every null as an error creates noise, so the requirement must come from the intended use.
Relationship and behavior checks
Cross-field validation tests logic between attributes. An end date cannot precede a start date, and a currency value must satisfy its defined business condition. Referential validation checks that a customer ID exists in customer master data and that a product ID exists in an approved reference dataset.
Statistical checks add a different layer. They can monitor row counts, null rates, means, standard deviations, and distributions, helping teams find silent drift that static rules may miss. Guidance on data consistency checks is useful when several sources should describe the same entity or event.
For critical elements, combine schema, domain, format, relationship, and distribution checks rather than depending on one category.
Designing Rules That Hold Up in Production
Effective data quality rules should be explicit, measurable, relevant, testable, maintainable, and traceable to a business requirement. A rule such as “customer data should be complete” isn't executable. “Customer ID must not be null for every billing record” is specific enough to test and assign.
Rules may target several quality dimensions:
Completeness: required fields contain values.
Validity: values conform to an approved format or domain.
Uniqueness: identifiers don't repeat where uniqueness is required.
Consistency: related systems agree on shared attributes.
Timeliness: data arrives within the agreed operating window.
Accuracy: values match a trusted source or verified condition.
Conformity: records follow the relevant structural and business standard.
Don't apply the same rigor to every column. Prioritize critical data elements and business-critical rules, especially fields used in financial processing, customer communications, regulated reporting, operational decisions, or high-impact analytics. A rule backlog can be ranked by implementation cost, potential blast radius, how easily a failure can be detected, and how reversible the resulting error would be.
Criticality Tier | Examples | Required Check Types | Cadence | Escalation Path |
|---|---|---|---|---|
High | Regulatory or financial fields | Layered structural, domain, relationship, and trend checks | At each relevant load | Data owner and incident process |
Medium | Customer-facing operational fields | Format, completeness, range, and consistency checks | Load-based or scheduled | Team queue with ownership |
Lower | Exploratory analytical attributes | Basic schema and anomaly checks | Appropriate to use | Review during dataset maintenance |
Parameterized templates reduce duplicated logic. Version rules with schema changes, record the business owner, and retire checks when the underlying requirement changes. Research on manually maintained technical data quality rules highlights why maintainability must be treated as part of the control design, not as an afterthought.
Measuring Validation Results
Validation becomes operationally useful when teams measure more than a single pass or fail signal. Core indicators include validation pass rate, validation failure rate, failed-record count, failed-rule count, failure trends, and critical-rule failure rate.
The standard calculation is:
Validation pass rate = records passing a rule / records evaluated × 100
The corresponding failure view can use the number of records that failed the rule, divided by the records evaluated, then multiplied by 100. Teams can also track mean time to detect, mean time to resolve, rule coverage, and a composite data-quality score index when those measures are defined consistently.
A 99% pass rate isn't automatically good or bad. If the failing records affect a low-risk analytical attribute, the threshold may be tolerable. If they affect a required billing identifier, the same rate may require immediate intervention. Thresholds must reflect business criticality, downstream impact, and the action attached to the rule.
Read trends, not isolated snapshots
Trend tracking shows whether failures are stable, improving, or worsening. A rule that passes today may still be degrading gradually across successive loads. SAP master-data tooling illustrates this approach through scheduled evaluations, trend tracking, current-state monitoring, and comparison against defined thresholds (SAP validation and monitoring documentation).
Composite scores should be weighted by data-element criticality rather than averaged uniformly. A dashboard that combines low-impact and high-impact rules into one unweighted score can make serious failures look insignificant. Detailed guidance on data quality metrics can help teams define a measurement model that connects rule outcomes with ownership and business use.
Data Validation Versus Data Quality and Observability
Data Quality is the broader concept of whether data is fit for use. It includes dimensions such as completeness, accuracy, consistency, timeliness, uniqueness, and validity. Data Validation is one mechanism for testing defined Data Quality requirements.
A validation check asks, “Does this record satisfy this rule?” A quality assessment asks whether the attribute or dataset can be trusted for its intended purpose. Broader Data Quality management can also include profiling, anomaly detection, reconciliation, timeliness monitoring, historical analysis, and remediation.
Validation and observability answer different questions
Data Validation asks:
Does this data satisfy this defined rule?
Data Observability asks:
What is happening to the data and how has its behaviour changed?
Validation is usually deterministic and rule-driven. Observability adds behavioral signals such as trends, anomalies, lineage context, freshness, and structural change detection. They complement each other. A validation rule can identify an invalid postal code, while anomaly monitoring can identify a sudden increase in postal-code failures.
Consider a customer address pipeline. Validation can reject malformed postal codes. Quality scoring can show declining completeness across address fields. Observability can detect that an upstream schema change introduced an unmapped field. Each layer answers a different operational question, and together they provide stronger diagnosis than any layer alone.
Automating and Monitoring Data Validation
Automated data validation should run as part of the data lifecycle, not as an isolated script that someone remembers to execute. Teams can trigger checks after load events through Airflow, Dagster, or Azure Data Factory, or schedule them for datasets that don't have reliable event signals.
Run checks in the warehouse or lakehouse where practical. In-database execution keeps data in place, supports lineage, and avoids unnecessary movement. Materialize results in a control schema with the rule identifier, run timestamp, scope, status, failed-record count, severity, and owner.
Build the response path
Use graduated alerting rather than treating every failure as an outage:
Warnings: Record a non-blocking issue for review.
Blocks: Stop publication when a critical rule fails.
Quarantine: Isolate invalid records for investigation or replay.
Escalation: Route urgent failures to Slack, PagerDuty, or ticketing workflows.
A triage queue should assign an owner, link to a runbook, identify the failing rule, and capture a root-cause category. The resolution should feed back into the control design. Sometimes the rule needs refinement. Sometimes the upstream contract or transformation must change.
dbt tests and Great Expectations provide widely adopted patterns for expressing checks in code. Operational teams also need idempotent runs, safe reruns, schema-drift handling, and clear service expectations for freshness versus validation lag. Teams building an operating model can also consult Hire-a.dev on monitoring for broader monitoring workflow considerations. Continuous data quality monitoring works best when technical signals connect directly to human ownership.
End-to-End Validation Workflow With a Customer Dataset
Take a customer dataset with a country_code field. The data contract declares the expected ISO-3166 alpha-2 format and an approved whitelist of 30 managed regions. The field must not be null, must belong to that domain, and must follow the expected structure.
After each batch, SQL checks identify null values, unknown codes, and distribution changes. An anomaly monitor compares today's country counts with the prior 30-day baseline and surfaces a sudden spike in XX placeholders. Analytics then shows when the deterioration began, rather than only reporting that the latest batch failed.
Step | Action | Tool or Layer | Output |
|---|---|---|---|
1 | Declare the field contract | Schema and business metadata | Expected format and approved domain |
2 | Run record-level checks | SQL validation layer | Null, domain, and format failures |
3 | Compare behavior over time | Anomaly detection | Unusual increase in |
4 | Route the incident | Alerting and ownership workflow | Ingestion team investigates |
5 | Correct and reconcile | ETL fix and backfill | Affected records repaired |
6 | Recompute downstream use | Analytics and reporting | Regional revenue views refreshed |
The ingestion team discovers that an upstream ETL process defaults to XX when geocoding fails. They correct the transformation, backfill the affected records, and rerun downstream revenue analytics by region. Validation identifies the bad values, anomaly monitoring reveals the unusual change, and analytics establishes the timeline. The three layers form a closed feedback loop rather than a disconnected collection of checks.
digna provides record-level Data Validation for explicit rules covering required fields, formats, domains, ranges, cross-field conditions, and referential or business constraints. Its complementary Data Anomalies capability can identify unusual changes in validation results or data behavior, while Data Analytics supports historical analysis of validation metrics and trends. Visit digna to evaluate how these capabilities could fit into your data quality monitoring workflow.
Frequently Asked Questions
What is Data Validation?
Data Validation is the process of applying defined rules, constraints, formats, domains, and business conditions to determine whether data meets specified requirements. It turns Data Quality expectations into explicit, testable controls.
What are Data Validation rules?
Data Validation rules are logical conditions applied to a defined scope, such as a field, record, table, or pipeline stage. They can trigger actions including warning, rejection, quarantine, or correction when data fails.
What are the main types of Data Validation?
Common types include format, data type, domain, range, nullability, cross-field, referential, uniqueness, schema, business-rule, and statistical or distribution checks. The right combination depends on how the data will be used.
How do you measure Data Validation?
Measure pass rate, failure rate, failed-record count, failed-rule count, failure trends, and critical-rule failure rate. Thresholds should reflect the importance of the data element and the consequence of failure.
Is Data Validation the same as Data Quality?
No. Data Quality is the broader assessment of whether data is fit for use. Data Validation is a practical mechanism for testing specific Data Quality requirements.
What is the difference between Data Validation and Data Verification?
Data Validation checks whether data conforms to defined requirements. Data Verification generally confirms that a value or process matches an expected source or result. Verification can support validation, but the terms describe different control activities.
Can Data Validation detect inaccurate data?
It can detect inaccuracy when the organization has a reliable reference, reconciliation rule, or business condition to test against. A value can pass format and domain checks while still being factually wrong, so validation should be combined with profiling, reconciliation, and anomaly detection.
How can Data Validation be automated?
Run rules inside ingestion and transformation pipelines, connect them to orchestrators such as Airflow, Dagster, or Azure Data Factory, store results in a control schema, and route failures through alerting and triage workflows. dbt tests and Great Expectations are common implementation patterns.
How does digna Data Validation support Data Quality?
digna Data Validation applies explicit record-level rules and logs outcomes for targeted quality controls. Used with anomaly detection and historical analytics, it helps teams connect individual failures with changing data behavior and longer-term quality trends.
