Explore & Analyze
Understand source tables, inspect data, and identify potential quality problems.
SQL PROJECT
Data Quality & Validation Engine
Clean. Validate. Trust Your Data.
A real-world SQL portfolio project designed to identify, investigate, and validate messy insurance data. Build reliable data quality rules, detect anomalies, and prepare trusted datasets for business reporting.
-- Checking duplicate customer IDs...
SELECT
customer_id,
COUNT(*) AS occurrence_count
FROM dq_project.customers_raw
GROUP BY customer_id
HAVING COUNT(*) > 1
ORDER BY occurrence_count DESC;
Example output · Illustrative
| customer_id | occurrence_count |
|---|---|
| CUST1024 | 5 |
| CUST8731 | 4 |
| CUST4412 | 3 |
postgres=#
Customers
10,000
Raw customer records
Policies
50,000
Raw policy records
Payments
100,000
Raw payment records
Data Quality
Validation
Multiple anomaly categories
Project Overview
This project simulates a real-world insurance data processing environment where raw customer, policy, and payment records may contain missing values, duplicate identifiers, invalid formats, and inconsistent information.
The objective is to investigate data quality issues, define validation rules, and develop SQL queries that support the creation of reliable datasets for downstream analytics and reporting.
Business Problem
How can an insurance company identify inaccurate customer, policy, and payment records before these issues affect operational reporting, financial reconciliation, and business decisions?
PROJECT METHODOLOGY
From Messy Data to Reliable Insights
A structured approach to investigating and improving insurance data quality.
Validate & Clean
Apply business validation rules and standardize inconsistent records.
Generate Reports
Summarize data quality issues and create useful analytical outputs.
Business Impact
Evaluate how improved data quality supports reliable business reporting.
DATA FOUNDATION
Dataset Overview
Three related insurance datasets containing 160,000 raw records for quality investigation.
| Dataset | Description | Records |
|---|---|---|
| customers.csv | Customer information, contact details, and demographic data. | 10,000 |
| policies.csv | Insurance policy details, premiums, coverage dates, and statuses. | 50,000 |
| payments.csv | Payment transactions, amounts, dates, and payment statuses. | 100,000 |
VALIDATION SCOPE
Data Quality Checks
Core business rules investigated throughout the project.
Duplicate Records
Identify repeated customer and policy identifiers.
Missing Values
Find incomplete or mandatory fields.
Invalid Contact Details
Check phone numbers and email formats.
Date Validation
Investigate inconsistent policy dates.
Financial Validation
Detect invalid premium and payment amounts.
Relationship Integrity
Identify unmatched records across datasets.
SOURCE CODE & DOCUMENTATION
View GitHub Repository
Explore the Project on GitHub
The repository will contain SQL scripts, source datasets, validation queries, project documentation, and analysis results.