Data Quality Engineer; Databricks

Responsibilities

• Develop Delta Lake metric aggregation structures supporting enterprise quality reporting.
• Calculate and publish Data Quality Index (DQI) scores, dimension‑level quality metrics, rule pass/fail rates, dataset compliance scores, and SLA adherence indicators.
• Provide curated outputs to support Power BI quality dashboards and executive reporting.
• Configure automated quality monitoring and alerting mechanisms.
• Implement threshold‑based notifications using Databricks SQL Alerts and Azure Monitor integrations.
• Develop predictive risk scoring models to identify datasets at risk of future quality degradation.
• Support proactive quality management and operational intervention activities.
• Apply Databricks machine learning and pattern analysis techniques to profiling and rule execution outputs.
• Support AI‑assisted root cause analysis across established remediation categories.
• Identify recurring quality issues, systemic defects, and process breakdowns.
• Produce prioritised remediation datasets for business and operational stakeholders.
• Export remediation outputs to Power BI and Excel to support Remediation Tiering Matrix and prioritisation scoring models.
• Maintain documentation, version control, and audit trails for all quality assets, rules, models, and processes.
• Participate in quality reviews, governance forums, and stakeholder workshops.

• Support AI‑assisted root cause analysis across established remediation categories.
• Identify recurring quality issues, systemic defects, and process breakdowns.
• Produce prioritised remediation datasets for business and operational stakeholders.
• Export remediation outputs to Power BI and Excel.

• Collaborate with Data Modellers and Data Catalogue Specialists to ensure quality controls align with authoritative data definitions and metadata standards.
• Support DDA and DGE governance processes by producing required quality artefacts and evidence.
• Maintain documentation, version control, and audit trails for all quality assets, rules, models, and processes.
• Participate in quality reviews, governance forums, and stakeholder workshops.

• Strong experience designing and implementing enterprise Data Quality frameworks.
• Advanced Databricks engineering experience.
• Strong PySpark development skills.
• Experience with Delta Lake, Unity Catalog, Databricks Workflows and Jobs, and Databricks SQL.
• Experience building scalable data validation and quality rule frameworks.
• Knowledge of machine learning techniques for anomaly detection and data quality monitoring.
• Experience using MLflow for model management and deployment.
• Strong understanding of data governance, metadata management, and data life‑cycle processes.
• Experience integrating data quality metrics into reporting platforms such as Power BI.
• Knowledge of cloud‑based data engineering and modern Lakehouse architectures.

• Configured Databricks Data Quality environment.
• Enterprise Data Quality Rule Factory.
• AI‑assisted profiling notebooks and baseline assessment outputs.
• Automated data quality validation and gating processes.
• Data cleansing and remediation pipelines, including Failed Record Register and reprocessing workflows.
• Data Quality metric aggregation tables and DQI reporting feeds for dashboards.
• Predictive quality monitoring and alerting solutions.
• Root Cause Analysis and remediation support datasets.
• Governance audit and compliance artefacts supporting DDA and DGE reviews.