Data Specialist

Partner | Permanent Recruitment Manager @ Vector Resourcing

Summary

This is a hands‑on Data Ops / Data Quality Engineer role with a strong focus on building data validation frameworks and automated testing for Azure-based data platforms. The role also includes Data Ops responsibilities, ensuring reliable, observable, and well-governed pipeline operations across Fabric Data Factory, Azure Data Factory and Synapse environments. Additionally, the engineer will take on Data Reliability Engineering (SRE) responsibilities.

Key Responsibilities

• Build, maintain, or leverage open-source data validation frameworks to ensure data accuracy, schema integrity, and quality across ingestion and transformation pipelines
• Test and validate data pipelines and PySpark notebooks developed by Data Engineers, ensuring they meet quality, reliability, and validation standards
• Define and standardize monitoring, logging, alerting, and KPIs/SLAs across the data platform to enable consistent measurement of data reliability
• Identify and create Azure Monitor alert rules and develop KQL queries to extract metrics and logs from Azure Monitor/Log Analytics for reliability tracking and alerting
• Write SQL queries and Power Shell (or another scripting language) to automate the execution of validation routines, verify pipeline outputs, and support end‑to‑end data quality workflows
• Collaborate with Data Engineering, Cloud, and Governance teams to embed standardized validation and reliability practices into their workflows
• Document validation rules, testing processes, operational guidelines, and data reliability best practices to ensure consistency across teams

What We’re Looking For

• Strong background in data validation frameworks, automated testing, data verification logic, and quality enforcement
• Automation experience for data validations, reconciliations, and generating alerts
• Experience with Azure Monitor, setting up alert rules, building dashboards using data queried (KQL) from Log Analytics
• Experience with Fabric Data Factory, Azure Data Factory, Synapse pipelines, and PySpark notebooks
• Hands‑on experience calling REST/OData APIs for validating data
• Experience writing SQL and scripts for programmatically doing data validations and reconciliation across systems
• Strong understanding of the Azure ecosystem, including identity, network security, storage, and authentication models
• Working experience with Azure Dev Ops and CI/CD
• Strong debugging, incident resolution, and system reliability skills aligned to SRE
• Ability to work independently while collaborating effectively across Data Engineering, Cloud, Analytics, and Governance teams

Beneficial Experience

• Experience in data space, with strong exposure to data testing, validations, and Data Reliability Engineering
• Experience defining and tracking data quality KPIs, operational KPIs, and SLAs to measure data reliability and performance
• Hands‑on experience using Azure Monitor, Log Analytics, and writing KQL queries to collect monitoring data and define alert rules
• Experience writing SQL and Power Shell (or another scripting language) to automate data validation, reconciliation, and rule execution
• Exposure to data validation frameworks such as Great Expectations, Soda, or custom SQL/PySpark rule engines
• Experience validating pipelines and PySpark notebooks developed by data engineering teams across Fabric Data Factory, Azure Data Factory, and Synapse
• Experience defining and documenting validation rules, operational testing guidelines, and reliability processes for consistent team adoption