Senior Data Engineer

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines using Azure Data Factory (ADF) and Apache Airflow, processing structured and semi-structured data across the Medallion architecture (Bronze → Silver → Gold).
• Implement incremental load patterns, change data capture (CDC), and event‑driven ingestion to ensure data freshness across the platform.
• Build and optimise Snowflake data warehouse objects—including tables, views, dynamic tables, streams, tasks, and stored procedures—for performance and cost efficiency.
• Develop modular, tested dbt models aligned to each Medallion layer, enforcing consistent naming conventions, documentation, and lineage across all transformations.
• Embed automated data validation at every Medallion layer using Elementary to ensure anomaly detection, freshness checks, and schema drift alerts are in place before data reaches consumers.
• Define and enforce data contracts between producers and consumers—row count checks, null rate thresholds, referential integrity, and value domain validation.
• Build and maintain data quality dashboards to give engineering and business stakeholders real‑time confidence in platform health.
• Manage and optimise Azure Data Lake Storage Gen2 (ADLS) folder structures, lifecycle policies, access tiers, and partition strategies.
• Build and maintain Azure Functions and Azure Logic Apps for lightweight event‑driven processing, orchestration triggers, and operational automation.
• Manage secrets, credentials, and environment‑specific configuration securely using Azure Key Vault—no hardcoded credentials in pipelines or code.
• Contribute to infrastructure‑as‑code practices (Terraform or Bicep) for provisioning Azure data services.
• Translate ambiguous business requirements into well‑defined data models and pipeline designs, working with analysts and stakeholders to validate assumptions before build.
• Participate in code reviews, enforce standards, and mentor junior engineers on data engineering best practices.
• Support CI/CD adoption for pipeline and dbt model deployment across Dev / Test / Prod environments.

• 4+ years of professional data engineering experience, with at least 2 years on Azure cloud data platforms.
• Advanced SQL skills—window functions, CTEs, recursive queries, query profiling.
• Experience with Snowflake, including streams, tasks, snowpipe, dynamic tables, row‑level security, virtual warehouse tuning, and credit cost optimisation.
• Proficiency with dbt and Elementary: writing, testing, and documenting production dbt models;
  Elementary integration for data observability and anomaly detection; dbt incremental strategies, snapshots, and semantic layer usage.
• Strong Azure cloud knowledge:
  Azure Data Factory pipeline authoring, triggers, parameterisation, linked services; ADLS Gen2 zone/folder design, lifecycle management, Parquet/Delta partitioning;
  Azure Key Vault secret management and managed identities;
  Azure Functions/Logic Apps event‑driven triggers and lightweight automation.
• Airflow experience: DAG authoring, task dependencies, XCom, sensors, connection management, deployment, and monitoring in cloud‑hosted environments.
• Python skills: data pipeline scripting, basic PySpark, REST API integration, unit testing of pipeline logic and transformation functions.
• Hands‑on experience implementing Bronze, Silver, Gold Medallion architecture; data validation checks at each layer; schema evolution handling and SCD Type 2 dimension management.

• Exposure to Snowflake Cortex, dbt Semantic Layer, or Boomi Data Hub for AI‑assisted data enrichment within pipeline layers.
• Experience integrating LLM‑based quality checks or AI‑assisted anomaly detection into data workflows.
• Familiarity with Microsoft Fabric and One Lake as a complementary or future‑state platform.
• Knowledge of data mesh or data product thinking and how it maps to Medallion layer ownership.
• Experience with Terraform or Bicep for Azure infrastructure provisioning.

Base salary range: $140,000 – $150,000 per year for full‑time employment, exclusive of benefits. Salary determined by individual skills, education, and experience.

• Comprehensive…