Senior Data Engineer

We are looking for a Senior Data Engineer to design, build, and operate modern data platforms  will work across cloud-native Azure infrastructure, Microsoft Fabric, Azure SQL, and AWS, building reliable pipelines, performant data models, and self-service analytics that drive real business decisions. A meaningful part of this role involves containerized workload execution -- designing and deploying pipeline jobs using Azure Container Apps Jobs for scheduled and event-driven data processing.

This is a hands‑on role suited for an engineer who thrives in ambiguity, values clean architecture, and moves comfortably between strategy and implementation.

• Design and build scalable data pipelines using Python and cloud-native orchestration tools, including Azure Data Factory, Azure Container Apps Jobs, and Fabric Data Pipelines.
• Architect data solutions across Microsoft Fabric Warehouses, Azure SQL Database, and AWS (S3, Redshift), selecting the right tool for the workload.
• Implement Medallion/layered architecture patterns (Bronze to Silver to Gold) for structured, governed data delivery.
• Manage and optimize large-scale data warehouse environments with a focus on performance, cost, and maintainability.

• Develop Python-based ETL/ELT pipelines to ingest and transform data from APIs, flat files, databases, and SaaS platforms.
• Build and deploy containerized pipeline jobs using Azure Container Apps Jobs, including scheduling, scaling rules, secrets management via Azure Key Vault, and integration with Azure Container Registry.
• Build and maintain data movement between on-premises SQL Server environments and cloud targets.
• Design idempotent, fault-tolerant pipeline patterns with robust logging, alerting, and retry logic.
• Collaborate with analytics and reporting teams to deliver clean, well-documented data models for Power BI or similar BI tools.

• Manage data infrastructure across Azure (Fabric, Azure SQL, Azure Data Lake, Key Vault, Container Apps, Container Registry) and AWS (S3, EC2, RDS/Redshift).
• Containerize data workloads using Docker; deploy and operate them as Azure Container Apps Jobs for scheduled batch processing and event-triggered pipeline execution.
• Implement infrastructure-as-code principles and version-controlled deployment practices using Git Hub, Bicep or Terraform, and CI/CD tooling (Azure Dev Ops or Git Hub Actions).
• Monitor platform health, optimize compute and storage costs, and enforce data security and access governance.

• Partner with data analysts, BI developers, software engineers, and business stakeholders to translate requirements into technical solutions.
• Maintain thorough technical documentation: pipeline specs, data dictionaries, runbooks, and architecture diagrams.
• Champion engineering best practices: code reviews, testing, modular design, and reusable frameworks.
• Mentor junior engineers and contribute to team standards and knowledge sharing.

• Python: fluent in writing production-grade pipelines, data transformations, and automation scripts.
• RDBMS:
  Advanced T-SQL and/or ANSI SQL; experience with SQL Server, Azure SQL DB, and cloud warehouse query engines (Redshift, Fabric).
• MS Fabric:
  Warehouses, Lake houses, Data Pipelines, One Lake, and Fabric's unified analytics model.
• Azure ecosystem:
  Azure Data Factory, Azure SQL Database, Azure Data Lake Storage, Azure Key Vault, Azure Container Apps Jobs, Azure Container Registry, and related services.
• Containerization:
  Docker image development, container registry management, and deploying workloads as Container Apps Jobs with schedule and event triggers, scaling rules, and environment…