Data Engineer

We are seeking a Data Platform Engineer to support the infrastructure, operations, and development of our modern Azure-based data platform. This role bridges infrastructure management and engineering development, with primary focus on building reliable systems, supporting production stability, and enabling data engineers and analysts to operate effectively.

The Data Platform Engineer will be responsible for platform infrastructure setup, monitoring, deployment automation, troubleshooting, and maintenance using Azure Data Factory, Databricks, Azure Data Lake Gen2, Delta Lake, and supporting tools. You will work closely with AI - BI teams, data analysts, and technical teams to ensure data pipelines run reliably, performance is optimized, and systems meet SLAs.

This is a hands-on role for someone comfortable with infrastructure concepts, Dev Ops practices, and data engineering—someone who enjoys solving operational problems, improving system reliability, and supporting both technical and business teams.

Key Responsibilities
Infrastructure & Dev Ops
Set up, configure, and maintain Azure Data Factory, Databricks work spaces, Unity Catalog, and Azure Data Lake Storage Gen2 environments across dev, test, and production.
Implement and manage CI/CD pipelines for data platform assets, code, notebooks, and pipeline configurations using Git-based version control and deployment automation.
Deploy data platform updates, patches, and infrastructure changes following change management and testing protocols.
Establish monitoring, alerting, and observability for data pipelines, jobs, and platform health using Azure native tools and logging.
Manage platform credentials, secrets, access controls, and implement security best practices including role-based access control (RBAC) and encryption.
Operations, Support & Troubleshooting
Troubleshoot pipeline failures, data discrepancies, job failures, and performance bottlenecks—identifying root causes and implementing fixes.
Monitor data platform SLAs, response times, and resource utilization; optimize performance for large datasets and complex queries.
Respond to operational incidents, provide on-call support, and document lessons learned and runbooks for common issues.
Maintain platform stability, manage upgrades, apply security patches, and perform capacity planning as data volume grows.
Collaborate with data engineers, BI teams, and business users to resolve operational blockers and improve platform reliability.
Development & Data Engineering
Design, build, test, and maintain data pipelines and ETL/ELT workflows using Azure Data Factory, PySpark, Spark SQL, Python, and T-SQL.
Support and maintain medallion meta data architecture using bronze, silver, and gold data layers; develop Delta Lake tables and optimize for query performance.
Ingest data from databases, flat files, APIs, ERP systems, and other operational platforms; build incremental load and change data capture processes.
Apply data validation, reconciliation, and quality checks across pipelines to ensure data accuracy and completeness.
Collaborate with BI teams to create reporting-ready datasets and support data modelling for analytics, including fact and dimension tables.
Work with stakeholders to understand requirements, clarify business logic, and estimate project delivery effort.

Required Qualifications
3–5 years of experience in Data Engineering, BI Engineering, Data Operations, or a related data platform role.
Experience supporting or maintaining production data platforms, pipelines, or infrastructure—troubleshooting issues and resolving operational problems.
Hands-on experience with SQL and Python or PySpark for data transformation and scripting.
Experience building or maintaining data pipelines using Azure Data Factory or similar orchestration tools.
Working knowledge of Databricks features, including cluster management and job execution.
Understanding of modern data lake or Lakehouse architecture and best practices.
Strong SQL skills, including joins, aggregations, CTEs, window functions, and query optimization.
Understanding of Unity Catalog concepts, including catalogs, schemas, permissions, lineage, and governance.
Famili…