Azure Data Engineer; ADF, Databricks, Spark SQL

Scope of Work:

Responsible for designing, developing, and maintaining Azure-based data pipelines using ADF and Databricks, including ETL/ELT workflows, Spark SQL processing, and Azure SQL integration, while ensuring data quality, performance optimization, and reliable data delivery.

Works on monitoring, troubleshooting, and resolving pipeline and cluster issues, and collaborates with Dev Ops teams using Azure Dev Ops, Git, CI/CD, and monitoring tools like Azure Monitor and Log Analytics to support stable and scalable data operations.

Required Skills

Strong knowledge of Azure services:

Azure Data Factory (ADF)

Azure Databricks

Azure SQL

Spark SQL

Understanding of:

ETL / ELT concepts

Data ingestion patterns

Data warehousing

Experience in:

Troubleshooting pipeline failures

Cluster issues

Performance bottlenecks

Familiarity with:

Azure Dev Ops

Git version control

CI/CD pipelines

Knowledge of monitoring tools:

Azure Monitor

Log Analytics

Databricks logs