Senior Azure Databricks Engineer

As a Senior Azure Databricks Engineer, you will:

• Design, develop, and maintain scalable and reliable data processing solutions using Azure Databricks.
• Build and manage robust batch and streaming data pipelines within Databricks environments.
• Develop and optimize data transformation and processing solutions using Python, PySpark, and SQL.
• Design and optimize data models to support scalable processing, performance, and reliability.
• Manage multiple parallel data processing workflows and shared data sources efficiently.
• Implement and maintain CI/CD pipelines using Azure Dev Ops and YAML-based configurations.
• Apply Infrastructure as Code (IaC) using ARM/Bicep for deployment and infrastructure automation.
• Monitor, troubleshoot, and optimize data processing workloads and Databricks environments.
• Collaborate with cross-functional engineering and business teams to deliver reliable and maintainable data solutions.
• Contribute to Agile development practices and continuously improve engineering standards, system stability, and performance.

• Strong hands-on experience with Azure Databricks as a core data engineering platform.
• Strong proficiency in Python, PySpark, and SQL.
• Hands-on experience developing batch and streaming data pipelines.
• Experience with data modeling, transformation, and optimization within Databricks environments.
• Good understanding of Azure cloud services relevant to data engineering.
• Experience with Azure Dev Ops, CI/CD, and YAML-based pipeline configurations.
• Hands-on experience with Infrastructure as Code, particularly ARM/Bicep.
• Experience working in Agile engineering and delivery environments.
• Understanding of modern cloud-based data architectures and end-to-end data engineering solutions.
• Strong communication, collaboration, troubleshooting, and problem-solving skills.

• Build scalable, high-performance, and reliable data processing solutions using Azure Databricks.
• Develop efficient PySpark and SQL-based data transformations.
• Design and manage complex batch and streaming workloads.
• Optimize data pipelines, processing performance, and resource utilization.
• Troubleshoot complex data engineering issues and improve system reliability.
• Implement automated deployment and infrastructure management practices.
• Make pragmatic technical decisions while maintaining scalability and maintainability.
• Work effectively with engineering, architecture, and business stakeholders.
• Drive continuous improvement and maintain high standards of code and solution quality.

• Opportunity to work on enterprise-scale Azure and Databricks data engineering initiatives.
• Exposure to modern cloud-based data platforms and engineering practices.
• A collaborative Agile environment focused on technical excellence and innovation.
• Opportunities to work with advanced data processing, pipeline engineering, and cloud technologies.
• Continuous learning and opportunities for technical and professional growth.
• A culture focused on quality, ownership, scalability, and sustainable engineering solutions.