Azure DataBricks Engineer

Job Description

Design, develop, and maintain cloud-native data engineering solutions using Azure Databricks. Build and manage PySpark notebooks to process large-scale structured and semi-structured datasets. Create and maintain Delta Lake tables with ACID transactions and schema enforcement. Design scalable data workflows and pipelines, optimize Spark job performance, and select appropriate technologies (PySpark or Spark SQL) based on use case.

• Support cloud data migration initiatives, moving pipelines from on-premises or legacy platforms to Azure Databricks.
• Integrate Databricks with upstream and downstream systems such as data sources, storage layers, and reporting tools.
• Ensure pipelines are robust, reusable, and maintainable following enterprise best practices.
• Implement error handling, logging, monitoring and recovery strategies for production-grade pipelines.
• Collaborate with data architects, analysts, and downstream consumers to understand data requirements.
• Perform debugging and root‑cause analysis for data quality, performance or pipeline failures.
• Support testing, validation and reconciliation during development, migration and production phases.
• Follow security, governance and compliance standards applicable to cloud data platforms.
• Actively participate in Agile/Scrum delivery, owning data engineering stories from development through deployment.
• Maintain documentation for notebooks, workflows, data models and migration approaches.
• Mentor junior data engineers when required.

• Bachelor of Computer Science.
• Proven experience with Azure Databricks, PySpark, Delta Lake and Spark SQL.
• Strong background in designing and implementing data pipelines, data validation and performance benchmarking.
• Experience with Agile/Scrum development and data governance practices.

$120,000 - $140,000 per year.

• Discretionary annual incentive.
• Comprehensive medical coverage: medical & health, dental & vision, disability, pet insurance.
• Family support: maternal & parental leaves.
• Insurance options: auto & home insurance, identity theft protection.
• Convenience & professional growth: commuter benefits, certification & training reimbursement.
• Time off: vacation, time off, sick leave & holidays.
• Legal & financial assistance: legal assistance, 401 K plan, performance bonus, college fund, student loan refinancing.