Azure DataBricks Engineer

Responsibilities

• Design, develop, and maintain cloud native data engineering solutions using Azure Databricks.
• Build and manage PySpark notebooks to process large scale structured and semi‑structured datasets.
• Design, create, and maintain Delta Lake tables, ensuring data reliability, ACID transactions, and schema enforcement.
• Develop scalable data workflows and pipelines using Databricks notebooks and orchestration patterns.
• Optimize performance of Spark jobs, including tuning partitions, memory usage, caching strategies, and query execution.
• Work extensively with PySpark and Spark SQL, choosing the appropriate approach based on use case and performance needs.
• Support cloud data migration initiatives, migrating data pipelines from on‑prem or legacy platforms to Azure Databricks.
• Integrate Databricks with upstream and downstream systems (e.g., data sources, storage layers, reporting tools).
• Ensure data pipelines are robust, reusable, and maintainable, following enterprise data engineering best practices.
• Implement error handling, logging, monitoring, and recovery strategies for production‑grade data pipelines.
• Collaborate with data architects, analysts, and downstream consumers to understand data requirements.
• Perform debugging and root cause analysis for data quality, performance, or pipeline failures.
• Support testing, validation, and reconciliation of data during development, migration, and production phases.
• Follow security, governance, and compliance standards applicable to cloud data platforms.
• Actively participate in Agile/Scrum delivery, owning data engineering stories from development through deployment.
• Maintain documentation for notebooks, workflows, data models, and migration approaches.
• Develop and maintain data engineering solutions using Azure Databricks and PySpark.
• Create, enhance, and optimize Databricks notebooks for data ingestion, transformation, and aggregation.
• Design and manage Delta Lake tables and pipelines supporting analytics and reporting use cases.
• Support cloud data migrations, including data validation and performance benchmarking.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with platform, Dev Ops, and data governance teams to ensure environment stability.
• Perform data pipeline testing and validation, ensuring correctness and completeness.
• Troubleshoot and resolve issues related to Spark jobs, Delta tables, and workflow execution.
• Participate in code reviews and enforce data engineering best practices.
• Support production deployments and post‑deployment stabilization.
• Provide inputs to data architecture and platform improvement initiatives.
• Mentor junior data engineers when required.

• Bachelor of Computer Science

$120,000-$140,000 per year

• Discretionary Annual Incentive
• Comprehensive Medical Coverage:
  Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans
• Family Support:
  Maternal & Parental Leaves
• Insurance Options:
  Auto & Home Insurance, Identity Theft Protection
• Convenience & Professional Growth:
  Commuter Benefits & Certification & Training Reimbursement
• Time Off:
  Vacation, Time Off, Sick Leave & Holidays
• Legal & Financial Assistance:
  Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing