Data Engineer - Databricks

We are seeking an experienced Databricks Data Engineer to design, develop, and optimize scalable data engineering solutions on the Databricks platform. The ideal candidate should have strong expertise in Databricks, PySpark, SQL, and modern data engineering practices, with experience building enterprise‑grade ETL/ELT pipelines and supporting AI/ML workloads. You will collaborate with cross‑functional teams to deliver high‑quality, scalable, and production‑ready data solutions.

Responsibilities:

• Design, develop, and maintain scalable data pipelines using Databricks and PySpark.
• Build reusable ETL/ELT frameworks for processing structured and unstructured data.
• Develop and optimize Spark jobs for performance, scalability, and cost efficiency.
• Design and implement Delta Lake‑based data solutions and manage Unity Catalog.
• Deploy and manage Databricks Asset Bundles (DAB) and CI/CD pipelines.
• Integrate data from APIs, databases, SFTP, CSV, JSON, and Parquet sources.
• Implement data quality checks, monitoring, audit logging, and reconciliation frameworks.
• Support AI/ML initiatives using MLflow, Model Serving, and Databricks AI capabilities.
• Integrate Databricks solutions with external applications and cloud services.
• Troubleshoot production issues and participate in architecture discussions, code reviews, and technical design sessions.
• Collaborate with architects, product owners, and business stakeholders to deliver scalable data solutions.

Requirements:

• 5+ years of experience in data engineering.
• 3+ years of hands‑on experience with Databricks.
• Strong expertise in Python, PySpark, and SQL.
• Experience with Databricks Workflows, Delta Lake, and Unity Catalog.
• Hands‑on experience with Databricks Asset Bundles (DAB), Git, and CI/CD pipelines.
• Strong understanding of ETL/ELT pipeline development and data warehousing concepts.
• Experience with Azure Data Factory or similar orchestration tools.
• Experience implementing data quality, monitoring, governance, and reconciliation frameworks.
• Excellent analytical, communication, and problem‑solving skills.
• Ability to work independently in a remote, collaborative environment.

Preferred

Skills:

• Experience with MLflow, Model Serving, and Databricks Genie.
• Knowledge of LLM integration, Retrieval-Augmented Generation (RAG), Vector Search, and AI application architecture.
• Experience with Databricks Mosaic AI.
• Exposure to Azure cloud services.
• Experience in retail, supply chain, or financial services domains.
• Strong ownership mindset with the ability to manage multiple concurrent projects.