Senior Databricks Data Engineer

Company Overview: A leading investment firm with over $185 billion in assets under management is seeking a Senior Databricks Data Engineer to join their team. The role involves building an enterprise-grade data platform on Azure Databricks to support the firm's dynamic business operations.

• Build and evolve an internal Azure Databricks Lakehouse platform using Delta Lake and standardized architectural patterns.
• Develop scalable ingestion and transformation frameworks for multiple source systems.
• Implement curated, trusted datasets and “data products” with clear contracts.
• Operationalize pipelines using Databricks Workflows (Jobs).
• Own performance and cost optimization across workloads.
• Implement and administer governance and security with Unity Catalog.
• Establish engineering standards for the platform.
• Partner with stakeholders to align platform roadmap, priorities, and adoption.
• Provide technical leadership across the team.

• 8+ years of data engineering experience, including 3+ years with Azure Databricks.
• Strong proficiency in Python (PySpark) and Spark SQL.
• Strong experience with Delta Lake.
• Experience product ionizing workloads with Databricks Workflows (Jobs).
• Strong Azure fundamentals relevant to a data platform.
• Knowledge of data architecture and platform concerns.
• Demonstrated ability to work cross-functionally and communicate technical decisions.
• Demonstrated technical leadership and mentoring experience.
• Databricks Certified Data Engineer Professional certification preferred.
• Experience with CI/CD for Databricks assets and familiarity with Terraform preferred.
• Experience with streaming or near-real-time ingestion and/or Delta Live Tables preferred.
• Familiarity with Databricks SQL and serving data to BI tools is preferred.
• Private equity, credit, or Alternative Asset Management industry experience preferred.

Salary: $150,000 – $225,000, Plus Bonus