Lead - Data Engineer

Department: Data
- Engineering

Employment Type: Full Time

Location: 1600 Corporate Center Dr., Monterey Park, CA 91754

Reporting To: Pravin Kumar Rangasamy Subramanian

We are seeking a Lead
- Data Engineer to lead the hands‑on design, development, and operation of key components of Astrana’s enterprise data platform, including the Constellation Lakehouse, Connect API (GraphQL) layer, and Lakebase‑backed applications.

This role will build scalable data pipelines, integrate healthcare data sources, and develop unified data models that support analytics, reporting, AI and machine learning, and operational applications. The position combines advanced data engineering with responsibility for platform reliability, performance, security, and automation within an Azure Databricks environment.

As a hands‑on technical lead, you will translate complex requirements into reliable production solutions, establish engineering standards, and mentor other engineers. You will collaborate with analytics, software engineering, infrastructure, security, business stakeholders, and external vendors to advance the platform and support its growing adoption.

• Put Patients First
• Empower Entrepreneurial Provider and Care Teams
• Operate with Integrity & Excellence
• Be Innovative
• Work As One Team

• Lead the design, development, and delivery of core platform components, including the Constellation Lakehouse in Databricks, Connect API (GraphQL) layer, and Lakebase‑backed applications.
• Build and maintain scalable batch and incremental data pipelines using Databricks, Apache Spark, PySpark, SQL, and Delta Lake.
• Integrate data from transactional databases, APIs, files, and external partners into governed datasets that support analytics and operational applications.
• Design and maintain unified healthcare data models spanning membership, providers, claims, encounters, clinical data, and other business domains.
• Develop reliable ingestion and transformation processes, including change data capture, schema evolution, deduplication, historical tracking, and reconciliation.
• Build and optimize data access layers and APIs in partnership with software engineering teams, supporting consistent business definitions and reliable downstream consumption.
• Develop and support Lakebase‑backed applications and PostgreSQL data structures, including integration with lakehouse datasets and application services.
• Maintain and optimize Databricks work spaces, compute resources, job orchestration, and environment configurations in collaboration with infrastructure and security teams.
• Implement data governance and access controls through Unity Catalog and established organizational standards, including permissions, lineage, and auditability.
• Establish automated data quality checks, source‑to‑target reconciliation, freshness monitoring, and alerting to ensure datasets are accurate, complete, and usable.
• Apply Dev Ops practices, including Git, automated testing, CI/CD, infrastructure as code, and repeatable deployment across development, testing, and production environments.
• Tune pipelines, Spark workloads, SQL queries, and compute configurations to improve performance, scalability, and cost efficiency.
• Own the investigation and resolution of platform incidents and pipeline failures; perform root‑cause analysis and implement preventive improvements.
• Partner with infrastructure teams on backup, recovery, and business continuity procedures, and maintain operational documentation and runbooks.
• Conduct code and design reviews, establish engineering standards, and mentor engineers across internal and vendor teams.

• Bachelor's degree in Computer Science, Engineering, Data Science, or a related field.
• 5+…