Databricks Engineer Job San Antonio area,Texas USA,Software Development

Job Title:

Databricks Engineer

Work Location:

Remote United States Candidate

Location:

Must Currently Reside in the United States

Experience:

6+ years of overall experience, including 3+ years of hands-on Databricks experience Industry Preference:
Healthcare, preferably Health Payer

Education:

Bachelor s degree in Engineering (minimum requirement) Travel:
No client-paid travel or expenses Extension:
Possible, depending on candidate performance Python Proficiency: P3 Advanced

We are seeking an experienced Databricks Engineer to design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, SQL, and Delta Lake. The ideal candidate will have strong expertise in enterprise data ingestion, transformation, and processing, along with experience building reusable frameworks for high-volume and metadata-driven data pipelines. The candidate will work with modern Databricks capabilities, including Unity Catalog, Auto Loader, Delta Live Tables (DLT)/Lakeflow, and CI/CD automation using Azure Dev Ops and Databricks Asset Bundles.

Experience in the healthcare payer industry is highly preferred.

Key Responsibilities

Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL.
Build robust batch and streaming ingestion frameworks to process large volumes of enterprise data.
Develop reusable, metadata-driven frameworks for data ingestion, transformation, and processing.
Implement and optimize Delta Lake architectures to ensure data reliability, scalability, and performance.
Configure and manage data governance, access controls, and security using Unity Catalog.
Implement incremental data ingestion and automated file processing using Auto Loader.
Develop and maintain Delta Live Tables (DLT)/Lakeflow pipelines for reliable data transformation and processing.
Implement data-quality checks, validation rules, error handling, and logging mechanisms.
Optimize Spark jobs and SQL queries for performance, scalability, and efficient resource utilization.
Implement CI/CD workflows and deployment automation using Azure Dev Ops and Databricks Asset Bundles.
Collaborate with cross-functional teams to understand business requirements and deliver reliable data solutions.
Troubleshoot pipeline failures, investigate data discrepancies, and improve overall pipeline reliability.
Follow engineering best practices for code quality, documentation, testing, and production support.

Required Technical Skills

Databricks: Strong hands-on experience developing and managing production-grade data pipelines.
PySpark: Advanced proficiency in distributed data processing and Spark transformations.
Python: Advanced programming skills, including reusable frameworks, modular code, and error handling.
SQL: Strong knowledge of complex queries, joins, aggregations, and query optimization.
Delta Lake: Experience with Delta tables, incremental processing, and reliable data architectures.
Data Engineering: Expertise in batch and streaming ingestion, ETL/ELT processes, and data transformation.
Unity Catalog: Understanding of data governance, access controls, and permissions.
Auto Loader: Experience implementing scalable and incremental file ingestion.
DLT/Lakeflow: Experience developing and managing data transformation pipelines.
Azure Dev Ops: Experience with source control, CI/CD pipelines, and deployment automation.
Databricks Asset Bundles: Experience packaging, configuring, and deploying Databricks projects across environments.
Performance Optimization: Ability to troubleshoot and optimize Spark jobs, queries, and data pipelines.

Preferred Qualifications

Experience working in the healthcare payer industry.
Familiarity with healthcare data, payer operations, claims processing, member enrollment, eligibility,…