Senior Databricks Engineer

About The Role

We are seeking an experienced Senior Databricks Engineer to design, develop, optimize, and manage large-scale data pipelines and analytics solutions using Databricks, Spark, and modern cloud platforms (Azure/AWS/GCP). The ideal candidate has deep expertise in distributed data processing, Delta Lake architecture, and end-to-end data engineering best practices. You will work with cross-functional teams—including product, analytics, and delivery—to deliver scalable data solutions for enterprise clients.

Job

Title

Senior Databricks Engineer

Location

Onsite – Sugarland, Tx.

Employment Type

Full-Time

Seniority level

Mid-Senior level

Job Function

Engineering and Information Technology

Industries

IT Services and IT Consulting

Key Responsibilities Databricks & Spark Engineering

• Design, develop, and optimize Spark/Databricks ETL/ELT pipelines at scale.
• Implement efficient Delta Lake architectures (bronze/silver/gold layers).
• Write high-performance PySpark/SQL code following best practices.
• Tune clusters, jobs, and workflows for performance and cost efficiency.

Data Architecture & Modeling

• Develop data models and schemas to support analytics, reporting, and ML workloads.
• Implement medallion architecture following industry and Databricks standards.
• Design scalable and secure storage patterns using cloud-native services.

Cloud Engineering & Integration

• Work with cloud platforms (Azure preferred, AWS or GCP acceptable).
• Implement CI/CD pipelines for Databricks using Git, Azure Dev Ops, or Git Hub Actions.
• Integrate Databricks with third-party systems, ADF, Event Hub, Kafka, APIs, etc.
• Configure workflows in Databricks (Jobs, Workflows, Unity Catalog).

Data Quality, Governance & Security

• Implement data validation, observability, and monitoring frameworks.
• Apply best practices in data governance, lineage, and cataloging.
• Enforce security guidelines using RBAC, Unity Catalog, and encryption policies.

Collaboration & Leadership

• Lead technical solutioning for client projects and POCs.
• Provide mentorship to junior engineers.
• Collaborate closely with business analysts, product managers, and stakeholders.
• Prepare architectural documentation, standards, and reusable accelerators.

Troubleshooting & Optimization

• Investigate data issues, performance bottlenecks, and operational failures.
• Drive continuous improvement across data engineering processes.
• Ensure reliable and timely data delivery for downstream applications.

#J-18808-Ljbffr