Lead Data Engineer - Databricks, PySpark & AWS

Location: Houston, TX - The Galleria (100% Onsite)

Experience: 8+ years

Open to GC holders or US Citizens

Overview

We are seeking a Lead Databricks Engineer to support large-scale data platform initiatives focused on data modernization, cloud migration, and advanced analytics. This is a hands-on senior role requiring deep Databricks expertise, strong AWS experience, and the ability to partner closely with business and technical stakeholders.

The ideal consultant has led complex Databricks implementations end-to-end, can code at a high level, and has experience modernizing legacy data platforms into scalable, cloud-native architectures.

Key Responsibilities

• Lead the design and implementation of enterprise-scale Databricks solutions on AWS
• Drive data modernization initiatives, including lift-and-shift and re-architecture of legacy data platforms
• Build and optimize data pipelines using Python, PySpark, and SQL
• Design and manage Delta Lake architectures and implement Unity Catalog for governance and access control
• Develop and support streaming and batch data workloads
• Configure and optimize Databricks clusters and serverless compute for performance and cost
• Integrate Databricks with upstream and downstream systems (APIs, data sources, analytics tools)
• Partner with stakeholders to gather user and business requirements and translate them into technical solutions
• Implement observability, monitoring, and cost controls, including usage, volume, and pricing metrics
• Support data domains related to billing, accounting, pricing, and volume metrics
• Provide technical leadership, best practices, and mentorship to engineering teams

Required Skills & Experience

• 8+ years of experience in data engineering, with recent hands-on focus on Databricks
• Strong experience deploying Databricks on AWS
• Advanced coding skills in Python, PySpark, and SQL
• Deep knowledge of Delta Lake, Unity Catalog, and Databricks workspace governance
• Experience with streaming data (Structured Streaming, Kafka, or similar)
• Strong understanding of Databricks cluster management, serverless compute, and performance tuning
• Experience integrating Databricks with enterprise systems and data sources
• Proven ability to work directly with business and technical stakeholders
• Experience supporting financial data domains (billing, accounting, pricing, usage metrics) is highly preferred
• Strong communication skills and ability to lead technical discussions
• Experience with AKS / Kubernetes environments

Nice to Have

• Databricks or AWS certifications
• Consulting or contracting background in large enterprise environments

Work Requirements

• 100% onsite role in Houston, TX
• Must be authorized to work in the United States