Senior Azure Databricks Data Engineer

Position:
Senior Azure Databricks Data Engineer

Location:

Fort Worth, Texas Duration:
Contract Job  Job Overview:
This Sr. Engineer 2 role is a hands-on, senior-level engineering position focused on designing and implementing scalable data and application solutions using Python, Spark, Databricks, and Azure Cloud (including Azure ML). The position is hybrid, requiring onsite work Tuesday through Thursday and virtual work on Mondays and Fridays. The engineer will work across multiple agile pods, driving best practices in data engineering, Dev Ops, monitoring, and governance, while collaborating closely with data scientists, analysts, and business stakeholders to deliver robust, production-grade solutions.

Responsibilities:
Participate in and contribute to end-to-end architecture, design, and implementation of scalable data and application solutions using Python and Azure ML. Design and develop complex ETL/ELT pipelines to process, transform, and integrate large-scale datasets from diverse sources using Spark and Python. Optimize Python and Spark jobs for performance and cost-efficiency through techniques such as partitioning, caching, cluster configuration tuning, and bottleneck analysis.

Collaborate with data scientists, analysts, and business stakeholders to translate complex business problems into robust, maintainable technical solutions. Implement and enforce data governance and security strategies, including access controls, encryption, data lineage, and frameworks such as Unity Catalog. Champion code quality, engineering best practices, and standards across pods, including code reviews, testing, and documentation. Design, build, and maintain Dev Ops CI/CD pipelines (e.g., Azure Dev Ops, Git Hub Actions, Jenkins) to automate build, test, deployment, and management of data assets and infrastructure.

Develop and implement comprehensive monitoring, logging, and alerting systems for production-grade pipelines to proactively detect, troubleshoot, and resolve issues. Support multiple agile pods by guiding project planning, sprint execution, and tracking deliverables across various initiatives. Promote and help enforce agile methodologies across pods, ensuring consistent ceremonies, backlog management, and delivery tracking. Leverage AI tools and automation to improve productivity, code quality, and operational efficiency.

Participate in technical interviews, including coding assessments, and contribute to the evaluation of candidates when needed. Work effectively in a hybrid environment, attending mandatory onsite meetings and interviews as required.

Qualifications:

7–10 years of overall experience in software engineering and data engineering roles. At least 7 years of hands-on experience with Python, ETL development, data handling, and SQL. Advanced proficiency in Python, Scala, and/or SQL for data processing and application development. Minimum 5 years of experience with Spark and Databricks programming in production environments. Strong experience designing and implementing complex ETL/ELT pipelines for large-scale datasets.

Solid background in Azure Cloud services, including Azure ML and other relevant Azure data and application services. Experience implementing data governance, security, and compliance frameworks, including access control, encryption, and data lineage (e.g., Unity Catalog). Proven experience designing and developing CI/CD pipelines that deploy code across multiple development and production environments, preferably on Azure using Git Hub Actions or similar tools.

At least 5 years of experience with logging, monitoring, and alerting systems for production workloads. Strong understanding of the full software development life cycle (SDLC), including requirements,…