Senior Data Engineer – Azure & Databricks

Senior Data Engineer – Azure & Databricks

Industry:Technology Consulting / Digital Transformation

Role Level:Lead II – Software Engineering

Location:

Alpharetta, Georgia (USA)

Employment Type:

Full-Time

About the Role:

We are seeking an experienced

Senior Data Engineer – Azure & Databrickswith a proven track record (8+ years) of designing, building, and scaling modern data platforms within the Azure ecosystem.

This role is ideal for someone who thrives in complex enterprise environments, is deeply hands-on with Databricks and Spark, and can lead end-to-end data engineering initiatives from architecture to implementation.

You will be responsible for building robust, high-performance data pipelines, establishing engineering best practices, and collaborating with cross-functional teams to deliver reliable, scalable data solutions across both batch and real-time streams.

Key Responsibilities:

• Architect, design, and implement scalable data platforms and pipelines using
  
  Azureand
  
  Databricks.
• Build and optimise ingestion, transformation, and processing workflows forbatch and real-time(streaming) use cases.
• Work extensively with
  
  ADLS,Delta Lake, and Spark (Python/PySpark) to enable large-scale data engineering capabilities.
• Lead the development of complex
  
  ETL/ELT pipelines, ensuring performance, reliability, and code quality.
• Design conceptual, logical, and physicaldata modelsto support analytics and operational workloads.
• Work with relational and lakehouse systems, including
  
  Postgre
  
  SQLandDelta Lake.
• Define, implement, and enforce best practices fordata governance,security,quality, and architecture standards.
• Partner closely with architects, data scientists, analysts, and business teams to translate requirements into scalable technical solutions.
• Troubleshoot production issues, drive performance optimisation, and support continuous platform improvements.
• Mentor junior engineers and contribute to the creation of reusable components and engineering standards.

Required Qualifications:

• 8+ yearsof hands-on data engineering experience in enterprise environments.
• Strong expertise in Azure services, particularly Azure Databricks, Azure Functions, and (preferably) Azure Data Factory.
• Advanced proficiency in Apache Spark with Python (PySpark).
• Strong SQL skills, includingquery optimisationandperformance tuning.
• Deep experience with
  
  ETL/ELT methodologies, scheduling, and data orchestration.
• Hands-on expertise with Delta Lake(ACID, schema evolution, performance tuning).
• Strong understanding ofdata modelling(normalised, dimensional, lakehouse).
• Proven experience withbatch and streamingtechnologies such as Kafka or Event Hub.
• Solid grasp ofdata architecture, distributed systems, and cloud-native design patterns.
• Ability to design and evaluate end-to-end technical solutions and recommend best-fit architectures.
• Excellent analytical, problem-solving, and communication skills.
• Ability to collaborate across teams and lead technical discussions.

Preferred

Skills:

• Experience with
  
  CI/CDtools such as Azure Dev Ops and Git.
• Familiarity with Infrastructure-as-Code(Terraform, ARM).
• Exposure todata governance and metadata cataloguing tools (e.g., Azure Purview).
• Experience supportingmachine learning or BI workloadson Databricks.

Benefits:

For Full-Time, Regular Employees

• Minimum
  10 days paid vacation annually
• 6 days paid sick leave(prorated for new hires)
• 10 paid holidays
• Paid bereavement and jury duty leave
• Eligibility for
  401(k) Retirement Planwith employer match
• Medical, dental, and vision insurance eligibility (employee + dependents)
• Company-paid:
• Basic life insurance
• Accidental death & disability coverage
• Short- and long-term disability
• Access to HSA and FSA programs (healthcare, dependent care, commuting)