Lead PySpark Engineer - Data, SAS, AWS

Lead Data Engineer - Pyspark / AWS / Python / SAS - Financial Sector

As a Lead PySpark Engineer, you will design, develop, and fix complex data processing solutions using PySpark on AWS. You will work hands-on with code, modernising legacy data workflows and supporting large-scale SAS-to-PySpark migrations. The role requires strong engineering discipline, deep data understanding, and the ability to deliver production-ready data pipelines in a financial services environment.

Essential Skills

PySpark & Data Engineering

Minimum 5+ years of hands-on PySpark experience.
SAS to Pyspark migration experience
Proven ability to write production-ready PySpark code.
Strong understanding of data and data warehousing concepts, including: ETL/ELT, Data models, Dimensions and facts, Data marts, SCDsSpark Performance & Optimisation

Strong knowledge of Spark execution concepts, including partitioning, optimisation, and performance tuning.
Experience troubleshooting and improving distributed data processing pipelines.
Python & Engineering Quality
Strong Python coding skills with the ability to refactor, optimise, and stabilise existing codebases.
Experience implementing parameterisation, configuration, logging, exception handling, and modular design.

SAS & Legacy Analytics

Strong foundation in SAS (Base SAS, SAS Macros, SAS DI Studio).
Experience understanding, debugging, and modernising legacy SAS code.

Data Engineering & Testing

Ability to understand end-to-end data flows, integrations, orchestration, and CDC.
Experience writing and executing data and ETL test cases.
Ability to build unit tests, comparative testing, and validate data pipelines.

Engineering Practices

Proficiency in Git-based workflows, branching strategies, pull requests, and code reviews.
Ability to document code, data flows, and technical decisions clearly.
Exposure to CI/CD pipelines for data engineering workloads.

AWS & Platform Skills

Strong understanding of core AWS services, including: S3, EMR / Glue, Workflows, Athena, IAM
Experience building and operating data pipelines on AWS.
Big data processing on cloud platforms.

Desirable Skills

Experience in banking or financial services.
Experience working on SAS modernisation or cloud migration programmes.
Familiarity with Dev Ops practices and tools.
Experience working in Agile/Scrum delivery environments.

I have three roles available all of which can be worked remotely so dont delay and apply today. I have interview slots ready to be filled

Randstad Technologies is acting as an Employment Business in relation to this vacancy