More jobs:
Python/Spark/AI Developer
Job in
Durban, 3611, South Africa
Listed on 2026-10-04
Listing for:
Together Staffing group
Full Time
position Listed on 2026-10-04
Job specializations:
-
Software Development
Python, Backend Developer
Job Description & How to Apply Below
Our client is seeking an experienced Python / Spark / AI Developer to help replatform legacy T-SQL workloads onto modern Spark and Delta Lake pipelines while developing robust APIs and AI-enabled capabilities. This intermediate-to-senior role is suited to a developer who is comfortable working autonomously in a specification-driven environment and delivering production-quality, well-tested code.
Key Responsibilities- Build production-grade Spark/PySpark pipelines using Delta Lake.
- Replatform legacy T-SQL logic into Spark SQL while preserving functional parity.
- Develop modern, type-safe Python using strong testing and static-analysis practices.
- Build REST APIs using FastAPI or equivalent frameworks.
- Implement authentication, idempotency and reliable job-status semantics.
- Validate migrated pipelines against legacy systems and provide evidence of output parity.
- Work from design documents and Architecture Decision Records (ADRs).
- Contribute tests and documentation alongside code changes.
- Participate in Git-based, PR-driven development and CI processes.
- 4+ years' professional Python development experience.
- 2+ years' production experience building Spark/PySpark or comparable distributed data pipelines.
- Strong Python 3.12 experience, including type hints, Pydantic, modern packaging and static analysis.
- Strong Apache Spark/PySpark knowledge, including Data Frame API, Spark SQL, partitioning and performance tuning.
- Understanding of Spark driver/executor architecture and Spark Connect.
- Production experience with Delta Lake or equivalent lakehouse technologies such as Iceberg or Hudi.
- Strong SQL skills and the ability to translate legacy T-SQL logic into Spark SQL.
- Experience with MERGE operations, schema evolution, idempotent writes, hashing, surrogate keys and deduplication.
- Strong automated testing experience using pytest.
- REST API development experience, preferably with FastAPI.
- Experience with Docker-based development environments.
- Strong Git and CI/CD discipline, including linting, type-checking and automated testing.
- Bachelor's degree in Computer Science, Engineering or equivalent experience.
- Ability to work independently from written specifications and technical requirements.
- LLM integration and AI feature development.
- Experience with Ollama, vLLM, llama.cpp, LM Studio or hosted AI APIs.
- Data governance, PII protection and privacy engineering.
- Experience with Hive Metastore, Trino or Apache Ranger.
- Azure data technologies including ADLS Gen2, Synapse/ADF, Key Vault or Service Bus.
- Observability tools such as Open Telemetry, Open Lineage or structured logging.
- Data migration and output-parity experience.
- Databricks Certified Developer for Apache Spark or equivalent certification.
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
Search for further Jobs Here:
×