Data Engineer

The ability to make a payment quickly and at a low cost is an underestimated bedrock of our economy in South Africa. Payments are the financial lifeline of every business and individual. We are on a mission to help entrepreneurs participate and prosper in the digital economy by solving their payment challenges. When you join Shop2

Shop, you join a team of people driven by a shared purpose for why we do what we do every day. The experience of collaboration across teams will be unrivalled. The feeling of belonging to a talented team of people will make this your last career stop. You will find a stimulating and challenging smorgasbord of problems to solve with the aim of supporting entrepreneurs in South Africa.

Every decision Shop2

Shop makes about merchants, risk and growth depends on data arriving on time and being right. The Data Engineer builds and runs the pipelines that make that possible.

You will design, build and maintain Shop2

Shop’s modern data platform, using Python and Dagster to develop data pipelines centred on software-defined assets. The role applies modern software engineering practices to data infrastructure, with a focus on testability, data lineage, declarative asset management and reliable data delivery.

Reporting to the Core Data Services Manager, you will work closely with data analysts, Dev Ops and business stakeholders to turn data requirements into scalable technical solutions.

Success = The people who rely on our data can trust it without checking it twice.

• You approach data infrastructure with a software engineering mindset and value clean, maintainable Python code.
• You care about testability, data lineage and declarative asset management.
• You take ownership of data quality, monitoring and the reliability of data pipelines.
• You enjoy automating deployments and connecting tools across a modern data stack.
• You work well with data analysts, Dev Ops teams and business stakeholders to translate data requirements into scalable solutions.
• You are curious about how AI tools can make you faster and better, and you check what they produce.

• Design, build and maintain robust, declarative software-defined assets and graphs in Dagster
• Use Dagster to track data lineage and state across the data stack

Success = Anyone can see where a number came from and how fresh it is.

• Write clean, efficient and maintainable Python code to extract, transform and load data from disparate sources into the data warehouse
• Use ETL or ELT processes as best fits each source
• Build data models in line with the architecture and modelling standards set for the platform

Success = Data lands in the warehouse on time, every time, without manual fixes.

• Implement rigorous data validation, testing and monitoring using tools such as Pytest and Dagster’s built-in features
• Make sure data quality issues are detected and flagged before they reach reports and dashboards

Success = Data problems are caught by our checks before someone finds them in a dashboard.

• Own the day-to-day running of the pipelines you build, including failures, reruns and backfills
• Investigate and resolve pipeline incidents, and fix the root cause so they do not recur
• Keep runbooks and documentation current so others can support the platform

Success = When a pipeline fails, it is noticed, fixed and understood quickly, and the same failure does not happen twice.

• Connect Dagster with the broader modern data stack, including dlthub, dbt, DuckDB, Athena, PostgreSQL and Terraform
• Integrate the data platform with AWS cloud infrastructure,…