Sr Data Platform Engineer Job Abu Dhabi UAE/Dubai,Software Development

This is a Cactus International job and is located in Abu Dhabi, UAE.

Hybrid: 4 days in office; 1 day working from home

Job Summary

The Data Platform Lead serves as the hands-on technical owner of the company's Azure Databricks platform. This role combines platform engineering, data engineering leadership, governance implementation, production operations, and people leadership.

The successful candidate will build, operate, secure, and continuously improve the enterprise. The role owns Databricks platform standards, pipeline architecture, data observability, access governance, performance, cost management, and operational readiness.

This is a highly hands-on leadership role. Approximately 60-70% of the position will be spent designing, building, troubleshooting, configuring, and optimizing the platform and its pipelines. The remaining time will focus on mentoring engineers, reviewing designs and code, establishing practical standards, supporting delivery, and helping shape future data, analytics, AI, data product, and application product capabilities.

What Success Looks Like

The Databricks platform is secure, governed, observable, cost-conscious, and ready to support business-critical workloads.
Production pipelines are reliable, monitored, documented, recoverable, and owned.
Data Engineers receive strong technical direction, design support, coaching, and clear engineering standards.
Certified datasets and reusable data products accelerate reporting, automation, AI, and digital product delivery.
Leadership has a trusted technical partner who can translate priorities into practical platform and engineering decisions.

Essential Functions,

Roles and Responsibilities Databricks Platform Ownership and Governance

Own the technical configuration and ongoing evolution of the Azure Databricks platform, including workspace and environment structure, catalogs, schemas, storage, compute, and workload separation.
Implement and maintain practical Unity Catalog standards for access control, lineage, data classification, ownership, catalog and schema organization, and least-privilege access.
Define and maintain Bronze, Silver, and Gold layer standards, including naming conventions, table ownership, audit fields, refresh patterns, lifecycle expectations, and production-readiness criteria.
Partner with cybersecurity, infrastructure, and Dev Ops teams on identity, networking, secrets management, private connectivity, monitoring, compliance, and deployment controls.
Establish and maintain cluster policies, compute standards, job standards, cost controls, capacity practices, and platform usage monitoring.
Maintain clear technical standards and guardrails without creating unnecessary process or slowing delivery.

Data Engineering and Pipeline Delivery

Design, build, troubleshoot, and optimize production-grade pipelines using Azure Databricks, Apache Spark, Python/PySpark, SQL, Delta Lake, and approved orchestration patterns.
Lead ingestion from ERP systems, PostgreSQL and other application databases, SaaS platforms, APIs, files, and enterprise data sources into the Lakehouse.
Define reusable engineering patterns for full loads, incremental loads, change data capture where appropriate, schema evolution, reprocessing, error handling, logging, reconciliation, and recovery.
Ensure production pipelines include source-to-target mapping, ownership, data quality rules, monitoring, alerting, operational documentation, and support handover.
Perform technical design reviews, code reviews, performance tuning, and hands-on problem solving for complex or high-risk pipelines.
Review vendor and contractor deliverables for maintainability, security, performance, documentation, supportability, and production readiness.

Platform Operations,…