Lead Data Engineer

Responsibilities

• Lead modernization of legacy data platforms, migrating ETL/ELT and ingestion frameworks to cloud (Azure/AWS).
• Own end-to-end data solution architecture, ensuring scalability, security, and business alignment.
• Provide technical leadership, mentoring teams and enforcing best practices.
• Drive decision-making in ambiguous environments, proactively mitigating risks and constraints.
• Implement data governance, security, and compliance frameworks (PII, regulatory standards).
• Collaborate with stakeholders to translate business needs into scalable data solutions.
• Architect and integrate data from multiple sources into high-performing cloud platforms.
• Design and optimize ETL/ELT pipelines using ADF, Databricks, Spark, and SSIS.
• Partner with analytics and BI teams to support reporting and advanced analytics use cases.
• Ensure data quality through profiling, validation, and monitoring processes.
• Establish CI/CD and Dev Ops practices for data engineering workflows.
• Monitor and optimize pipeline performance, scalability, and cost efficiency.
• Lead testing (unit, system, integration) and maintain comprehensive documentation.
• Drive continuous improvement and innovation across tools, frameworks, and architecture.

• 12+ years of experience in designing and implementing data warehouse and analytics solutions (on-premise and cloud).
• Expertise in data warehousing concepts (ETL/ELT, data quality management, privacy/security, MDM) with hands-on experience using ADF, Data Factory, SSIS, and related tools.
• Experience with cloud data and cloud-native data lakes/warehouses. Microsoft Azure services (Fabric Lakehouse, ADF, Data Factory, Synapse, etc.).
• Experience in Python, Scala, or Java for use with distributed processing and analytics, such as Spark.
• Familiarity with CI/CD practices and tools such as Azure Dev Ops, Git, or Jenkins.