Senior Data Engineering Lead​/Data Architect – Azure Databricks & Unity Catalog Onsite Charlottee

Senior Data Engineering Lead / Data Architect

We are seeking a highly experienced Senior Data Engineering Lead / Data Architect with deep expertise in Azure Databricks, Unity Catalog, and modern cloud-based data platforms. This role requires a hands-on technical leader who can design scalable data architectures, build enterprise-grade data pipelines, and drive best practices in data governance, security, and performance optimization. The ideal candidate will possess strong experience in Azure cloud services, distributed data processing, data lakehouse implementations, and large-scale data transformation frameworks.

This role involves close collaboration with architects, business stakeholders, and engineering teams to deliver robust analytics and reporting solutions.

• Design and implement scalable, secure, and high-performance data architectures on Azure.
• Develop and optimize ETL/ELT pipelines using Azure Databricks and PySpark.
• Build enterprise-grade data ingestion frameworks for structured, semi-structured, and unstructured data sources.
• Implement data transformation, cleansing, enrichment, and validation processes.
• Design and manage Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
• Establish reusable frameworks, notebooks, workflows, and shared data engineering standards.

• Lead implementation and administration of Unity Catalog for centralized governance.
• Define and enforce data access controls, security policies, lineage tracking, and metadata management.
• Collaborate with security and compliance teams to ensure adherence to organizational standards.
• Implement data governance best practices across enterprise data assets.

• Build and orchestrate data workflows using Azure Data Factory (ADF).
• Integrate Azure Databricks with Azure services including:
  • Azure Data Lake Storage Gen2 (ADLS)
  • Azure SQL Database
  • Azure Synapse Analytics
  • Azure Event Hub
  • Azure Key Vault
• Optimize Databricks clusters, autoscaling configurations, and compute resources for performance and cost efficiency.
• Implement CI/CD processes using Git and Azure Dev Ops.

• Tune and optimize Spark workloads for scalability and efficiency.
• Monitor and troubleshoot production data pipelines and platform issues.
• Conduct root cause analysis and implement preventive measures.
• Establish coding standards, code review processes, and operational best practices.

• Provide technical leadership and mentorship to data engineering teams.
• Collaborate with solution architects, analysts, and business stakeholders to translate business requirements into technical solutions.
• Create technical documentation, architecture diagrams, design specifications, and operational runbooks.
• Participate in Agile/Scrum ceremonies and contribute to delivery planning.

• 10+ years of experience in Data Engineering, Data Architecture, or related roles.
• Strong hands-on experience with Azure Databricks, including notebooks, jobs, workflows, and Delta Lake.
• Expertise in Unity Catalog implementation, governance, and security management.
• Advanced proficiency in PySpark, Spark SQL, performance tuning, and debugging.
• Strong experience with Azure Data Factory (ADF), including pipelines, triggers, and integration runtimes.
• Hands-on experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• Strong SQL development and query optimization skills.
• Experience with Git-based source control and Azure Dev Ops CI/CD pipelines.
• Deep understanding of Lakehouse Architecture and modern data platform design patterns.
• Experience designing and implementing enterprise-scale data solutions.

• Experience with Azure Synapse Analytics.
• Knowledge of Azure Event Hub, Kafka, and streaming architectures.
• Experience with Structured Streaming and real-time data processing.
• Familiarity with Azure Functions and Databricks REST APIs.
• Strong understanding of dimensional modeling and data warehousing concepts.
• Experience implementing data quality, metadata management, and data governance frameworks.
• Exposure to financial services, banking, or highly regulated industries is preferred.

• Excellent leadership and stakeholder management skills.
• Strong analytical and problem-solving capabilities.
• Ability to work independently and drive technical initiatives.
• Excellent verbal and written communication skills.
• Proven ability to collaborate across cross-functional teams in Agile environments.
• Strong focus on quality, scalability, and continuous improvement.