Lead Data Engineer; Hands

Location: Cary, NC — On-site / Hybrid
Employment Type: Full-Time
Experience

Required:

12–18 years
Eligibility: U.S. Citizens only
About the Engagement
This role supports the development of a centralized,
AI-first enterprise Data Hub for a global insurance and financial services organization, built on Azure Databricks
.
The platform ingests 150+ inbound data feeds
, distributes data to 35+ downstream systems
, and follows a modern medallion architecture (Bronze / Silver / Gold).
AI capabilities are embedded throughout the platform from day one, including ingestion, canonical mapping, data quality, reconciliation, anomaly detection, and business-user access.
This is a senior, hands-on technical leadership role
. The Lead Data Engineer will own the end-to-end technical design of both the data and AI layers, establish reference implementations for the engineering team, and continue to write and review production-grade Python, Scala, and PySpark code every week
.
Candidates who have not written or reviewed production code within the past year will not be considered.
Key Responsibilities
Data Platform Architecture & Engineering

• Own the enterprise lakehouse architecture, including:
  • Bronze / Silver / Gold data contracts
  • ADLS Gen2 zone design
  • Delta Lake table architecture
  • Partitioning strategies
  • Schema evolution
  • Retention policies
• Design and implement scalable, production-grade data engineering patterns for enterprise workloads.

• Build metadata-driven and parameterized ingestion frameworks supporting:
  • Batch files
  • Database extracts
  • CDC feeds
  • Real-time and streaming workloads
• Work extensively with:
  • Azure Event Hubs
  • Kafka
  • Spark Structured Streaming

• Develop canonical PySpark and Scala Spark implementations.
• Establish engineering, coding, and automated testing standards.
• Conduct code and pull-request reviews.
• Troubleshoot production incidents and Spark workload failures.
• Optimize Spark clusters, workloads, and associated cloud costs.
• Define and enforce engineering guardrails for performance, scalability, and reliability.

• Implement CI/CD pipelines for Databricks and Azure Data Factory using:
  • Azure Dev Ops
  • Databricks Asset Bundles
  • Terraform
• Establish observability using:
  • Azure Monitor
  • Log Analytics
• Drive automated testing and repeatable deployment practices across environments.

• Design AI-assisted ingestion and canonical mapping solutions.
• Build capabilities for:
  • Automated bridge-document generation
  • DML generation
  • Canonical table-definition generation
  • AI-assisted source-to-canonical mapping
• Implement appropriate human-review and approval gates for AI-generated mappings and transformations.

• Build AI/ML-based solutions for:
  • Data drift detection
  • Schema drift detection
  • Volume anomaly detection
  • Reconciliation failures
  • Automated reconciliation
• Develop privacy-preserving synthetic test-data approaches.

• Help design the enterprise semantic layer and knowledge graph.
• Build GPT-powered conversational data-access capabilities, including:
  • Text-to-SQL
  • Semantic-layer retrieval
  • RAG-based access patterns
• Enforce row-level and column-level security within AI-powered data-access solutions.

• Implement governance using Databricks Unity Catalog
  , including:
  • Lineage
  • Access control
  • PII standards
  • Data discovery and governance
• Participate in and lead:
  • Architecture Review Boards
  • AI governance forums
  • Design reviews
• Mentor engineers and establish strong technical documentation standards.
• Present and defend architecture decisions to both technical and non-technical stakeholders.

• 12–18 years of total experience in data engineering,…