Data Engineer

Lead the migration from Glue/Redshift to Databricks

10+ years of professional data engineering experience, with at least 2+ years in a tech‑lead or lead engineer capacity.

5-6 years of experience in each of the following areas:

• Deep, hands‑on experience with Apache Spark (PySpark and Spark SQL), including performance tuning (partitioning, shuffles, skew, caching, file sizing).
• Hands‑on production experience with Databricks — Delta Lake, Unity Catalog, Workflows, and either Delta Live Tables or a comparable declarative pipeline framework.
• Robust production experience with AWS Glue (jobs, crawlers, Data Catalog, triggers) and Amazon Redshift (including Redshift Spectrum, workload management, distribution and sort keys, performance tuning, and Redshift SQL / stored procedures).
• Proven experience leading at least one substantial data platform migration or re‑platforming effort end‑to‑end.
• Expert SQL skills and strong software‑engineering fundamentals:
  Python, version control, testing, and CI/CD.
• Deep knowledge of AWS data services (S3, IAM, KMS, VPC, Lambda, Step Functions, MSK/Kinesis, Cloud Watch).
• Strong understanding of data modeling for analytics (dimensional, Data Vault, or medallion patterns) and lakehouse architecture.
• Excellent communication skills; able to translate between executive stakeholders and engineers, and to write clear design docs and migration plans.