Data Lead​/Architect; Databricks

Data Lead/Architect (Databricks)

Location:

Remote

Duration: 6+ months

Primary Skill Required for the Role:
Databricks

Level Required for Primary Skill:
Advanced (6-9 years experience)

• Design, develop and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks
• Write and tune PySpark / Spark SQL notebooks for large-scale data transformation
• Architect end-to-end data solutions across dev → UAT → prod environments using Unity Catalog
• Lead and drive design discussions with client architects and other counterparts
• Collaborate with different teams on data contracts and schema agreements
• Lead design and optimization of high-volume data pipeline
• Define and enforce data engineering standards — naming conventions, partitioning strategies, cluster configurations, Spark tuning
• Drive performance optimization — AQE tuning, liquid clustering, broadcast joins, shuffle partition management
• Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies
• Conduct root cause analysis on production incidents and implement permanent fixes
• Mentor junior and mid-level engineers through code reviews and pair programming
• Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs)

• Python, PySpark, Spark SQL, SQL Server
• Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor)
• Databricks (Delta Lake, Unity Catalog, Workflows)
• Apache Airflow
• Git / Azure Dev Ops
• Deep Spark internals (DAG optimization, spill analysis, skew handling)
• Delta Lake advanced features (time travel, deletion vectors, predictive I/O)
• Unity Catalog governance (row/column security, external locations, system tables)
• IaC — Terraform, Azure ARM templates