Data Lead​/Architect; Databricks

Position: Data Lead/Architect (Databricks)
Role:
Data Lead/Architect (Databricks)
Location:
Remote
Duration: 6+ months

Primary Skill Required for the Role:
Databricks
Level Required for Primary Skill:
Advanced (6-9 years experience)

Responsibilities:
§ Design, develop and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks
§ Write and tune PySpark / Spark SQL notebooks for large-scale data transformation
§ Architect end-to-end data solutions across dev → UAT → prod environments using Unity Catalog
§ Lead and drive design discussions with client architects and other counterparts
§ Collaborate with different teams on data contracts and schema agreements
§ Lead design and optimization of high-volume data pipeline.
§ Define and enforce data engineering standards — naming conventions, partitioning strategies, cluster configurations, Spark tuning
§ Drive performance optimization — AQE tuning, liquid clustering, broadcast joins, shuffle partition management
§ Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies
§ Conduct root cause analysis on production incidents and implement permanent fixes
§ Mentor junior and mid-level engineers through code reviews and pair programming
§ Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs)

Mandatory Skills

• Python, PySpark, Spark SQL, SQL Server

• Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor)

• Databricks (Delta Lake, Unity Catalog, Workflows)

• Apache Airflow

• Git / Azure Dev Ops

• Deep Spark internals (DAG optimization, spill analysis, skew handling)

• Delta Lake advanced features (time travel, deletion vectors, predictive I/O)

• Unity Catalog governance (row/column security, external locations, system tables)

• IaC — Terraform, Azure ARM templates