Senior Data Engineer Databricks Exp Remote

Job Title: Senior Databricks Engineer

Location: 100% Remote

Interview Process: Video (2 Rounds)

Job Description:
Job Description Lead/Senior Databricks Data Engineer

Position:
Lead Databricks Data Engineer / Databricks Architect

Experience: 10+ Years

Location: Remote

We are looking for an experienced Lead Databricks Data Engineer / Databricks Architect with 10+ years of overall experience in Data Engineering and strong hands-on expertise in Databricks, Apache Spark, PySpark, SQL, Delta Lake, and cloud-based data platforms
.

The candidate will be responsible for designing and implementing scalable Lakehouse architectures
, enterprise data pipelines, data integration solutions, governance frameworks, and high-performance analytics platforms using Databricks.

• Design and develop scalable data engineering solutions using Databricks and Lakehouse architecture
  .

• Build robust ETL/ELT pipelines using PySpark, Spark SQL, Python, and SQL
  .

• Design and implement Bronze, Silver, and Gold/Medallion architecture
  .

• Develop and optimize Delta Lake tables, including MERGE, schema evolution, Change Data Feed, and incremental processing.

• Build batch and real-time/streaming pipelines using Structured Streaming, Auto Loader, and Lakeflow
  .

• Develop and manage Databricks Jobs/Workflows for pipeline orchestration, scheduling, dependencies, retries, and monitoring.

• Implement enterprise data governance using Unity Catalog
  , including access control, data lineage, auditing, catalogs, schemas, and external locations. Unity Catalog provides centralized governance, access control, lineage, and auditing across Databricks data and AI assets. ()

• Perform Spark and Databricks performance tuning
  , including cluster configuration, partitioning, caching, query optimization, Photon, and workload optimization.

• Design data models supporting Data Warehousing, BI, Analytics, and AI/ML workloads
  .

• Integrate Databricks with cloud platforms such as AWS, Azure, or Google Cloud Platform
  .

• Work with cloud services such as AWS S3, Azure ADLS Gen2, Azure Data Factory, AWS Glue, Synapse, Event Hubs/Kafka/Kinesis
  , as applicable.

• Implement CI/CD pipelines using Git, Azure Dev Ops/Git Hub/Jenkins and Databricks deployment capabilities.

• Work with Terraform/IaC for infrastructure provisioning and automation.

• Troubleshoot production pipeline failures, performance issues, data-quality problems, and Spark/cluster issues.

• Establish data quality, monitoring, logging, and observability practices.

• Provide technical leadership, code reviews, architecture guidance, and mentorship to junior/mid-level engineers.

• Collaborate with Data Architects, Data Scientists, Business Analysts, Dev Ops teams, and application teams.

• Databricks Lakehouse Platform

• Delta Lake

• Unity Catalog

• Databricks Workflows/Jobs

• Lakeflow / Delta Live Tables

• Auto Loader

• Databricks SQL

• Databricks notebooks

• Databricks Asset Bundles

• Photon

• Cluster/workload optimization

• Apache Spark

• Py Spark

• Spark SQL

• Structured Streaming

• Kafka

• Batch and real-time data processing

• Python

• SQL

• Py Spark

• Scala good to have

• AWS: S3, Glue, EMR, Lambda, Redshift, IAM, Kinesis

• Azure: ADLS Gen2, ADF, Synapse, Azure Dev Ops, Event Hubs, Key Vault

• Google Cloud Platform: GCS, Big Query, Dataflow, Pub/Sub

• ETL/ELT

• Data Warehousing

• Dimensional Modeling

• Data Lake/Lakehouse

• Medallion Architecture

• CDC

• Data Quality

• Data Governance

• Metadata and Data Lineage

• Git

• Azure Dev Ops / Git Hub

• Jenkins

• Terraform

• CI/CD automation

• Infrastructure as Code

• MLflow

• Databricks Machine Learning

• Feature Store

• Mosaic AI / GenAI

• dbt

• Apache Airflow

• Power BI / Tableau

• Delta Sharing

• Lakehouse Federation

• Liquid Clustering

• Data security and PII masking

MLflow is particularly useful if the role touches ML/AI, as Databricks supports model tracking, lifecycle management, and deployment workflows alongside governed data. ()

• Bachelor's degree in Computer Science, Engineering, Information Technology, or related field.

• 10+ years of experience in Data Engineering / Big Data / Analytics.

• 4+ years of hands-on Databricks experience preferred.

• Strong experience designing enterprise-scale data platforms.

• Demonstrated experience leading technical projects and mentoring engineers.

• Strong communication and stakeholder-management skills.