Principal Databricks Data Engineer

Principal Databricks Data Engineer

Experience

Required:

12–18 Years

Key Requirements

• 12–18 years of overall Data Engineering experience

• 8+ years of experience with enterprise Data Warehouse and Data Lake platforms

• 5+ years of hands-on experience with Databricks and Apache Spark at scale

• Strong experience modernizing legacy Cloudera platforms including:

• CDH/CDP

• Hive

• HBase

• Impala

• Spark

• Modernize Cloudera platforms to Databricks Lakehouse architecture

• Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake

• Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines

• Ensure data reconciliation, audit integrity, and consistency during migration

• Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures

• Implement layered data architecture including:

• Raw / Landing Layer

• Curated / Conformed Layer

• Semantic / Consumption Layer

• Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures

• Strong experience with finance and risk data models including:

• General Ledger

• Sub-ledger

• Financial Hierarchies

• Credit Risk

• Liquidity Risk

• Market Risk

• Enable reporting capabilities including:

• Aggregation

• Drill-down

• Drill-back

• Build and manage semantic and consumption layers for BI, reporting, and analytics

• Define business metrics, dimensions, hierarchies, and KPIs

• Experience with:

• Databricks SQL

• Delta Tables

• dbt or similar frameworks

• Develop and optimize large-scale data pipelines using:

• Py Spark

• Spark SQL

• Delta Lake

• Implement Medallion Architecture including:

• Bronze Layer

• Silver Layer

• Gold Layer

• Optimize workloads using:

• Z-ORDER

• OPTIMIZE

• Caching

• Cluster Configuration Tuning

• Implement:

• Data Governance

• Data Quality Frameworks

• Reconciliation Controls

• Exception Handling

• Establish data lineage and metadata management

• Ensure data security, access control, and compliance standards

• Experience with AWS or Azure cloud platforms

• Experience with CI/CD pipelines using:

• Git

• Terraform

• Jenkins

• Azure Dev Ops

• Familiarity with:

• Apache Airflow

• Databricks Workflows

• Experience with dbt is an advantage

• Act as a technical authority and lead enterprise architecture decisions

• Mentor senior engineers and establish engineering standards

• Collaborate with finance, risk, analytics, and governance stakeholders

• Translate complex data structures into business-ready insights

Nice-to-Have Skills

• Experience in BFSI, Capital Markets, or Regulatory Reporting

• Exposure to:

• SAP Finance

• Oracle Financials

• SAP S/4

HANA

• Experience supporting AI/ML workloads

• Databricks and Cloud Certifications

Key Responsibilities

• Lead Cloudera to Databricks transformation initiatives

• Shape enterprise finance and risk data platforms

• Support regulatory, management, and analytical reporting systems

Essential Skills

• Databricks

• Apache Spark

• Py Spark

• Spark SQL

• Delta Lake

• Lakehouse Architecture

• Cloudera (CDH/CDP)

• Hive

• HBase

• Impala

• Data Warehouse & Data Lake

• Medallion Architecture

• Databricks SQL

• dbt

• AWS / Azure

• Airflow / Databricks Workflows

• CI/CD

• Terraform

• Jenkins

• Git

• Finance & Risk Data Models

• Enterprise Data Architecture