Lead Data Engineer - IM
Listed on 2026-07-31
-
Software Development
Data Engineering
Job Description
Lead Data Engineer - IM Must Have Technical/Functional SkillsData Bricks , EBT ,Airflow , Asset Management exp
Roles & Responsibilities Lead Data Engineer – Information Management (IM) Experience8–15 Years
Role Overview
We are looking for a highly skilled Lead Data Engineer with strong hands‑on experience in Databricks, dbt, and Python, and a proven track record of leading and executing data platform migration and modernization programs. The ideal candidate will have experience migrating legacy data warehouses, ETL platforms, and cloud/on‑prem data ecosystems to a modern Databricks Lakehouse architecture using dbt‑based transformation frameworks. This is a hands‑on leadership role requiring deep technical expertise, solution design capabilities, and the ability to mentor engineering teams while driving enterprise‑scale data modernization initiatives.
Key Responsibilities Data Engineering & Development- Design, build, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Python.
- Develop and optimize ELT/ETL processes for large‑scale data ingestion and transformation.
- Implement robust data quality, monitoring, and reconciliation frameworks.
- Build reusable data engineering components and frameworks.
- Lead migration of legacy data platforms, data warehouses, and ETL ecosystems to Databricks Lakehouse.
- Transform existing ETL workloads to modern ELT patterns using dbt.
- Analyze source environments and define migration approaches, roadmap, and execution strategy.
- Drive code conversion, performance optimization, and workload modernization efforts.
- Databricks Lakehouse Platform
- Delta Lake
- Unity Catalog
- Databricks Workflows
- Structured Streaming
- Medallion Architecture (Bronze, Silver, Gold)
Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
dbt Development- Build and maintain dbt models, macros, tests, and documentation.
- Implement incremental processing and reusable transformation frameworks.
- Establish data lineage, testing, and CI/CD best practices.
- Work closely with business and analytics teams to build trusted datasets.
- Lead a team of data engineers and developers.
- Perform code reviews and enforce engineering standards.
- Collaborate with arch itects, product owners, business analysts, and stakeholders.
- Mentor junior engineers and drive adoption of best practices.
Salary Range- $110,000-$140,000 a year
#J-18808-Ljbffr(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).