Consulting Partner - Data Engineering & Modern Data Platforms
Listed on 2026-09-14
-
IT/Tech
Data Engineering, Cloud Computing: Infrastructure & Operations
Job Description
Consulting Partner - Data Engineering & Modern Data Platforms
- Databricks platform engineering.
- 15+ years of experience in data engineering, data platforms, analytics, or technology architecture.
- 10+ years of deep hands-on technical experience in data engineering and data platform technologies.
- Demonstrated experience architecting and delivering complex enterprise-scale data platforms.
- Strong hands-on experience with Spark/PySpark and SQL.
- Strong practical experience with at least one modern cloud data platform such as Databricks or Snowflake.
- Experience with modern data transformation frameworks such as dbt.
- Experience with workflow orchestration technologies such as Airflow.
- Strong understanding of distributed computing, data processing performance, and scalability.
- Experience leading large-scale data modernization or migration programs.
- Experience managing technical teams distributed across multiple locations.
- Ability to engage directly with engineers, architects, platform teams, and executive stakeholders.
- Cloud-to-cloud data platform migration.
- Migration of large PySpark estates.
- Legacy ETL modernization.
- Stored procedure modernization to dbt or modern transformation frameworks.
- Databricks platform engineering.
- Data platform governance and security.
- Large-scale data migration involving hundreds of terabytes or petabyte-scale environments.
- AI-assisted engineering and migration accelerators.
- Building reusable engineering frameworks and accelerators.
- Experience with platform reliability, Fin Ops, and performance optimization.
We are seeking a highly accomplished Consulting Partner with deep hands-on Data Engineering and Modern Data Platform expertise to lead complex enterprise data transformation programs.
This is a technical leadership role, not a purely advisory or relationship-management position. The successful candidate will combine executive consulting capabilities with strong architecture and engineering expertise and must have demonstrated hands-on experience designing, building, modernizing, and troubleshooting large-scale data platforms and pipelines.
The Consulting Partner will work with client executives to shape transformation strategies while remaining sufficiently close to technology and engineering teams to influence architecture, solve complex technical problems, review implementations, and guide critical delivery decisions.
The role requires an individual who can move seamlessly from an executive discussion with a CIO or CDO to a deep technical discussion involving Databricks, Spark/PySpark, dbt, SQL, cloud architecture, data pipelines, orchestration, performance optimization, and large-scale data migration.
Key Responsibilities- Technical Consulting & Architecture Leadership
- Serve as the senior technical advisor for complex data engineering and data platform transformation programs.
- Define enterprise data platform, lakehouse, and data engineering architectures.
- Lead architecture assessments and develop modernization strategies for legacy and cloud-based data ecosystems.
- Design scalable architectures covering data ingestion, transformation, storage, orchestration, governance, security, and consumption.
- Review and challenge architecture decisions made by engineering and solution teams.
- Establish technology standards, engineering principles, and reference architectures.
- Identify technical risks and scalability challenges early in the program lifecycle.
- Provide hands-on technical guidance for complex architecture and engineering issues.
The Consulting Partner must maintain strong hands-on technical depth and be capable of working directly with engineering teams when required.
Responsibilities Include- Reviewing and optimizing PySpark and Spark applications.
- Analyzing query and pipeline performance issues.
- Reviewing complex SQL transformations.
- Designing and reviewing dbt models, transformation frameworks, and testing strategies.
- Troubleshooting complex data processing and distributed computing issues.
- Reviewing data partitioning, clustering, file sizing, and storage optimization strategies.
Providing technical direction on Delta Lake, table design, and data lifecycle management. - Reviewing Airflow DAGs and orchestration architecture.
- Helping teams resolve critical production issues and performance bottlenecks.
- Conducting architecture and code reviews for high-risk or complex components.
The role does not require day-to-day individual…
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).