Senior Databricks Data Architect; AWS - Hands-On Technical Leader

JOB TITLE:

Senior Databricks Data Architect (AWS)
- Hands-On Technical Leader PURPOSE OF THE JOB:

• Lead the strategic design and architectural roadmap of the Enterprise Data Platform on Databricks/AWS while remaining actively hands-on in design, prototyping, coding, and performance tuning.
• To provide high-level technical consultancy and solution design to internal business stakeholders and external partners, acting as the bridge between complex business requirements and technical implementation.
• To govern the technical quality of the platform by establishing design standards, conducting code reviews, and guiding the engineering team on best practices for Databricks Serverless and AWS integration.
• As part of the role, you will be required to provide technical assurance for all major data initiatives, ensuring that solutions are robust, cost-effective, and compliant with enterprise security standards.
• This role requires 100% onsite presence Monday through Friday and will cover core business hours based on the CST (Central Standard Time) timezone.

• Lead all architecture design and review sessions for strategic Databricks initiatives.
• Actively contribute code to critical architectural components, building complex frameworks, reusable libraries, and core ETL/ELT pipelines using Python and PySpark.
• Lead by example by prototyping Proof of Concepts (PoCs) for new features (e.g., Databricks Asset Bundles, Delta Live Tables) before handing them off for wider team implementation.
• Perform hands-on performance tuning and debugging of high-latency jobs by deep-diving into Spark UI, execution plans, and spill logs.
• Defining and enforcing coding standards, data modeling best practices (Medallion Architecture), and CI/CD protocols for the development team to follow.
• Conducting deep-dive technical reviews of designs and code produced by the engineering team to ensure efficiency, maintainability, and security.
• Designing the strategy for Unity Catalog governance, ensuring granular data access controls and compliance with data privacy regulations.
• Acting as the Technical Design Authority for the Databricks platform, making high-level decisions regarding the adoption of new Serverless features and AWS services.
• Mentoring and coaching Senior and Junior engineers, fostering a culture of technical excellence and continuous improvement.
• Own the technical relationship with key business stakeholders and translate business strategy into robust data architecture.
• Engage and collaborate with Executive Stakeholders to present technical roadmaps and demonstrate the business value of data initiatives.
• Actively seeks opportunities to develop knowledge in emerging technologies (e.g., Generative AI on Databricks, Vector Databases) and evaluate their impact on the business.
• Provide guidance to the Change Management board regarding the risk and technical impact of proposed architectural changes.
• Develop strategies to architect resilience, high availability, and disaster recovery into the data platform.
• Develop clear communication channels with C-level executives and Product Managers to ensure technical strategies align with business goals.

• Head of Data & Analytics / CIO
• Business Unit Directors & Product Managers
• Lead Data Engineers and Data Scientists
• Enterprise Security & Infrastructure Architects

• Ownership of the Technical Roadmap and Architecture for the Data Estate
• Maintain excellent relationships with all key liaisons
• Responsible for architectural decisions impacting critical data assets
• Primary Technical Advisor in the CST timezone

• Educated to degree level in Computer Science, Engineering, or equivalent (Master’s degree preferred)

• Exceptional "Customer Facing" skills: proven ability to lead workshops, manage stakeholder expectations, and present complex technical solutions to non-technical audiences.
• Minimum 8-10 years’ experience in Data Engineering/Architecture, with at least 4 years specialized in Databricks and AWS
  .
• Expert-level knowledge of Data Lakehouse Architecture, Delta Lake internals, and Spark optimization strategies.
• Proven experience in Data Modeling (Dimensional,…