Principal Architect

Principal Architect – Machine Learning & Generative AI

In this role you will build and scale machine learning and Generative AI (GenAI) solutions from development to production. You will design and develop end‑to‑end data pipelines, feature engineering workflows, and ML/GenAI model deployment using Databricks, Apache Spark, SQL, and Python. You will act as an architect for large‑scale Data Engineering and ML/GenAI initiatives, driving architecture decisions across lakehouse platforms, distributed compute and cloud ecosystems (AWS, GCP, Azure) to ensure scalability, reliability and maintainability.

Key responsibilities include:

• Lead the design, development, and delivery of machine learning and GenAI solutions, from problem framing to production deployment and business impact realization.
• Build robust data foundations for AI by developing high‑quality, scalable ETL/ELT pipelines in Databricks and ensuring data availability, consistency and performance for downstream ML/GenAI use cases.
• Define and institutionalize evaluation, validation and governance frameworks for ML/GenAI systems, including model performance tracking, prompt evaluation, safety guardrails, hallucination mitigation and compliance.
• Partner directly with business stakeholders and product leaders to translate objectives into data‑driven AI/ML solutions and measurable value through KPIs and experimentation frameworks.
• Establish and enforce best practices in MLOps, LLMOps, Data Ops and Dev Ops, including CI/CD pipelines, Databricks workflows, monitoring, observability, reproducibility and cost optimization.
• Architect and oversee scalable cloud‑based data and AI platforms, integrating Databricks Lakehouse, Spark processing layers and cloud‑native services for unified analytics and AI workloads.
• Lead cross‑functional collaboration across data engineering, data science, platform engineering and business teams to deliver integrated, production‑grade AI solutions.
• Provide mentorship to L4 and L5 engineers in data engineering (Spark, SQL, Databricks) and AI/ML development (Python, GenAI frameworks).
• Stay at the forefront of advancements in data engineering and generative AI, driving adoption of new technologies and best practices.

• Doctorate degree and 2 years of experience, OR Master’s degree and 4 years of experience, OR Bachelor’s degree and 6 years of experience, OR Associate’s degree and 10 years of experience, OR High school diploma / GED and 12 years of experience.

• Deep expertise in machine learning, deep learning and generative AI (LLMs, transformers, embeddings, fine‑tuning techniques).
• Proven track record of leading and delivering production‑grade ML/GenAI systems end‑to‑end with measurable business impact and experience designing scalable system architectures for ML and GenAI.
• Expertise in MLOps/LLMOps ecosystems (MLflow, Kubeflow, Airflow, CI/CD, Docker, Kubernetes) and strong system design, architecture and problem‑solving skills.
• Proficiency in leveraging cloud platforms (AWS, Azure, GCP) for data engineering solutions, including cost optimization strategies.
• Strong experience mentoring and guiding junior and mid‑level engineers (L4/L5).
• Cloud certifications (AWS, Azure, or GCP) are a plus.
• Experience with big data ecosystems, Apache Spark, Hadoop and large‑scale distributed data processing.
• Expertise in building scalable data pipelines using Databricks, Spark, SQL, and Python.
• Proficiency in Python and modern ML/AI frameworks (PyTorch, Tensor Flow, Hugging Face, Lang Chain).
• Experience designing robust evaluation and validation frameworks, safety testing and monitoring.
• Experience with Retrieval‑Augmented Generation (RAG) architectures and vector databases.
• Knowledge of responsible AI practices, fairness, explainability, governance and regulatory compliance.
• Hands‑on experience with cloud‑native AI/ML services across AWS, Azure or GCP.
• Experience with Databricks Lakehouse platform for enterprise‑scale data engineering, ML, and GenAI workloads.
• Experience with workflow orchestration tools such as Apache Airflow.
• Strong experience in data modeling and performance tuning for OLAP and OLTP systems.