Sr. Machine Learning Ops Engineer

POSITION PURPOSE

The Senior ML Ops Engineer leads the design and maintenance of scalable, secure infrastructure for ML model deployment, lifecycle management, and Generative AI enablement. This role is responsible for building and operating the firm's ML Ops platform on Databricks, with a strategic focus on product ionizing GenAI/LLM solutions including Retrieval-Augmented Generation (RAG) systems and vector database implementations.

The Senior ML Ops Engineer ensures models transition from development to production while meeting regulatory and compliance standards. This role collaborates closely with Data Science, Platform Engineering, Information Architecture, and business vertical teams (Fund Accounting, Investor Relations, Investments) to accelerate ML-driven insights, enhance model accuracy, and govern the ML/AI ecosystem.

Beyond technical execution, the role defines ML Ops strategy and architecture, addressing the "last mile" challenge of AI value realization by automating and scaling ML models and GenAI applications as tangible business assets. This role serves as the key pillar that enhances efficiency, boosts model accuracy, accelerates time-to-market for AI solutions, and ensures the scalability and robust governance of machine learning and generative AI initiatives.

• Lead the design, implementation, and ongoing maintenance of scalable ML infrastructure on Databricks, including ML flow for experiment tracking, model registry, and model serving endpoints.
• Oversee the development of the ML Ops platform and automated pipelines for deploying, monitoring, and maintaining models within production environments.
• Implement robust solutions for model versioning, systematic retraining, and comprehensive artifact management using Databricks Unity Catalog for ML governance.
• Design and manage Databricks Feature Store for consistent feature engineering across training and inference pipelines.

• Architect and implement Retrieval-Augmented Generation (RAG) systems for document Q&A, enabling business teams to query fund documents, investor letters, and market research.
• Design, deploy, and manage vector database solutions (Databricks Vector Search, Pinecone, or similar) for semantic search and retrieval across enterprise documents.
• Lead LLM fine-tuning and customization initiatives, training models like Claude or open-source alternatives with CIM proprietary data while ensuring data privacy and compliance.
• Develop and optimize document processing pipelines including PDF parsing, chunking strategies, and embedding generation for RAG applications.
• Implement prompt engineering best practices and LLM evaluation frameworks to ensure output quality, relevance, and factual accuracy.
• Build guardrails and safety measures for GenAI applications, including hallucination detection, output validation, and source attribution.

• Design and implement extensive automation across the ML workflow, covering model training, testing, validation, and deployment using Databricks Workflows and Asset Bundles.
• Set up robust CI/CD pipelines for both traditional ML models and GenAI applications, leveraging Git Hub Actions, Azure Dev Ops, or similar tools.
• Automate complex data and model workflows utilizing orchestration tools such as Airflow, Prefect, or Databricks Workflows.

• Implement comprehensive monitoring and alerting systems for real-time tracking of model performance, data quality, and GenAI output quality.
• Utilize specialized tools (Evidently AI, Why Labs, Prometheus/Grafana) to proactively detect model drift, data quality anomalies, and RAG retrieval degradation.
• Develop evaluation frameworks for GenAI applications including relevance scoring, faithfulness metrics, and human feedback loops.
• Troubleshoot issues within production environments, including debugging model deployment failures, RAG retrieval issues, and LLM response quality problems.

• Build and maintain sophisticated feature stores on Databricks, ensuring precise alignment between training and inference data pipelines.
• Collaborate with data engineers and information architects to build robust ETL pipelines that feed into the Databricks Lakehouse.
• Design embedding pipelines and vector index management strategies for RAG applications, including incremental updates and versioning.

• Integrate robust security measures directly into ML Ops and GenAI pipelines, including access controls via Unity Catalog and data encryption.
• Implement Trustworthy AI guardrails addressing bias detection, explainability, prompt injection prevention, and responsible AI practices.
• Ensure GenAI applications handling sensitive fund and investor data comply with regulatory requirements and internal policies.
• Collaborate with Legal and Compliance to establish AI governance policies and audit trails for model decisions.