GenAI Platform Engineering Lead

Manages the activities of Engineering Team Leaders, Engineering Supervisors and/or engineering units responsible for the reliability, observability and operational readiness of the Bank’s AI platform. Provides day-to-day direction for the teams and applications in alignment with departmental goals and the needs of the clients they support.

• Lead the reliability, observability and operational readiness of the AI platform, including production monitoring, incident response, service assurance, infrastructure automation and operational controls.
• Establish and maintain operational runbooks, escalation procedures, service-level indicators, service-level objectives and incident response practices.
• Measure operational performance through platform availability, incident response time, mean time to recovery, model latency, token consumption and infrastructure cost.
• Provide leadership during production incidents. Coordinate troubleshooting, communication, escalation, root-cause analysis and corrective actions.
• Build and maintain observability pipelines using Open Telemetry, Prometheus, Grafana, Azure Monitor, Log Analytics and/or comparable technologies.
• Oversee the deployment and operation of Azure infrastructure and services, including Azure API Management, Azure Monitor, Log Analytics, Microsoft Entra  infrastructure managed through Terraform.
• Promote infrastructure-as-code and automated deployment practices using Terraform, Git Hub Actions, Git Lab CI and/or comparable tools. Ensure platform changes are deployed through controlled pipelines rather than manual processes.
• Drive automation of recurring operational activities using Python, Bash, Power Shell and other appropriate scripting technologies.
• Oversee AI-specific operational capabilities, which may include token cost tracking, model latency monitoring, provider failover, caller-level rate limiting, prompt logging, personally identifiable information interception and infrastructure-layer content filtering.
• Partner with cybersecurity and risk teams to support Security Information and Event Management integrations and security event feeds from application infrastructure.
• Manage and participate in consultations with client management to analyze short-range business requirements and recommend innovations that anticipate the future impact of changing business and technology needs. Build and maintain positive client relationships.
• Monitor technology direction, industry trends and vendor applications related to AI platforms, site reliability engineering, cloud infrastructure, observability and service assurance.
• Research and initiate changes to existing processes, technologies and operating models when necessary.
• Lead vendor and product analysis and provide recommendations.
• Oversee application development support, testing efforts, technology infrastructure, project management and other assigned technology domains.
• Serve as a subject matter expert for AI platform operations, production reliability, observability and service assurance.
• Build rapport across the organization and maintain a professional level of communication and cooperation with technology, business, risk, cybersecurity and vendor partners.
• Maintain relationships with vendors and professional organizations.
• Direct team activities, assign personnel to projects and provide technical and operational guidance.
• Ensure schedules and commitments are completed. Lead short-term staffing and capacity planning.
• Implement technology consistent with Division standards and long-range plans. Ensure adherence to Department and Technology standards and procedures, including documentation, audit trail and change management requirements.
• Translate business and operational requirements as needed to assist staff in preparing detailed specifications for system enhancements.
• Evaluate and manage recommended designs based on business, operational and technology requirements. Identify, communicate and resolve issues and concerns.
• Manage project plans and coordinate major project and production-readiness activities. Remain current on work outside the team that may affect the team, platform or client environment.
• De…