GenAI Platform Engineering Lead

Manages the activities of Engineering Team Leaders, Engineering Supervisors and/or engineering units responsible for the reliability, observability and operational readiness of the Bank’s AI platform. Provides day-to-day direction for the teams and applications in alignment with departmental goals and the needs of the clients they support.

Serves as the technical lead for AI operations and service assurance, with accountability for the operational layer of the AI platform. Works closely with the Platform Engineering Manager, who owns the broader platform, to ensure production systems are secure, resilient, observable, cost-effective and operationally ready. Oversees incident response practices, runbooks, monitoring, infrastructure automation and production support.

Responsible for managing client relationships and expectations, prioritizing the project queue and achieving individual and organizational objectives at minimum cost.

Lead the reliability, observability and operational readiness of the AI platform, including production monitoring, incident response, service assurance, infrastructure automation and operational controls.

Establish and maintain operational runbooks, escalation procedures, service-level indicators, service-level objectives and incident response practices.

Measure operational performance through platform availability, incident response time, mean time to recovery, model latency, token consumption and infrastructure cost.

Provide leadership during production incidents. Coordinate troubleshooting, communication, escalation, root-cause analysis and corrective actions.

Build and maintain observability pipelines using Open Telemetry, Prometheus, Grafana, Azure Monitor, Log Analytics and/or comparable technologies.

Oversee the deployment and operation of Azure infrastructure and services, including Azure API Management, Azure Monitor, Log Analytics, Microsoft Entra  infrastructure managed through Terraform.

Promote infrastructure-as-code and automated deployment practices using Terraform, Git Hub Actions, Git Lab CI and/or comparable tools. Ensure platform changes are deployed through controlled pipelines rather than manual processes.

Drive automation of recurring operational activities using Python, Bash, Power Shell and other appropriate scripting technologies.

Oversee AI-specific operational capabilities, which may include token cost tracking, model latency monitoring, provider failover, caller-level rate limiting, prompt logging, personally identifiable information interception and infrastructure-layer content filtering.

Partner with cybersecurity and risk teams to support Security Information and Event Management integrations and security event feeds from application infrastructure.

Manage and participate in consultations with client management to analyze short-range business requirements and recommend innovations that anticipate the future impact of changing business and technology needs. Build and maintain positive client relationships.

Monitor technology direction, industry trends and vendor applications related to AI platforms, site reliability engineering, cloud infrastructure, observability and service assurance.

Research and initiate changes to existing processes, technologies and operating models when necessary.

Lead vendor and product analysis and provide recommendations.

Oversee application development support, testing efforts, technology infrastructure, project management and other assigned technology domains.

Serve as a subject matter expert for AI platform operations, production reliability, observability and service assurance.

Build rapport across the organization and maintain a professional level of communication and cooperation with technology, business, risk, cybersecurity and vendor partners.

Maintain relationships with vendors and professional organizations.

Direct team activities, assign personnel to projects and provide technical and operational guidance.

Ensure schedules and commitments are completed. Lead short-term staffing and capacity planning.

Implement technology consistent with Division standards and long-range plans. Ensure adherence to Department and Technology standards and procedures, including documentation, audit trail and change management requirements.

Translate business and operational requirements as needed to assist staff in preparing detailed specifications for system enhancements.

Evaluate and manage recommended designs based on business, operational and technology requirements. Identify, communicate and resolve issues and concerns.

Manage project plans and coordinate major project and production-readiness activities. Remain current on work outside the team that may affect the team, platform or client environment.

Develop and manage multiple cost center budgets, including cloud infrastructure and AI platform operating costs.

Recommend and implement policies and procedures that improve the performance, reliability and effectiveness of the…