Senior AI Observability Engineer

Build AI observability systems at BMO in a hybrid role, using Python, telemetry, and monitoring to evaluate production models and agents.
This senior engineering role focuses on artificial intelligence observability and platform capabilities. You will trace LLM calls, monitor token usage, detect drift, and govern autonomous agent behaviors. The work also includes machine learning systems, experiments, algorithms, and specialized technical support.

Key Responsibilities:

Design systems tracking, evaluating, and debugging non-deterministic AI models
Implement Open Telemetry tracing for agent trajectories, prompts, and tool calls
Build automated evaluations for relevance, groundedness, and hallucinations
Monitor drift, latency, errors, token usage, and data quality
Define guardrails and self-remediation boundaries for autonomous agents
Requirements:
Bachelor's degree in Computer Science, Engineering, or relevant domain
Seven to ten years of Site Reliability Engineering experience
Artificial intelligence observability engineering and automation experience
Python plus SQL, LogQL, or PromQL experience
Prometheus, Grafana, or Datadog observability pipeline experience
Apply senior AI engineering expertise across observability, evaluation, tracing, monitoring, and agent governance at BMO.