AI Engineer – Agentic Systems & RLAIF

Contract (1 month 26 days)

Published 2 days ago

Python Engineering

root cause analysis

mlops

CI/CD

AI Agent Development

We are building an RLAIF (Reinforcement Learning from AI Feedback) Plane: a governed learning system that turns production evidence from Client's customer-facing agents into better agent decisions—safely, measurably, and reversibly. Today, small errors compound across interconnected agents, missing capabilities and stale business context silently produce fluent-but-wrong answers, and prompt, policy, and model changes lack one governed path to independent proof.

As an AI Engineer you will own the engineering behind that path: detecting capability and context failures before they become customer-visible, converting root-cause analysis into one-factor variants, running them through hard gates and calibrated AI judges, and promoting only what is approved, versioned, monitored, and reversible.

What You Will Do:

Build the agent runtime
- Design, build, and operate production agents and multi-agent workflows using Lang Graph, Lang Chain, and deep-agent patterns—governed, monitored, and versioned from day one.

Instrument for observability
- Implement tracing, telemetry, and failure taxonomies across agents, tools, memory, and knowledge sources so every failure has a reproducible trail back to its root cause.

Engineer the evaluation harness
- Own golden sets, offline and online evals, and AI-judge pipelines with order-swap bias control and human calibration loops; report against business outcomes, not just model metrics.

Drive harness optimization
- Use observed failures to improve prompts, tools, skills, and rules through one-change-at-a-time experiments with regression-safe release: hard gates, approval, canary, and rollback.

Separate the optimization paths
- Keep harness, contextual-policy, and model-level changes distinct and independently evaluated; build the mechanisms that detect capability and knowledge-coverage gaps and propose governed updates.

Ship on GCP
- Deploy and scale the RLAIF Plane on Google Cloud (Vertex AI, GKE/Cloud Run, Big Query, Pub/Sub) with CI/CD, IaC, and cost/latency controls.

Collaborate and lead
- Partner with architects, data scientists, SRE, and Client business stakeholders; document decisions, mentor engineers, and raise the bar on agent engineering practice.

Key Responsibilities:

Ingesting and analyzing traces

Reconstructing the sequence of events, tool calls, and business flow from traces

Creating prompt/context/routing variants

Defining and implementing evaluation dimensions

Building and calibrating AI judges

Designing scoring and reward-signal frameworks

Generating autonomous optimization recommendations

Identifying root causes such as:

Hallucinations

Misrouting

Policy/compliance issue

Missing or underperforming tools

Re-configuration opportunities (agent) for identification and recommendation and project outcomes.

What You Bring (Required):

Demonstrable experience building and running AI agents in production—shipped systems serving real users, with ownership of reliability, latency, cost, and incident response.

Hands-on experience with AI observability and evals: tracing frameworks (e.g., Lang Smith, Arize Phoenix, Open Telemetry), LLM-as-judge design, golden-set curation, and regression testing.

Deep, practical experience with Lang Graph and Lang Chain, including deep-agent patterns (planning, sub-agents, tool use, memory, human-in-the-loop, and state persistence).

Strong Python engineering skills; comfortable with async services, APIs, containers, and testing discipline in a CI/CD environment.

Working understanding of RLAIF/RLHF concepts, prompt and policy optimization, and how to isolate a single causal change in a complex…