Agentic Engineer

Agentic Enginner

Dubai

Production agentic AI systems, LLM orchestration, MCP/A2A tool connectivity, RAG, evaluation, observability, safety and cost optimisation

• Create dashboards:
  Agent success rate, error rate, average latency, cost per query

• Define agent workflows:
  What are the agent's goals? What tools does it need (APIs, databases, search)? What's the decision loop?
• Choose orchestration framework:
  Lang Chain, Llama Index, Azure Semantic Kernel, or custom agent loop (depends on complexity)
• Design tool interfaces:
  What functions can agents call? What parameters? What's the expected response format? (Function calling / structured output)
• Implement error handling:
  What if a tool fails? Can the agent recover? When should it elevate to human?
• Design multi-turn conversations:
  Context window management (summarize old turns, drop less relevant context); prevent infinite loops

• Evaluate models for use case:
  Latency requirements (Claude 3.5 Sonnet is fast for reasoning; GPT‑4o for multimodal), cost per 1M tokens, context window (4K vs. 200K)
• Write system prompts:
  Clear role definition, task boundaries, safety guidelines (e.g., "Never execute user code directly"; "Always verify customer identity")
• Implement few-shot prompting:
  Provide examples of desired behavior; show edge case handling
• Implement chain-of-thought:
  Ask agents to reason step‑by‑step before answering; improves accuracy on complex tasks
• Test prompt robustness:
  Does it handle jailbreak attempts? Adversarial inputs? Different languages (Arabic for UAE)?

• Define tool schemas:
  Functions agents can call (search database, fetch customer data, send email, make payment)
• Implement tool wrappers:
  Validate inputs (prevent SQL injection, large data fetches), execute safely in sandboxed environment, return structured responses
• Implement guardrails:
  Rate limit tool calls (prevent denial of service), check permissions (agent can't access customer data it doesn't own), audit logging
• Handle tool failures:
  Retry logic, fallback tools, error messages that help agent understand what went wrong
• Optimize tool calls:
  Batch calls if possible (fewer API round‑trips), cache results (same question asked multiple times)

• Implement prompt caching (Azure OpenAI, Claude Prompt Caching):
  Reuse long context (documents, code) without re‑processing; save :90% of tokens for cached portions
• Batch requests:
  If processing multiple documents/queries, batch them into single API call (if semantics allow)
• Use cheaper models for certain tasks: GPT‑4o Turbo for complex reasoning, GPT‑4o Mini for simple classification
• Implement token counting:
  Estimate costs before running agents; flag agents that are exceeding budget
• Monitor API costs:
  Track spend by agent, by model, by use case; optimize high‑cost agents
• Safety & Guardrails
• Implement input validation:
  Filter adversarial prompts, prompt injection attempts
• Implement output filtering:
  Ensure agent doesn't leak PII, corporate secrets, or biased content
• Handle refusals:
  Agent refuses to perform unsafe action; provide clear explanation to user ("I can't modify payment history without approval")
• Audit logging:
  Log all agent interactions (user query, agent reasoning, tools called, final response) for compliance/debugging
• Implement human‑in‑the‑loop:
  For sensitive actions (deleting data, transferring funds), require human approval before execution

• Monitor agent latency:
  Time from user query to response; breakdown by LLM call, tool execution, parsing
• Track accuracy metrics:
  If possible, grade agent outputs (correct answer? safe? complete?); compare model versions
• Implement observability:
  Log structured data (timestamps, tokens used, tool calls, errors) to Azure Application Insights or ELK
• Create dashboards:
  Agent success rate, error rate, average latency, cost per query
• Debug failures:
  Detailed logs of reasoning steps, function inputs/outputs; be able to replay conversations for investigation

• Work with Backend engineers:
  Define API contracts for agent function calling; ensure APIs return agent‑friendly…