AI Python Engineer – Chatbot Backend on-site

AI Python Engineer – Chatbot Backend (Abu Dhabi, on-site)
About the role

This is an on-site position in Abu Dhabi, UAE. We are looking for an AI Engineer to design, build, deploy, and operate production conversational AI and LLM systems for complex, multi-step customer journeys.

You will work primarily in Python and own stateful agent orchestration, persistent context, tool and API integration, MCP, RAG, evaluation, observability, testing, and containerized deployment. This is a hands-on role for an engineer who is comfortable moving beyond prototypes and supporting AI systems through the full production lifecycle.

• Design complex conversational workflows that perform multi-step search, servicing, booking, payment, and related operational tasks.
• Build stateful agent graphs with primary assistants, specialist agents, sub-agents, routing layers, and structured handoffs.
• Implement short- and long-term context management using checkpointers, cache, and persistent stores so users can resume journeys across sessions.
• Apply planning-and-execution patterns for complex requests while keeping simple interactions fast and predictable.
• Implement tool binding, structured outputs, exception handling, retries, fallbacks, and recovery paths for production reliability.
• Support asynchronous and streaming agent execution with progress updates to user interfaces.

• Build and maintain MCP clients and servers for secure access to internal tools, APIs, and data sources.
• Develop production Python services that integrate with enterprise REST APIs for search, validation, servicing, cart, payment, and related workflows.
• Implement secure authentication and authorization flows using OAuth 2.0, OIDC, managed identities, and service credentials.
• Define typed tool schemas, validate inputs and outputs, and handle partial failures across multi-step executions.
• Deliver streaming tool results through Web Sockets, Server-Sent Events, or equivalent real-time interfaces.

• Create focused RAG and FAQ workflows with reliable retrieval, grounding, and response traceability.
• Work with SQL, No
  
  SQL, vector databases, caches, and persistent checkpoints across transactional and conversational workloads.
• Implement data ingestion, chunking, vector representations, semantic search, routing, and retrieval optimization where appropriate.
• Design robust state models and validation layers for agent memory, tool results, and application configuration.

• Build automated evaluation pipelines using scenario-based tests, LLM-as-judge methods, regression suites, and synthetic-user simulations.
• Instrument agent workflows for traces, structured logs, prompt versions, tool activity, errors, quality signals, latency, and cost.
• Create repeatable conversation-review and feedback loops that support prompt improvement and model comparison.
• Develop concurrency-safe load tests for agent endpoints and investigate bottlenecks across models, tools, databases, and streaming layers.
• Troubleshoot hallucinations, routing failures, state corruption, retrieval quality, latency, and integration defects in production.

• Build typed, asynchronous, production-grade Python services with clear packaging, configuration, and dependency management.
• Write unit, integration, contract, regression, and asynchronous tests for AI workflows and external integrations.
• Create container images and support Kubernetes-based deployment across development, testing, and production environments.
• Contribute to CI/CD and Git Ops workflows, including security scanning, smoke tests, image publishing, and controlled releases.
• Implement secrets management, health endpoints, structured logging, telemetry, graceful shutdown, and operational runbooks.
• Work closely with AI, backend, frontend, platform, security, and product specialists while owning technical deliverables end-to-end.

• Define contracts for streaming responses, tool progress, errors, and structured widgets used by conversational interfaces.
• Collaborate with frontend engineers on Type Script-based chat components and real-time user experiences.
• Ensure backend streaming behavior remains compatible with Web Socket, SSE, and component-based UI patterns.

• Ability to investigate ambiguous problems, communicate clearly in…