Lead Software Engineer in Test – Agentic AI; Remote

Lead Software Engineer in Test – Agentic AI (Remote - US)

Focuses on testing and validating agentic AI, LLM workflows and RAG pipelines—so it's about building and ensuring AI-driven systems work; leans into rapid AI iteration and AI dev tooling.

Lead quality engineering for agentic AI products, designing scalable automation frameworks and validating complex LLM workflows, RAG pipelines, and distributed orchestration systems across UI, API, backend, and AI layers. Drive testing strategy, CI/CD integration, and mentor QA engineers to ensure resilient, performant AI-driven applications.

The Lead Software Engineer in Test – Agentic AI will lead quality engineering strategy and execution for AI-powered applications and distributed systems. The role focuses on designing scalable automation frameworks, validating agentic AI behaviors, and ensuring high-quality delivery across UI, API, backend, orchestration, and cloud-native layers.

• Design, develop, and maintain scalable automation frameworks and regression test suites for UI, API, backend, and AI-driven applications.
• Lead testing initiatives for Agentic AI systems including LLM workflows, RAG pipelines, autonomous agents, and distributed orchestration platforms.
• Validate complex agent behaviors such as multi‑agent coordination, tool calling, prompt chaining, memory retention, conversational workflows, and fallback/retry mechanisms.
• Develop and execute functional, integration, regression, end‑to‑end, and performance testing strategies in cloud‑native and distributed environments.
• Collaborate with Software Engineers, Product Managers, and Dev Ops to integrate testing throughout the SDLC and CI/CD pipelines.
• Build reusable automation solutions for APIs, orchestration layers, asynchronous workflows, enterprise integrations, and AI‑driven business processes.
• Design and maintain test environments, CI/CD integrations, cloud execution platforms, and automation pipelines to improve efficiency and reliability.
• Establish best practices for code quality, maintainability, scalability, dependency injection, observability, and testability.
• Lead validation of AI workflow resiliency, distributed system reliability, data consistency, and orchestration performance under failure and edge‑case conditions.
• Mentor QA engineers and develop scalable QA standards, automation strategies, reporting practices, and continuous improvement initiatives.

• Bachelor’s degree in Computer Science, MIS, Engineering, or a related technical field.
• 8+ years of experience in Software Development or SDET/SET roles.
• Strong experience with automation tools such as Playwright, Cypress, Postman, K6, or equivalent frameworks.
• 2+ years testing AI‑powered applications, LLM systems, Agentic AI platforms, orchestration frameworks, or autonomous workflows.
• Hands‑on experience validating agent orchestration workflows, conversational AI applications, multi‑step reasoning systems, and AI‑driven business processes.
• Experience testing AI orchestration frameworks such as Lang Graph, CrewAI, Auto Gen, Semantic Kernel, or similar technologies.
• Expertise in API testing, backend integration testing, UI automation, and end‑to‑end workflow validation.
• Solid understanding of automation architecture, design patterns, dependency injection, and inversion of control.
• Experience with cloud‑native and containerized environments (Azure, AWS, GCP, Docker, Kubernetes) and integrating automated tests into CI/CD pipelines.
• Experience validating asynchronous, event‑driven, and distributed systems with emphasis on resiliency and fault tolerance.
• Strong analytical, debugging, troubleshooting, and problem‑solving skills.
• Excellent organizational, communication, and stakeholder…