Associate Director, Lead Quality Engineer

Job Overview

Design and execute comprehensive test strategies for AI systems and models, including prompt engineering, output evaluation, bias and safety testing, and deep understanding of LLM behavior—tokenization, embeddings, attention mechanisms, and inference—to anticipate failure modes and assess quality across accuracy, tone, coherence, and bias dimensions. Apply evaluation metrics specific to generative AI and establish appropriate thresholds.

• Test AI systems integrated with RAG pipelines and knowledge bases, validating data quality, retrieval accuracy, and their impact on model outputs.
• Understand vector database mechanics, similarity‑search thresholds, embedding drift, and test edge cases such as near‑duplicate documents, sparse vs. dense embeddings, and performance under scale.
• Leverage Lang Chain and Lang Graph frameworks to read code, understand chain and graph construction, identify failure points, and create test harnesses.
• Validate integration points using Model Context Protocols, testing tool availability and error handling.
• Define and execute comprehensive test strategies across functional, regression, integration, and performance testing, establishing standards and best practices.
• Design, build, and maintain automated test suites to accelerate release cycles and improve coverage.
• Validate end‑to‑end deal workflows, ensuring data integrity through reconciliation testing and reporting.
• Coordinate regression testing for platform releases, patches, and infrastructure changes, ensuring stability and backward compatibility.
• Partner with developers, analysts, and product owners to clarify requirements, identify edge cases, and ensure testability of new features; lead defect‑triage sessions.
• Define and monitor key quality indicators, present findings to leadership, recommend improvements, mentor junior QA members, and foster a culture of quality.
• Provide production support during critical processing windows, investigate incidents, and coordinate root‑cause analysis.

• 7+ years of quality assurance or quality engineering experience, with at least 3 years in a lead or senior capacity.
• Strong domain knowledge in Capital Markets or similar asset classes.
• Bachelor’s degree in Computer Engineering, Computer Science, or equivalent professional experience.

• Hands‑on experience with test automation tools (Selenium, Robot Framework, Playwright, or similar).
• Proficiency in Java and Python with demonstrated framework implementation expertise.
• Hands‑on API automation and backend system validation experience.
• Database query development, data validation, and reconciliation testing.
• Experience with CI/CD pipelines and Dev Ops practices (Jenkins, Git Hub, or similar).

• Core understanding of LLM architecture and behavior.
• Hands‑on experience with Lang Chain and/or Lang Graph frameworks.
• Knowledge of RAG pipelines, vector databases, and agentic solutions.
• Familiarity with Model Context Protocols (MCPs) and integration testing.
• Understanding of bias, safety, and red‑team testing methodologies for AI systems.

• Passion for automation, continuous learning, and solving open‑ended problems.
• Excellent analytical, problem‑solving, and communication skills.
• Ability to manage multiple priorities in a fast‑paced, deadline‑driven environment.
• Ability to mentor and lead teams while maintaining focus on quality excellence.

• Leadership support through coaching and opportunities for development.
• Dynamics, collaboration, and high‑performing team environment.
• Opportunities to take on progressively greater accountabilities.
• Access to a variety of job opportunities across business and geographies.
• Education opportunities and industry certifications (ISTQB, CSTE) a plus.