AI QA Engineer, AVP

Quality Assurance Engineer – AI Testing, Charles River Development / State Street Alpha

Why this role is important to us

The team you will be joining is responsible for advancing quality engineering for AI-enabled capabilities within the CRIMS / State Street Alpha product suite. As a Quality Assurance Engineer focused on AI testing, you will work within a scrum team to design and implement scalable test frameworks, validate AI-driven product behavior, and strengthen the team’s ability to test intelligent features with speed, rigor, and confidence.

This role is ideal for a mid-level QA professional who combines strong software testing fundamentals with hands‑on experience applying AI in the test process and testing AI functionality embedded in enterprise products.

• Partner closely with product managers, software engineers, architects, and scrum team members to ensure AI-enabled capabilities are testable, reliable, and aligned to business and client needs.
• Design and build reusable QA frameworks and test approaches tailored to validating AI-driven functionality, including model-integrated workflows, intelligent recommendations, generated outputs, and workflow automation.
• Define test strategies that address both traditional application quality and AI-specific quality dimensions such as consistency, accuracy, robustness, explainability, and edge-case handling.
• Contribute to sprint planning, story refinement, and acceptance criteria development to ensure quality considerations are embedded early in the software development lifecycle.
• Help evolve the team’s quality engineering practices by introducing practical uses of AI to improve test design, coverage analysis, defect detection, and test execution efficiency.

• Develop and maintain automated and manual test frameworks that support validation of AI-enabled features across the CRIMS / Alpha platform.
• Create test plans, test cases, and test data strategies for functional, integration, regression, and non-functional testing of AI-driven capabilities.
• Validate product behavior for AI use cases, including quality of outputs, workflow integration, response accuracy, exception handling, and user experience impacts.
• Design and execute tests for AI guardrails, probabilistic outputs, and response boundaries to ensure generated results are accurate, safe, consistent, and aligned with intended business outcomes.
• Help define approaches for monitoring AI behavior in production, including drift detection, output quality trends, and feedback loops to confirm AI-enabled features continue learning and performing as expected over time.
• Identify, document, and drive resolution of defects, with clear analysis of reproducibility, business impact, and root cause patterns.
• Work with engineering teams to integrate automated testing into CI/CD pipelines and improve test coverage across the scrum team’s delivery scope.
• Apply AI tools and techniques within the test lifecycle to improve productivity, accelerate coverage creation, and enhance defect discovery.
• Collaborate with developers and product partners to define measurable acceptance criteria and quality gates for AI-related user stories and features.
• Support risk-based testing and help prioritize test efforts across new development, regression areas, and high-impact client workflows.
• Contribute to continuous improvement of QA standards, test assets, documentation, and team-level best practices for testing AI-enabled products.
• Communicate testing progress, risks, and quality insights effectively within the scrum team and to broader stakeholders as needed.

• 5–8 years of professional software testing or quality assurance experience, preferably within agile or scrum-based product development teams.
• Proven experience implementing AI capabilities within the software testing process, such as intelligent test generation, test optimization, defect analysis, or productivity acceleration.
• Critical to the role is hands‑on experience testing AI functionality embedded within a software product, including validation of generated outputs, model-driven behavior, workflow integration, and…