AI QA, AVP

Position: AI QA, AVP-2
Quality Assurance Engineer - AI Testing, Charles River Development / State Street Alpha

Job Description

Why this role is important to us

The team you will be joining is responsible for advancing quality engineering for AI-enabled capabilities within the CRIMS / State Street Alpha product suite. As a Quality Assurance Engineer focused on AI testing, you will work within a scrum team to design and implement scalable test frameworks, validate AI-driven product behavior, and strengthen the team's ability to test intelligent features with speed, rigor, and confidence.

This role is ideal for a mid-level QA professional who combines strong software testing fundamentals with hands-on experience applying AI in the test process and testing AI functionality embedded in enterprise products.

Role Description

* Partner closely with product managers, software engineers, architects, and scrum team members to ensure AI-enabled capabilities are testable, reliable, and aligned to business and client needs.

* Design and build reusable QA frameworks and test approaches tailored to validating AI-driven functionality, including model-integrated workflows, intelligent recommendations, generated outputs, and workflow automation.

* Define test strategies that address both traditional application quality and AI-specific quality dimensions such as consistency, accuracy, robustness, explainability, and edge-case handling.

* Contribute to sprint planning, story refinement, and acceptance criteria development to ensure quality considerations are embedded early in the software development lifecycle.

* Help evolve the team's quality engineering practices by introducing practical uses of AI to improve test design, coverage analysis, defect detection, and test execution efficiency.

Responsibilities

* Develop and maintain automated and manual test frameworks that support validation of AI-enabled features across the CRIMS / Alpha platform.

* Create test plans, test cases, and test data strategies for functional, integration, regression, and non-functional testing of AI-driven capabilities.

* Validate product behavior for AI use cases, including quality of outputs, workflow integration, response accuracy, exception handling, and user experience impacts.

* Design and execute tests for AI guardrails, probabilistic outputs, and response boundaries to ensure generated results are accurate, safe, consistent, and aligned with intended business outcomes.

* Help define approaches for monitoring AI behavior in production, including drift detection, output quality trends, and feedback loops to confirm AI-enabled features continue learning and performing as expected over time.

* Identify, document, and drive resolution of defects, with clear analysis of reproducibility, business impact, and root cause patterns.

* Work with engineering teams to integrate automated testing into CI/CD pipelines and improve test coverage across the scrum team's delivery scope.

* Apply AI tools and techniques within the test lifecycle to improve productivity, accelerate coverage creation, and enhance defect discovery.

* Collaborate with developers and product partners to define measurable acceptance criteria and quality gates for AI-related user stories and features.

* Support risk-based testing and help prioritize test efforts across new development, regression areas, and high-impact client workflows.

* Contribute to continuous improvement of QA standards, test assets, documentation, and team-level best practices for testing AI-enabled products.

* Communicate testing progress, risks, and quality insights effectively within the scrum team and to broader stakeholders as needed.

Skills Required

* 5-8 years of professional software testing or quality assurance experience, preferably within agile or scrum-based product development teams.

* Proven experience implementing AI capabilities within the software testing process, such as intelligent test generation, test optimization, defect analysis, or productivity acceleration.

* Critical to the role is hands-on experience testing AI functionality embedded within a software product, including validation of generated outputs, model-driven…