QA Automation Engineer

Residency

All applicants must currently reside in the United States

Ask Sage( BigBearAI company) is seeking a QA Automation Engineer that will translate nuanced product requirements and customer workflows into effective test coverage, build and maintain automation with Playwright, and partner with Product and Engineering to make informed release decisions as the platform evolves. We are looking for an engineer who can reason about complex, nondeterministic AI behavior, design risk-based coverage that reflects real customer impact, and produce automation that engineers trust.

The engineer will own the automated coverage and the evidence behind release readiness across browser, application, and backend boundaries, where auditability, security, and government compliance expectations are part of everyday delivery.

• Translate requirements into testable outcomes. Understand product goals, customer use cases, and complex feature interactions; identify ambiguities and define acceptance criteria with Product and Engineering.
• Own automated test coverage. Design, implement, and maintain Playwright tests covering critical user journeys, feature functionality, and regressions, supported by API and integration testing.
• Develop a risk-based testing strategy. Prioritize coverage according to customer impact, feature dependencies, and the product roadmap; adapt deliberately as priorities change. Test conversational AI workflows. Validate streaming responses, conversation history, file uploads, document retrieval and citations, model selection, and tool execution, including interruptions, timeouts, and partial failures.
• Evaluate AI response quality. Build representative evaluation datasets and scoring criteria for accuracy, grounding, instruction following, and appropriate handling of unsafe requests. Account for natural variation in model responses.
• Make automation reliable and useful. Integrate tests into CI/CD, investigate flaky tests, maintain isolated test data, and provide actionable failure diagnostics.
• Communicate release readiness. Report defects with reproducible evidence, customer impact, and severity; explain coverage gaps and residual risks before UAT and release.
• Preserve decision history. Document expected behavior, approved changes, and the rationale behind testing decisions so the team can distinguish intended changes from regressions.
• Cover platform and AI infrastructure surfaces. Extend automation across backend APIs, authentication flows, billing and token behavior, model routing, AI workflow execution, file parsing, MCP/tool execution, agentic harnesses, and passthrough APIs, including provider-facing API compatibility.
• Control test cost and execution footprint. Design AI workflow coverage that minimizes unnecessary token usage, external provider calls, latency, and execution cost without sacrificing signal.
• Automate security-sensitive validation. Build repeatable coverage for user isolation, permission boundaries, input validation, sanitization, rate limits, replay prevention, safe error handling, and layered control behavior.
• Apply AI-assisted testing tools responsibly. Use AI assistance to accelerate test design, generation, triage, and maintenance while critically reviewing generated tests, assertions, and proposed repairs against reliability, reviewability, and deterministic validation standards.
• Scale the automation footprint. Maintain test infrastructure, fixtures, and test data management that must grow with an expanding product surface area and an active engineering team.

• 4+years of experience QA automation engineering experience
• Demonstrated experience building and maintaining automated tests with Playwright, including fixtures, resilient…