Quality Assurance Engineer

Jul 14, 2026

AI71 is seeking a Senior QA Automation Engineer to lead the validation and verification strategies for EDGE Group’s AI transformation. You will be responsible for defining "what good looks like" for non-deterministic AI systems, ensuring that Large Language Models (LLMs) and predictive engines meet the strict reliability standards of the defense sector.

You will act as the bridge between Agile development and formal Systems Engineering. Your mandate is to build automated testing frameworks that not only verify software functionality but also validate AI behaviors against "Ground Truth" datasets. Working within a structured "Stage Gate" delivery model, you will ensure our AI agents pass the rigorous Test Readiness Reviews (TRR) and Functional Configuration Audits (FCA) required for deployment.

• AI & LLM Validation (LeverEDGE)
• Non-Deterministic Testing: Architect automated frameworks to evaluate Generative AI outputs (e.g., drafted CONOPS, technical requirements) for hallucination, consistency, and factual accuracy against "Gold Standard" datasets.
• RAG Evaluation: Implement automated metrics (e.g., RAGAS, faithfulness, answer relevance, etc.) to verify that the Retrieval-Augmented Generation pipeline is accurately citing internal technical documentation and regulatory texts.
• Prompt Regression: Design regression suites that monitor "prompt drift," ensuring that changes to the underlying model or system instructions do not degrade the quality of AI-generated engineering documents.
• Integration & System Verification (Supply Chain)
• ERP Integration Testing: Build robust integration tests to validate data consistency between AI agents and critical enterprise systems (e.g., SAP S/4
  
  HANA, Ariba, etc.), ensuring no corruption of Bill of Materials (BOM) or financial data.
• Performance Benchmarking: Design performance tests to validate the latency and throughput of forecasting models and risk scoring engines, ensuring they meet the real-time requirements of supply chain dashboards.
• API Validation: Automate the testing of secure API gateways, verifying that Role-Based Access Control (RBAC) and PII redaction logic are functioning correctly before data reaches the AI models.
• Governance & Traceability
• V-Model Alignment: Map automated test cases directly to "System Requirements" and "User Needs," creating the digital evidence required for formal Verification and Validation (V&V) reports described in the Systems Engineering Handbook.
• Stage Gate Compliance: Prepare "Test Readiness" packages for formal Stage Gate reviews, providing quantitative evidence that the system is stable enough to move from MVP to Production.
• Defect Lifecycle Management: Manage the feedback loop between the "Requirements Quality Assistant" and the development teams, ensuring that defects found in AI logic are traced back to specific model versions or data sets.

• Core Automation: Expert proficiency in Python for building custom test harnesses (Pytest) and standard automation libraries (Selenium/Playwright for UI, Requests for API).
• Core Performance Testing: Expert proficiency in crafting Performance Test Plans + Implementations (e.g., Locust, Jmeter, K6, etc.)
• AI Evaluation: Experience utilizing frameworks for evaluating LLMs (e.g., Deep Eval, Tru Lens, or custom Python evaluators). Understanding of "Ground Truth" dataset creation and management.
• Data Validation: Proficiency with SQL and data validation tools (e.g., Great Expectations) to verify data quality within Data Lake houses and Vector Databases.
• CI/CD Integration: Strong experience integrating automated tests into Git Lab CI/CD pipelines, enforcing "Quality Gates" that prevent non-compliant code or models from merging.
• Traceability Tools: Familiarity with requirements management tools (e.g., Jira, Linear, Jama, Polarion, etc.) and how to link automated test results to specific requirement IDs.
• Test Management/Reporting Tools :
  Strong hands-on on Managing Test Reports + Artifacts (e.g., Test Rail, Allure, etc.)
• Version Control :
  String knowledge maintaining code based frameworks (e.g., Git, Gitlab, etc.)
• Quality Engineering practices :
  Stron…