QA Engineer Job New York New York USA,Software Development

Location: New York

Responsibilities

Kforce has a client that is seeking a QA Engineer in New York, NY.

Design, develop, and maintain automated test suites for AI platforms and services
Integrate testing capabilities into CI/CD pipelines to support rapid and reliable releases
Build automated regression, functional, integration, and end-to-end testing frameworks
Collaborate with platform engineering teams to improve testing coverage and release confidence
Monitor test execution results and proactively identify quality risks
Develop and maintain evaluation harnesses used to assess AI model and agent performance
Create and curate golden datasets that represent expected business outcomes and user interactions
Establish deterministic validation methods to verify AI responses against predefined expectations
Design scoring methodologies, evaluation metrics, and performance rubrics aligned to business objectives
Build adversarial and edge-case testing suites to assess model robustness, safety, and resilience
Implement continuous evaluation capabilities that extend beyond pre-production testing into live environments
Monitor AI outputs against established quality thresholds and service-level expectations
Develop dashboards and reporting mechanisms that provide visibility into model quality and drift
Identify performance degradation, anomalous behavior, and emerging risks
Support root-cause analysis and remediation efforts when quality issues are identified
Validate sandbox environments, platform guardrails, and operational control frameworks
Verify that platform controls function as intended across development, testing, and production environments
Support testing of AI safety mechanisms, access controls, content filters, and governance policies
Ensure evaluation standards remain aligned with regulatory, compliance, and organizational requirements

Requirements

1-3 years of experience in Software Quality Assurance, Test Automation, Quality Engineering, Software Development, or a related technical discipline
Experience building automated tests using Python, Java, JavaScript, or similar programming languages
Familiarity with CI/CD pipelines and modern software delivery practices
Understanding of software testing methodologies, including integration, regression, and performance testing
Strong analytical and troubleshooting skills
Excellent written and verbal communication skills

Preferred Qualifications

Experience building evaluation frameworks or automated validation tools
Experience with cloud platforms such as Azure, AWS, or Google Cloud
Experience with testing frameworks such as PyTest, Playwright, Selenium, JUnit, or equivalent tools
Exposure to Generative AI, Large Language Models (LLMs), AI agents, or machine learning systems
Exposure to observability and monitoring platforms
Familiarity with prompt engineering, retrieval-augmented generation (RAG), or agentic workflows
Understanding of AI safety, governance, model evaluation, and risk management concepts

Technical Skills

Testing & Automation
Python
Py Test
API Testing
Test Automation Frameworks
CI/CD Pipelines
Git/Git Hub

AI & Data

Generative AI
LLM Evaluation
Prompt Testing
Golden Dataset Creation
Evaluation Metrics
Continuous Monitoring

Cloud & Platforms

Azure
Azure Dev Ops
Git Hub Actions
Container Technologies
Monitoring & Observability Tools

The pay range is the lowest to highest compensation we reasonably in good faith believe we would pay at posting for this role. We may ultimately pay more or less than this range. Employee pay is based on factors like relevant education, qualifications, certifications, experience, skills, seniority, location, performance, union contract and business needs. This range may be modified in the future.

We offer comprehensive benefits including medical/dental/vision…