Member of Technical Staff Job New York New York USA,Quality Assurance

Member of Technical Staff (QA Engineer - Agentic Systems)

Join a fast-growing AI startup transforming how pharmaceutical companies create and launch marketing campaigns. You'll be the first dedicated QA hire, owning quality across a platform powered by LLMs, agentic systems, and AI workflows. This is a highly technical role where you'll work directly with leadership and engineering to build the testing, evaluation, and quality infrastructure that keeps the platform reliable, compliant, and production-ready.

Responsibilities

Build and own evaluation frameworks for LLM and agentic systems
Create automated testing infrastructure across AI, backend, and frontend systems
Design tests that identify agent failures, regressions, and compliance risks
Own end-to-end testing using Playwright and manual exploratory QA
Implement CI/CD quality gates, test automation, and release sign-off processes
Partner with Engineers and leadership to define the company's quality standards

Requirements

Strong experience with Python and building testing infrastructure
Hands-on expertise with Playwright or modern end-to-end automation frameworks
Experience testing LLM, ML, or non-deterministic systems
Knowledge of evaluation methodologies including golden datasets, LLM-as-judge, and regression testing
Strong understanding of quality metrics, statistical analysis, and error investigation
Ability to operate independently and build processes from 0 → 1
Agentic AI, Lang Chain, Lang Graph, or LLM tooling experience strongly preferred

Why apply?

First QA hire with ownership over the entire quality function
Work on the cutting edge of AI and agentic systems
Partner closely with an elite team backed by top healthcare and AI operators
Competitive compensation with meaningful equity upside
Health, dental, vision, 401(k), relocation support, and unlimited PTO
Help shape a platform already trusted by 50+ pharmaceutical brands