Junior AI QA Engineer

At Tax Relief Advocates, we’re committed to serving our clients and community with integrity and empathy. We balance hard work with a vibrant, energetic culture, aiming to make a meaningful impact nationwide. Our mission is to guide people through the tax resolution process with trust, education, and care.

Tax Relief Advocates is looking for a Junior AI QA Engineer to help us test and validate the AI-driven applications, workflows, and features we build. You'll work alongside our AI QA Engineer and development team, combining traditional software QA with AI-specific testing: prompt and output validation, regression testing, hallucination detection, edge-case testing, and evaluation of AI responses. In this role, you'll help test things like AI-driven workflows, document and data extraction, chat and agent behavior, business rules and decision logic, integrations, and the React/API applications that surround our AI systems.

• Execute manual and automated test cases for AI-enabled applications and workflows.
• Work under the direction of the AI QA Engineer to build and maintain regression test suites.
• Test AI responses for accuracy, relevance, consistency, completeness, and adherence to business rules.
• Create test datasets covering normal cases, edge cases, malformed inputs, missing information, and adversarial scenarios.
• Compare expected vs. actual AI outputs and document discrepancies.
• Identify hallucinations, unsupported conclusions, incorrect classifications, extraction errors, and inconsistent AI behavior.
• Test prompts and prompt changes to catch regressions before they ship.
• Perform functional testing of web applications, APIs, dashboards, and backend workflows.
• Validate API requests and responses using tools such as Postman.
• Perform regression testing before releases.
• Reproduce reported defects and provide developers with clear repro steps, logs, screenshots, inputs, and expected behavior.
• Validate fixes in development/staging environments before production releases.
• Help maintain QA documentation, test cases, results, defect tracking, and release validation records.
• Collaborate closely with our AI QA Engineer, AI/ML engineers, full-stack developers, and product teams.

We're not expecting you to already be an ML engineer. We're looking for someone who understands the fundamental difference between deterministic software and probabilistic AI, and who's excited to dig into concepts like:

• LLM prompts and responses
• Hallucinations, and grounded vs. unsupported responses
• Structured output / JSON validation
• RAG (retrieval-augmented generation) at a basic level
• Context windows
• Prompt injection and basic AI security testing
• Regression testing for AI, and accuracy/consistency measurement
• Testing AI against business rules

• 0–2 years of software QA, testing, development, data analysis, or related technical experience.
• Basic understanding of software testing concepts and the software development lifecycle.
• Familiarity with REST APIs and JSON.
• Experience or coursework using Postman or similar API testing tools.
• Strong attention to detail and ability to systematically reproduce problems.
• Interest in AI/LLM applications and willingness to learn AI evaluation techniques.
• Ability to document test scenarios and defects clearly.

• Experience testing LLM or generative AI applications.
• Playwright, Cypress, Selenium, or similar automation experience.
• SQL knowledge.
• Familiarity with OpenAI/Claude/Gemini APIs.
• Experience with Lang Smith, Langfuse, or other LLM observability/evaluation platforms.
• Experience testing document extraction, OCR, classification, RAG, or conversational AI.

At TRA, you’ll be part of a high-impact, forward-thinking team revolutionizing tax resolution through innovative AI and automation. You'll take on exciting challenges, collaborate with top talent, and play a central role in delivering client-first solutions at scale.

We’re not just building smarter systems—we’re building a better experience for every person who turns to us for help.

Location:

Irvine, CA - onsite

Compensation: $27/Hour

Position Type:
Full-time, permanent