QA Engineer, AI

Applied Business Software, Inc., (ABS), the maker of The Mortgage Office®, is the industry leader in private lending and loan management technology. Since 1978, our software has powered thousands of lenders, including private money firms, municipalities, CDFIs, Tribal Nations, universities, franchisors, third-party servicers and other non-bank lenders, helping them streamline, automate, and scale their lending operations with confidence.

We are hiring for a QA Engineer, (AI), who will be responsible for validating both:
Deterministic systems (APIs, backend logic, UI flows) and Non-deterministic AI systems (LLM outputs, prompt behavior, edge cases). We are building an AI-first product where core functionality depends on prompt engineering and LLM behavior. This position requires someone to design, execute, and automate testing strategies for LLM-based features, and AI-driven workflows. Additionally, they will test end-to-end product flows across frontend, backend, and APIs.

• Test end-to-end product flows across frontend, backend, and APIs of an event based system
• Design structured test cases for prompt-based features
• Build and maintain test datasets
• Define evaluation criteria for AI outputs
• Run regression testing when prompts or models change
• Automate repeatable tests
• Track quality metrics over time (output quality, failure rate, latency issues)

• 3+ years in QA / software testing, preferably for LLM based product
• Strong Python skills (Automation, LLM, Evaluations)
• Experience testing APIs and full-stack systems
• Ability to think in edge cases and break systems intentionally
• Experience working with LLM-based products

• Comprehensive health, dental, and vision coverage.
• 401(k) with company funded match.
• Mission-driven company with a 45+ year legacy and industry-leading products trusted by thousands of lenders.
• Collaborative, supportive team known for long tenure, deep expertise, and a culture built on trust and integrity.
• Help modernize and evolve the leading platform in private lending technology.
• Beautiful San Mateo office with a tight-knit culture.
• Competitive compensation and room for professional growth.
• Performance-based bonuses.
• Generous PTO.
• Modern workspace with on-site amenities.
• Awesome company Christmas party.

The pay range for this role is:

120, USD per year (Hybrid, San Mateo, CA)