QA Engineer, AI

About Applied Business Software, Inc.,

Applied Business Software, Inc., (ABS), the maker of The Mortgage Office , is the industry leader in private lending and loan management technology. Since 1978, our software has powered thousands of lenders, including private money firms, municipalities, CDFIs, Tribal Nations, universities, franchisors, third-party servicers and other non-bank lenders, helping them streamline, automate, and scale their lending operations with confidence.

About the role

We're looking for a QA Engineer (AI) to own quality across our AI-first product; a platform where core functionality depends on prompt engineering and LLM behavior. You'll be responsible for building and running the automated testing infrastructure that keeps both our deterministic systems (APIs, backend logic, UI flows) and non-deterministic AI systems (LLM outputs, prompt behavior, edge cases) reliable.

This role is our automation expert: you'll design and build automated end-to-end test suites (using tools like Playwright or Selenium) that cover the full product: frontend, backend, and API layers, while also developing testing strategies specifically for LLM-based features and AI-driven workflows. You'll be the person the team turns to for how we test, what we automate, and how we know when something's broken;

whether that's a UI flow or a model's output quality.

What you'll do

• Design and build automated end-to-end test suites (Playwright, Selenium, or similar) covering frontend, backend, and API flows across our event-based system
• Own automation strategy for the team: identify what should be automated, build the frameworks, and maintain them over time
• Design structured test cases for prompt-based features
• Build and maintain test datasets for AI evaluation
• Define evaluation criteria for AI outputs
• Run regression testing when prompts or models change
• Track quality metrics over time (output quality, failure rate, latency issues)
• Serve as the team's go-to expert on automated and end-to-end testing practices

Qualifications

• 5+ years of experience in QA / software testing, preferably for LLM-based products
• Hands-on experience with automation frameworks such as Playwright or Selenium
• Strong Python skills (automation, LLM evaluation, scripting)
• Experience testing APIs and full-stack systems
• Genuine curiosity about AI and a solid understanding of how the space is evolving
• Ability to think in edge cases and break systems intentionally
• Experience working with LLM-based products and prompt engineering workflows

Why Join Us?

• Comprehensive health, dental, and vision coverage.
• 401(k) with company funded match.
• Mission-driven company with a 45+ year legacy and industry-leading products trusted by thousands of lenders.
• Collaborative, supportive team known for long tenure, deep expertise, and a culture built on trust and integrity.
• Help modernize and evolve the leading platform in private lending technology.
• Beautiful San Mateo office with a tight-knit culture.
• Competitive compensation and room for professional growth.
• Performance-based bonuses.
• Generous PTO.
• Modern workspace with on-site amenities.

The pay range for this role is:

130, USD per year (Hybrid, San Mateo, CA)