Senior AI QA Engineer LLMs & RAG; Remote

We're a small team doing real work with enterprise clients. Fully remote, no fluff.

Fully Remote Full-time Immediate start

AI Developer in Test Fully Remote Full-time Immediate start Open

We're looking for experienced AI Developers in Test to work on validating LLM-based applications, multi-agent systems, and RAG pipelines. You'll make sure AI outputs are safe, reliable, and actually useful for end users.

• Design and implement testing frameworks for LLM-based applications, multi-agent systems and RAG pipelines
• Evaluate AI outputs for accuracy, factual correctness, contextual relevance, bias, safety and compliance
• Implement governance workflows, risk scoring and compliance reporting
• Set up continuous monitoring for production AI systems with automated alerts for anomalies

• Integrate AI testing suites into CI/CD pipelines for regression testing, benchmarking and deployment gates
• Version control for prompts and test cases using Git, plus MLOps tools like MLflow or Weights & Biases
• Work with JSON, CSV, Parquet, JSONL — build evaluation datasets from real-world scenarios
• Design automated metrics and human-in-the-loop workflows including inter-annotator agreement
• Create and maintain domain-specific evaluation benchmarks and ground truth datasets

• Build and lead a testing team, be the go-to expert on AI quality
• Own client relationships — retention and growth
• Share your knowledge within the B-Sure Digital consulting community

• Hands-on experience with GenAI apps — prompt engineering, API integrations, output workflows
• Familiarity with RAG, vector databases, embedding strategies and multi-agent systems
• Understanding of GenAI challenges: hallucinations, prompt sensitivity, output variability
• Solid testing background in enterprise environments
• Proficiency with API testing frameworks, CI/CD and Git
• Data handling skills and knowledge of AI evaluation methodologies
• Comfortable working in ambiguous environments — this is a new field, you'll help define it
• Strong communication skills — you'll talk to both engineers and stakeholders

• Experience using LLMs (ChatGPT, Claude, etc.) for test generation
• Knowledge of computer vision for visual testing
• Experience with AWS, Azure, or GCP
• ISTQB or similar certifications

• Fully remote — work from wherever
• Budget for training and conferences
• Access to the latest AI tools and platforms

We're always open to hearing from good engineers. Drop us a message and we'll talk.