Adversarial Evaluator, AI Search Red Teaming Evaluator
-
IT/Tech
Job Description & How to Apply Below
Contract
· Remote
· 5-week pilot
· 15 openings
Truelancer AI is building a specialist red team for an AI search safety pilot with a global enterprise client. You’ll work as an individual contributor. You’ll own specific attack surfaces, design and run adversarial scenarios against an AI search and RAG system, and produce evidence that holds up under QA review.
We value complete evidence and defensible coverage more than prompt volume.
What you’ll doPlan attacks: form risk hypotheses and design original, bounded adversarial scenarios.
Run attacks: execute multi-turn, adaptive prompt attacks, including context escalation, on authorised systems.
Evaluate outputs for these failure types:
- grounding errors
- source contradictions
- stale information
- unverifiable citations
- trust-boundary violations, such as indirect prompt injection
Record evidence that can be reproduced:
- exact prompt sequences
- session and version context
- timestamps
Submit finding cards to QA, and adjust your judgements as rubrics and guidelines are updated.
Classify each outcome as a candidate finding, a quality issue or inconclusive. Know when to stop and elevate an unsafe or uncertain case.
Stay within scope: keep to client-approved boundaries and privacy policies at all times.
Keep current on client taxonomies, system changes, policy revisions and new risk definitions.
Work with the team: the Red-Team Lead, Project Manager and QA analysts.
Experience: 3 or more years in one or more of the following:
- AI red teaming, LLM evaluation or GenAI testing and QA
- Trust & Safety, content safety or content policy operations
- Fraud, abuse or risk investigations
- Responsible AI, AI governance or AI safety
Methods: hands-on experience with LLM and model evaluation, RLHF or human-feedback work, and prompt engineering.
Adversarial skill: working knowledge of prompt injection, jailbreak testing and adversarial techniques.
Mindset: sceptical and analytical. You can rule out false positives and check citations against sources.
Communication: excellent written and spoken US English, suited to structured reporting.
Detail: very high attention to detail and ownership of what you deliver.
Education: graduate or undergraduate degree.
- Defensible coverage of your assigned surfaces, and how reliably your findings reproduce
- QA calibration and adjudication scores, with a target of at least 85% agreement
- Compliance with client boundary and privacy policies
- Quality of your finding cards and how well you take on calibration updates
Truelancer AI verifies each project before posting.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction