GenAI Evaluation Engineer

Job Description

• Strong understanding of LLMs and generative AI concepts, including model behavior and output evaluation
• Experience with AI evaluation and benchmarking methodologies, including baseline creation and model comparison
• Hands-on expertise in Eval testing, creating structured test suites to measure accuracy, relevance, safety, and performance
• Ability to define and apply evaluation metrics (precisionrecall, BLEUROUGE, F1, hallucination rate, latency, cost per output) Prompt engineering and prompt testing experience across zero-shot, few-shot, and system prompt scenarios
• Python other programming languages, for automation, data analysis, batch evaluation execution, and API integration
• Experience with evaluation tools/frameworks (OpenAI Evals, Hugging Face evals, Promptfoo, Ragas, Deep Eval, LM Eval Harness)
• Ability to create datasets, test cases, benchmarks, and ground truth references for consistent scoring
• Test design and test automation experience, including reproducible evaluation pipelines
• Knowledge of AI safety, bias, security testing, and hallucination analysis

• RAG evaluation experience
• Azure OpenAI
• OpenAI
• Anthropic
• Google AI platforms
• Performance benchmarking (speed, throughput, cost)
• Domain knowledge Office apps enterprise systems networking

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.