AI Evaluation Specialist

Location: Remote (Continental U.S.)

Company: JR Software Solutions Inc. (JRSS)

Employment Type: Full-time

JRSS is a certified Women-Owned Small Business (WOSB), Small Disadvantaged Business (SDB), and HUBZone business headquartered in Tampa, FL. We specialize in Cloud Services, AI and Machine Learning, Big Data and BI Analytics, Enterprise Resource Planning, IT Project Management, and Staff Augmentation for Federal Civilian agencies (e.g., CDC) and Fortune 500 clients. Our AI/ML teams help public health agencies adopt generative AI responsibly, with the rigor, transparency, and safeguards that federal missions require.

The JRSS AI/ML Team is seeking an AI Evaluation Specialist to determine the effectiveness, safety, usability, and readiness of enterprise AI tools for adoption in a federal public health environment. You will partner with product managers, data scientists, AI engineers, and security and privacy teams to run structured assessments of generative AI products and platforms, including OpenAI tools and life sciences models such as Rosalind.

Your findings will guide which AI solutions are adopted, comparing them across quality, accuracy, reliability, safety, privacy, and security.

• Plan and execute hands-on evaluations of AI tools, including OpenAI products and life sciences models such as Rosalind, using representative business scenarios, workflows, prompts, datasets, and user personas.
• Assess AI solutions for response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness.
• Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI-enabled applications.
• Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations in collaboration with security, privacy, legal, and governance stakeholders.
• Conduct comparative evaluations and proof-of-concept assessments across multiple AI products, models, configurations, and vendors, including analysis of functionality, integration requirements, performance, and cost.
• Evaluate AI agents and tool-enabled workflows, including retrieval-augmented generation (RAG), enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP)-enabled capabilities.
• Translate evaluation findings into clear, actionable recommendations for technical and business stakeholders.
• Stay current with advancements in generative AI, model evaluation, AI safety, and enterprise AI products, and incorporate relevant methods into the evaluation program.
• Contribute to knowledge sharing and a strong, collaborative team environment.

• 5+ years of experience in model evaluation or model quality assurance, with hands-on experience evaluating AI-enabled products or services.
• Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria.
• Hands-on experience with generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes.
• Experience assessing AI systems for accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost.
• Familiarity with common generative AI architectures and capabilities, including LLMs, RAG, AI agents, APIs, enterprise search, and tool or connector integrations.
• Must be a U.S. Citizen or Lawful Permanent Resident (Green Card holder).
• Ability to obtain and maintain a Public Trust clearance.

• Experience…