AI Tester

Back AI Tester
Other Area(s) Albany , New York W-2 hourly Nov 5, 2026

Our goal is to solve problems and deliver results for our clients. At Cloud and Things, you can be a part of transforming the public sector’s IT environment. Our team is on the forefront of helping to solve the government's most complex IT challenges. If you are seeking a role that offers the opportunity to work on rewarding projects, consider a career with Cloud and Things.

* This is an exempt position. Salary commensurate with experience*

Job Title
: AI Tester
Location
:
Albany, NY (Remote Available)
Duration
: 12 Months (with potential for extension)
W2 Hourly Rate
: $47-$57/hour
Final date to receive applications
:
November 8, 2026

We are seeking an AI Tester who will support our NYS client on the Medicaid Eligibility and Client Management (MECM) modernization program. The consultant will test AI and generative AI solutions, including conversational AI, retrieval-augmented generation (RAG), and agentic AI, to ensure alignment with business, functional, and technical requirements. This role will assess output quality, validate safeguards, and provide testing metrics and readiness recommendations.

• Develop and execute test plans, test cases, and evaluation scenarios for conversational AI, RAG, agentic AI, and other GenAI solutions.
• Assess AI outputs for accuracy, relevance, consistency, completeness, and alignment with functional, business, and technical requirements.
• Perform negative, edge-case, and adversarial testing to identify unexpected AI behavior and failure modes.
• Validate content retention, guardrails, and overall solution quality; document findings and coordinate defect resolution.
• Track test results and quality metrics and communicate clear readiness recommendations to project stakeholders.

• 2-5 years of hands-on experience as an AI Tester, including testing and quality assurance of AI solutions.

• Experience with software quality assurance and testing; AI/ML or generative AI testing experience preferred.
• Familiarity with large language models (LLMs) and AI evaluation techniques.
• Experience with test management and defect-tracking tools such as Jira.
• Strong analytical skills and ability to recognize unexpected AI behavior.
• Experience working in regulated or public-sector environments.

• Opportunities to lead impactful transformations in the public sector infrastructure.
• Engagement with high-profile government initiatives, supporting meaningful change.
• Professional development through exposure to innovative Agile practices and leadership collaboration.
• A collaborative environment that values ownership, initiative, and continuous learning.

Ready to make a difference?

We’re eager to connect with qualified candidates committed to delivering results and fostering excellence within client projects.

Cloud and Things complies with all applicable federal, state, and local laws regarding recruitment and hiring. All qualified applicants are considered for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, protected veteran status, or any other category protected by applicable federal, state, or local laws.