Automation Engineer; AI​/ML & API Testing

Job Title

Pay Range: $58.52hr - $67.85hr

• 10+ years of proven experience in the software development industry, participating in a team environment.
• 5+ years of experience in UI automation, such as Selenium Web Driver.
• 5+ years of proven experience in testing web services such as RESTful API.
• 3+ years of proven experience in performance testing using tools such as Apache JMeter.
• 3+ years of working in a cloud-based development and production environment such as AWS.
• 3+ years of proven experience with programming languages such as Java/C#. Familiarity with Python is a plus.
• Proficient with both SQL and No
  
  SQL databases.
• Experienced working in Continuous Integration/Continuous Deployment (CI/CD) environments.
• Having AWS/Cloud technologies and AI/ML powered applications knowledge are big plus.
• Experience with AI/ML-powered applications, Generative AI, and Agentic AI systems is highly desirable.
• Understanding of AI testing concepts, including prompt validation, model output verification, hallucination detection, bias testing, guardrail validation, and AI quality evaluation.
• Familiarity with LLMs (Large Language Models), AI agents, retrieval-augmented generation (RAG), and AI-assisted workflows is a strong plus.
• Ability to develop QA strategies and test approaches for AI-driven and autonomous agent-based applications.

• Design and execute functional and automation testing for applications.
• Develop and execute automated test scripts using testing tools; document and summarize results.
• Build, update, and maintain UI automation test cases using Selenium with C# (flexibility to learn C# if not already proficient).
• Perform manual API testing using tools such as Postman or Insomnia.
• Perform API automation testing using industry-standard frameworks and tools.
• Participate actively in Agile ceremonies (daily stand-ups, sprint planning, retrospectives).
• Collaborate closely with Development and Business teams to ensure project success.
• Quickly adapt to new technologies and tools (e.g., Amazon Web Services).
• Work with SQL and No
  
  SQL databases for backend validation and robust data testing.
• Create comprehensive end-to-end test plans, test specifications, and test case templates.
• Develop and document detailed test plans and test cases.
• Facilitate defect tracking, reporting, and resolution management.
• Review requirements (Functional Specifications, User Stories, Change Requests) to ensure full coverage.
• Identify and manage test data; prepare automation-ready test cases.
• Conduct system testing, regression testing, and support User Acceptance Testing (UAT).
• Apply test case design techniques (e.g., boundary value analysis, equivalence partitioning).
• Implement Shift Left Testing practices by participating in early requirement and design reviews, writing acceptance criteria up front, and integrating automated tests in CI/CD pipelines.
• Be actively involved in the deployment process, collaborating with Dev Ops and engineering teams to ensure smooth releases.
• Design and execute test strategies for AI/ML and Agentic AI applications, including validating AI-generated outputs for accuracy, consistency, and reliability.
• Test AI agents for workflow execution, decision-making, and multi-step task completion.
• Evaluate AI model behavior, prompt responses, hallucinations, and edge cases.
• Verify AI safety controls, guardrails, security, and compliance requirements.
• Create test datasets and benchmarks for AI quality assessment.
• Monitor and report AI model performance, accuracy, and user experience metrics.

• Experience with application monitoring tools (e.g., New Relic, Datadog, Cloud Watch).
• Working knowledge of Python for scripting and automation.
• Familiarity with AWS services for cloud-based testing and deployment.
• Hands-on experience with performance and load testing using JMeter.
• Experience with testing AI/ML-powered applications, including validating model outputs, creating test datasets, evaluating model performance.