Skip to content
×

GC​/USC Senior AI Evaluations Engineer; AWS; _ Remote

Remote / Online - Candidates ideally in
San Jose, Santa Clara County, California, 95025, USA
Listing for: TechnoSphere, Inc.
Full Time, Remote/Work from Home position Listed on 2026-11-06
Job specializations:
Salary/Wage Range or Industry Benchmark: 180000 - 240000 USD Yearly USD 180000.00 240000.00 YEAR

Job Description & How to Apply Below

Position: Full-Time (GC/USC only) Role: Senior AI Evaluations Engineer (AWS)_ Remote
Job Title:

Senior AI Evaluations Engineer (AWS)

Location:

Remote Job Type: Remote

We are seeking a Senior AI Evaluations Engineer to AWS practice and work embedded with AWS Professional Services (Pro Serve) teams, designing and operating the evaluation frameworks that determine whether agentic AI systems are accurate, safe and ready for production at large enterprise customers, primarily in financial services. You will define quality rubrics and scoring architectures, build LLM-as-judge and agent-as-judge pipelines, and run offline regression and online, live-traffic evaluation for multi-agent workflows on Amazon Bedrock.

The ideal candidate combines expert generative AI engineering skills with a rigorous, metrics-driven approach to quality.

Required Qualifications
  • GenAI Expertise:
    Expert-level (Level 400) generative AI experience with advanced Python.
  • Production Systems:
    Proven experience building and running production-grade systems.
  • Agent Evaluation:
    Hands-on experience with offline/on-demand regression testing and online/continuous live-traffic scoring of agents.
  • LLM-as-Judge:
    Proficiency in rubric design and scoring at session, response and tool-call granularity.
  • Evaluation Metrics:
    Task completion, tool-call accuracy, trajectory quality, groundedness and safety.
  • Agentic Frameworks:
    Hands-on experience with Strands, Lang Graph, CrewAI or similar.
  • AWS Stack:
    Amazon Bedrock (models, Guardrails) and Cloud Watch, plus Lambda, S3 and IAM fundamentals.
  • AWS Certification:
    Active AWS Certified Generative AI Developer – Professional or AWS Certified Machine Learning Engineer – Associate (required).
Preferred Qualifications
  • Bedrock Agent Core Evaluations:
    On-demand SDK runner, online monitoring and A/B testing.
  • Advanced Evaluation Methods:
    Agent-as-a-judge frameworks and human-in-the-loop (HITL) evaluation pipelines.
  • Benchmarking:
    Benchmarking frameworks (tau-bench, SWE-bench, GAIA) and custom dataset construction.
  • RAG Evaluation: RAG quality metrics such as faithfulness, context precision and recall.
  • CI/CD & Observability: CI/CD integration for evaluation regression, tracing and structured logging.
  • Legacy Modernization:
    Familiarity with Java/Spring, Sybase, DB2 or COBOL as source context for agentic modernization or integration pattern design.
  • Financial Services:
    Financial services or regulated-industry experience with compliance-aware delivery (trade surveillance, data controls, audit requirements).
  • Scale:
    Large-scale portfolio delivery: reusable patterns or components applied across 50+ applications, including wave planning and throughput measurement.
  • Consulting:
    Prior experience with AWS Pro Serve or in a consulting delivery role.

Position Requirements

10+ Years work experience
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction

Search for further Jobs Here:

SEARCH JOBS: 10+ Million Listings
(Try combinations for better Results! Or enter less keywords for broader Results)
Education Level
Experience Level (years)
Posted in last:
Salary