Platform Engineer AWS AgentCore Evaluation

Job Title

Responsibilities

Integrate AWS Agent Core Evaluation API into CI/CD pipelines to support automated deployment quality gates

Configure and maintain on demand evaluation workflows as part of build and release processes

Develop and support Git Hub Actions workflows for automated evaluation execution and deployment validation

Configure and manage online evaluation modes with appropriate sampling strategies

Build and maintain Cloud Watch dashboards to track evaluation metrics for agents and evaluators over time

Design and configure Cloud Watch alarms to identify quality degradation and operational issues

Collaborate with AI and platform engineering teams to define evaluation thresholds and quality standards

Monitor evaluation results and contribute to regression detection and release validation processes

Support observability, reporting, and operational monitoring initiatives across AI platforms

Create and maintain technical documentation related to evaluation workflows, monitoring, and deployment automation

Requirements

3+ years of experience in platform engineering, Dev Ops engineering, or a related field

Experience integrating quality validation processes into CI/CD pipelines

Hands on experience designing and maintaining deployment gates for automated release workflows

Experience building and supporting AWS Cloud Watch dashboards and monitoring solutions

Experience configuring Cloud Watch alarms and operational alerting strategies

Knowledge of cloud infrastructure, automation, and deployment best practices

Experience working with monitoring, observability, and platform operations tools

Strong analytical and troubleshooting skills

Good written and verbal communication skills

Nice to have

Experience with AWS Agent Core Evaluation online mode configuration and sampling strategies

Experience using evaluations to compare runtime versions and identify regressions

Knowledge of Cloud Watch Embedded Metric Format (EMF) for structured metric collection and reporting

Experience with AI platform monitoring and evaluation frameworks

Exposure to automated quality assurance processes for machine learning or agent based systems

Experience supporting enterprise scale cloud platforms and deployment pipelines

We Offer

Vacation as per the laws of your country. We do ask you to take a proper rest

Health insurance we help you to take out an insurance policy for you and your loved ones

Sick pay 10 days without a doctor's note, afterwards - as per the laws of your country

Equal opportunities

Time off for state holidays according to the official calendar, regardless of the client's schedule

Pleasant environment two large corporate parties and many small get-togethers for colleagues

Comfort service solving technical and everyday problems at work

More about the perks List of benefits in PDF The benefits package may vary depending on the region and the type of contract