Python Developer AI Evaluation Frameworks

Job Title

Responsibilities

Develop custom AI evaluation services using Python and AWS Lambda

Build LLM as judge evaluators to assess subjective quality dimensions of AI generated content

Implement tool response validation using JSON Schema based evaluation mechanisms

Develop workflow contract compliance checks at session level

Create numerical accuracy validation logic and trace level evaluation controls

Implement PII detection mechanisms using pattern matching techniques and AWS Bedrock Guardrails

Integrate evaluation services with enterprise AI platforms and workflows

Design and maintain reusable evaluation frameworks and quality standards

Collaborate with AI engineers and architects to define evaluation methodologies

Document evaluation rules, quality metrics, and implementation standards

Support monitoring, troubleshooting, and continuous improvement of evaluation services

Requirements

3+ years of experience in Python software engineering

Experience implementing quality assurance, validation, or evaluation frameworks for AI or machine learning systems

Experience developing and deploying AWS Lambda functions

Experience designing backend services and data validation processes

Knowledge of prompt engineering techniques for AI assessment and evaluation

Experience working with JSON based APIs and schema validation approaches

Understanding of AI quality metrics, testing methodologies, and governance concepts

Experience working in cloud environments and modern software development practices

Strong analytical and problem solving skills

Nice to have

Experience registering custom evaluators within AWS Agent Core Evaluation

Experience integrating AWS Bedrock Guardrails for PII detection and content validation

Experience using Cloud Watch Logs for monitoring, reporting, and evaluation result tracking

Experience with AI observability and monitoring frameworks

Knowledge of enterprise AI governance and compliance requirements

Experience building automated quality control solutions for AI platforms

Familiarity with large language model evaluation methodologies and benchmarking techniques

Experience working with enterprise scale AI workloads and cloud native architectures

We offer

Vacation as per the laws of your country. We do ask you to take a proper rest

Health insurance. We help you to take out an insurance policy for you and your loved ones

Sick pay. 10 days without a doctor's note, afterwards - as per the laws of your country

Equal opportunities

Time off for state holidays according to the official calendar, regardless of the client's schedule

Pleasant environment. Two large corporate parties and many small get-togethers for colleagues

Comfort service. Solving technical and everyday problems at work

More about the perks List of benefits in PDF The benefits package may vary depending on the region and the type of contract