AI Engineer

Plexus has partnered with an early-stage AI company building an evaluation platform that helps enterprise teams validate generative AI models and agents before they go live. The product is already live, with a public leader board benchmarking leading LLMs, and the team is now growing.

The role

You will build the infrastructure that runs evaluations of frontier models and agents s is a hands-on role with real ownership, working directly with the founders.

Responsibilities

• Build scalable pipelines for benchmark execution, scoring and results aggregation across multiple LLM providers and custom endpoints
• Develop systems that generate custom evals on demand from customer data
• Design secure, multi-tenant services and APIs to enterprise privacy standards
• Own orchestration, observability, performance and cost as usage grows

Experience required

• Proven experience building AI systems and AI infrastructure in production
• Strong Python (or similar) and backend or distributed systems skills
• Hands-on experience with LLM APIs and open or self-hosted models
• Cloud (AWS, GCP or Azure), Docker/Kubernetes and infrastructure-as-code
• Comfortable in an early-stage, fast-moving environment

Nice to have

• Experience with LLM evaluation, benchmarking or agent frameworks
• Startup or scale-up background

Comp:
Base + Equity