Machine Learning Evaluation Engineer

We are Pepoltek, an IT company building a portfolio of SaaS products used across education, HR, and enterprise operations, including BIDYATek, a school management platform for institutions across Bangladesh, our Intelligent HR Solution, a cloud-based HRMS used by growing companies, and an Enterprise ERP suite for supply chain and financial operations. Alongside our own products, we work with software development partners across the globe to extend our engineering capacity on their behalf.

You will own evaluation of a complex multi-agent system end to end, the instrument that tells a research-driven team whether their identity representations are actually getting better. The product works, and it has real black-box behavior, so the job is defining what better means for representations that change over time, without ground truth, and building the machinery that measures it. If you love constructing measurements from nothing, reading traces to find what's actually broken, and shipping the fix yourself, this role is built for you.

• Design the evals, defining the scores and what better means for an entity representation that changes over time, and keeping that definition current as the product and methods move
• Build the pipelines and harnesses, including data ingestion, labels, versions, reruns, and judges, so the team can ask a new question this week and get an answer this week
• Run the evals and harvest insights, reading traces and results to find what's actually broken and proposing fixes that produce higher-fidelity representations
• Build simulation agents at scale, raising the ceiling on how many entities can be modeled and measured at once
• Own the loop end to end, from the question through the pipeline, the rerun, and the writeup, without handing work off or waiting on someone else's sprint

• 3+ years designing and owning LLM evaluation systems
• Experience designing, building, and running evals end to end for LLMs
• Production-quality Python, including tests, maintenance, and release cadence
• Hands-on experience with PyTorch, Hugging Face, and Weights and Biases

• Work on memory, identity, personalization, agent evaluation, or agentic optimization
• A shipped research record, as first or co-author at NeurIPS, ICML, ICLR, or equivalent, or self-published open-source work of comparable quality
• Data pipeline tooling experience, such as Hydra, Kafka, or SQL
• A degree in computer science, mathematics, statistics, or a related quantitative field

You will own evaluation for the product end to end, shaping what better means for the company's core measurement problem, with direct influence over the research direction and methods the team relies on.

• $220,000 to $300,000 base salary
• Competitive equity

Candidates must have current work authorization in the United States without need for employer sponsorship.

• Employment type:
  
  Full time
• Location:
  
  New York, NY
• Workplace:
  On-site, 5 days in-office
• Vacancies: 1