AI Automation Engineer

Job Description

We're looking for a junior engineer to be the primary engineer building and maintaining our technical interview pipeline - from the question bank to AI-assisted scoring to full workflow automation - working alongside the team to grow this role into a broader automation function over time. This isn't a "call an API and ship it" job: you'll be designing the systems that add/calibrate interview questions and evaluate candidate answers, which means understanding how to prompt and structure AI models for consistency, how to measure and validate output quality against ground truth, and how to catch and correct the ways these systems get things wrong.

This is a role we expect to grow - the shape of that growth is still being defined.

We're looking for a junior engineer to be the primary engineer building and maintaining our technical interview pipeline - from the question bank to AI-assisted scoring to full workflow automation - working alongside the team to grow this role into a broader automation function over time. This isn't a "call an API and ship it" job: you'll be designing the systems that add/calibrate interview questions and evaluate candidate answers, which means understanding how to prompt and structure AI models for consistency, how to measure and validate output quality against ground truth, and how to catch and correct the ways these systems get things wrong.

This is a role we expect to grow - the shape of that growth is still being defined.

We're looking for a junior engineer to be the primary engineer building and maintaining our technical interview pipeline - from the question bank to AI-assisted scoring to full workflow automation - working alongside the team to grow this role into a broader automation function over time. This isn't a "call an API and ship it" job: you'll be designing the systems that add/calibrate interview questions and evaluate candidate answers, which means understanding how to prompt and structure AI models for consistency, how to measure and validate output quality against ground truth, and how to catch and correct the ways these systems get things wrong.

This is a role we expect to grow - the shape of that growth is still being defined.

• Add, revise, and retire technical interview questions across roles/levels/skill areas
• Maintain question metadata (difficulty, topic tags, expected answer criteria, role relevance)
• Create variant/randomized versions of questions to reduce answer-sharing risk
• Build validation checks (automated and manual) to catch incorrect, ambiguous, or miscalibrated questions before they enter the repository
• Build automation to update problem sets and recalculate average difficulty ratings as new results come in
• Review and update questions periodically to prevent leakage/staleness and keep content aligned with actual job requirements

• Architect scoring workflows that evaluate and compare candidate answers against reference answers/rubrics, grounded in a clear methodology for what "correct" and "well-scored" mean
• Calibrate and benchmark scoring prompts/models against human-graded samples, measuring accuracy and identifying systematic errors or bias
• Understand and account for model failure modes (inconsistency, hallucination, prompt sensitivity) and design safeguards around them
• Flag edge cases or low-confidence scores for human review rather than fully automating high-stakes decisions

• Automate the end-to-end workflow: question selection -> test delivery -> answer collection -> scoring -> reporting
• Help transition our source code and repositories to an internal Git platform
• Track test results to help establish scoring benchmarks over time

• Track scoring consistency and accuracy over time; report on pipeline health
• Watch for and mitigate AI evaluation bias across candidate demographics or answer styles
• Document the process, prompts, and scoring logic for auditability

• Apply the same automation and AI-integration skills developed here to other internal workflows as the role expands
• Partner…