Agentic AI Systems Evaluation Engineer

At HP, you'll have a chance to create tools, technology, and solutions that reshape the way the world works in the future. If you're looking to join a company that allows you to connect with a network of professionals eager to support you in doing your best work, we want to talk to you. Our legendary culture guides every employee toward success-fostering collaboration and driving innovation.

Operating in over 170 countries, HP is always creating new services, products, and capabilities giving you more opportunities to advance your career. Here, innovation is the key to professional development and career mobility.

When you join HP, you're joining a company that believes every voice matters and that we all deserve a seat at the table. From the boardroom to factory floor, we create a culture where everyone is respected and where people can be themselves. You will be part of a global laboratory where different perspectives and experiences will help you solve problems in new ways.

This is where you can build a long and wide-ranging career.

We are looking for a Systems Evaluation Engineer to continuously assess whether our AI software delivers the customer experiences and system performance it is designed to achieve. You will connect user stories and product goals to measurable outcomes, develop evaluation methodologies, and interpret results throughout development.

Working closely with product, architecture, and engineering teams, you will evaluate AI workloads, compare model and runtime implementations, and explain how technical performance affects the customer experience. Your findings will help teams understand trade-offs, identify gaps, and make better design and implementation decisions.

This role requires deep technical understanding of AI inference and PC systems, combined with strong analytical judgment. You will own the evaluation approach and interpretation of evidence, providing continuous feedback that shapes the developing solution.

• Translate user stories and intended customer experiences into measurable success criteria, evaluation scenarios, and performance targets.
• Design repeatable benchmarking frameworks that assess AI workload effectiveness and system behavior throughout development.
• Compare model formats, model runners, runtime implementations, and hardware configurations using representative workloads and controlled measurements.
• Analyze inference behavior, including prompt processing, time to first token, token generation rate, latency, and memory consumption.
• Evaluate system responsiveness and resource utilization under individual and concurrent AI workloads, including effects on other PC activities.
• Interpret benchmark and telemetry data to distinguish meaningful improvements, regressions, measurement variability, and gaps in evaluation coverage.
• Provide actionable feedback to architects and engineers, connecting technical findings to customer impact and recommending further investigation.
• Build automation and reporting tools that make evaluations reproducible and results accessible to development teams.
• Communicate performance trends, trade-offs, and progress against customer outcomes through technical reviews and concise executive reports.

• 7+ years of experience in systems evaluation, performance engineering, software engineering, advanced validation, or a related field.
• Strong understanding of AI systems and inference workloads, including model execution, prompt processing, and runtime behavior.
• Experience designing benchmarking methodologies, selecting meaningful metrics, and interpreting complex system performance data.
• Familiarity with model formats such as ONNX and GGUF, model runners, and the implications of different execution approaches.
• Understanding of system-level performance, including compute utilization, memory constraints, concurrency, and interactions between AI workloads and the operating system.
• Strong programming and automation skills in Python, C++, or another relevant language, with experience building evaluation tools or data analysis workflows.
• Ability to translate technical measurements into customer impact and communicate actionable findings across product, architecture, and engineering teams.
• Bachelor's degree in computer science, statistics, software engineering, a related field, or equivalent practical experience.

• Experience evaluating AI workloads on Windows and Linux PCs or other edge devices.
• Knowledge of attention…