On-Premise LLM Inference & GPU Systems Engineer

North Carolina, Charlotte

06/05/2026

Contract

Active

We are seeking an On-Premise LLM Inference & GPU Systems Engineer to build, optimize, and support a large-scale enterprise Generative AI infrastructure environment. This role is focused exclusively on Large Language Model (LLM) inference operations within a private on-premises ecosystem utilizing NVIDIA H200 GPU clusters and Open Shift AI. The ideal candidate will possess deep expertise in GPU runtime optimization, inference serving platforms, Kubernetes-based orchestration, and production-scale deployment of open-source LLMs.

This position will be responsible for maximizing inference performance, operational efficiency, and platform reliability across enterprise AI workloads.

• Design, deploy, and maintain large-scale on-premises LLM inference infrastructure supporting enterprise Generative AI workloads.
• Optimize runtime performance of token generation pipelines, including prefill/decode optimization and KV cache management.
• Deploy, configure, and manage inference serving platforms such as vLLM and TensorRT-LLM.
• Optimize GPU utilization, throughput, batching strategies, latency, and resource efficiency across production inference environments.
• Manage workload scheduling and orchestration using Kubernetes-based GPU orchestration platforms and RunAI.
• Oversee the complete lifecycle of open-source language models, including onboarding, deployment, version management, monitoring, and retirement.
• Manage and support enterprise Hugging Face model deployment workflows and operational processes.
• Operate, maintain, and optimize the Open Shift AI ecosystem supporting Generative AI applications and services.
• Monitor platform performance, identify bottlenecks, and implement optimization strategies to improve inference efficiency and scalability.
• Collaborate with AI, platform engineering, infrastructure, and operations teams to ensure reliable service delivery.
• Implement operational best practices related to platform availability, monitoring, security, and governance.
• Develop automation, deployment processes, and operational procedures to support platform scalability and maintainability.
• Troubleshoot and resolve infrastructure, inference, performance, and deployment issues across the AI ecosystem.
• Create and maintain technical documentation, operational runbooks, and platform standards.

• 5+ years of experience as an LLM Systems Engineer, AI Infrastructure Engineer, AI Platform Engineer, or related role.
• 5+ years of hands‑on experience supporting NVIDIA GPU environments and runtime optimization techniques.
• Experience optimizing token generation pipelines, including KV cache management and prefill/decode optimization strategies.
• Strong experience deploying and managing inference frameworks such as vLLM and TensorRT-LLM.
• 3+ years of experience with Open Shift AI and containerized AI platform operations.
• 3+ years of experience with GPU orchestration technologies, including RunAI and Kubernetes-based environments.
• Experience deploying, managing, and supporting open-source Large Language Models in production environments.
• Proven experience managing the Hugging Face model lifecycle, including onboarding, deployment, version management, and retirement.
• Strong understanding of AI inference architectures, GPU resource management, workload optimization, and performance tuning.
• Experience working with containerization technologies, Kubernetes, and cloud-native application platforms.
• Strong troubleshooting, performance analysis, and problem‑solving skills.
• Excellent communication and collaboration skills with the ability to work across infrastructure, platform, and AI engineering teams.

• Experience supporting enterprise-scale Generative AI platforms and private AI infrastructure environments.
• Experience optimizing large-scale LLM inference workloads in highly regulated or secure environments.
• Knowledge of AI observability, monitoring, logging, and performance analytics tools.
• Experience implementing infrastructure automation and operational tooling for AI platforms.
• Familiarity with enterprise governance, security, and compliance practices for AI workloads.
• Experience supporting multi-cluster Kubernetes or Open Shift environments.
• Knowledge of emerging trends and best practices in LLM inference optimization and AI platform engineering.