Principal Software Engineer - LLM Optimization

At JPMorgan

Chase, we are building the infrastructure that powers the next generation of enterprise AI — and we need the best minds in LLM inference to help us do it. This is your opportunity to work at the intersection of cutting-edge machine learning and large-scale production systems, directly influencing how one of the world's largest financial institutions deploys and optimizes AI at scale.

As a Principal Software Engineer at JPMorgan

Chase within the AI/ML Data Platform team, you will serve as the firm's deepest technical voice on LLM inference performance — owning optimization strategy, benchmarking rigor, and efficiency  will work directly with senior engineering leadership to shape how our platform evolves, ensuring every model we serve is fast, cost-efficient, and production-ready. This is a high-visibility individual contributor role where your technical decisions will have direct, measurable impact on the firm's AI capabilities

• Own systematic benchmarking and performance characterization across all production LLM workloads. Establish reproducible baselines, catch regressions early, and quantify the impact of every configuration change before it touches production

• Design and execute quantization experiments — FP8, INT8/INT4 (GPTQ/AWQ), next-generation precision formats on current hardware — measuring accuracy delta, throughput improvement, memory reduction, and cost-per-token impact

• Drive speculative decoding strategy across the model portfolio: draft model, n-gram, and multi-token prediction approaches. Own acceptance rate measurement and per-workload configuration recommendations

• Build and maintain a GPU efficiency scorecard: utilization, memory headroom, cost per 1K tokens, and waste identified — giving leadership a data-driven view of platform efficiency at all times

• Benchmark our platform against external providers and published industry numbers — know what good looks like, and close the gap

• Lead inference engine upgrade evaluations: new scheduler architectures, async tensor parallelism, disaggregated prefill/decode, advanced speculative decoding — systematic validation before production promotion

• Collaborate with the EKS and disaggregated serving teams on KV-cache optimization, prefix caching strategies, and multi-node serving architecture

• Design and run GPU chaos engineering: induced failure scenarios, hardware diagnostic monitoring, detection and recovery measurement

• Architect and govern agentic AI-enabled engineering workflows (using enterprise-authorized tools within the work environment) to improve delivery speed, code quality, and operational outcomes at scale (e.g., AI-driven PR review assistance, test generation/maintenance, release readiness checks, incident triage and root-cause acceleration), while defining guardrails for validation, security, resiliency, and reuse across teams.

• Apply knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation at scale.

• Formal training or certification on software engineering concepts and 7+ years applied experience

• Deep, hands‑on experience with LLM inference systems — vLLM, TensorRT‑LLM, SGLang, LLM‑D or equivalent production serving engines

• Strong grasp of GPU memory architecture: KV cache sizing and dynamics, memory‑bandwidth vs compute bottlenecks, the practical implications of quantization at inference time

• Experience with quantization techniques and their real‑world tradeoffs at scale

• Familiarity with speculative decoding and the variables that drive acceptance rates in production workloads

• Rigorous benchmarking instincts — GuideLLM, custom harnesses, or equivalent. Every claim has a number behind it

• Comfort operating in cloud GPU infrastructure at scale (AWS; EKS, managed inference services)

• Demonstrated awareness of the LLM inference competitive landscape, with a track record of applying industry benchmarks to drive platform improvements communicate technical trade‑offs clearly to senior engineering and business stakeholders — this…