Skip to content
×

Member of Technical Staff, Inference Systems

Listing for: Ashby, Inc.
Full Time position Listed on 2026-11-20
Job specializations:
Salary/Wage Range or Industry Benchmark: 180000 - 280000 USD Yearly USD 180000.00 280000.00 YEAR

Job Description & How to Apply Below

About the role

We are looking for founding members to build the systems that serve our frontier models at scale: the kernels, runtimes, batching, KV cache, load balancing, and scheduling that determine how quickly and cost-effectively each token is delivered across thousands of GPUs. As agentic workloads drive inference demand sharply higher, solving these problems will define the next generation of AI infrastructure.

Your mandate is to unlock the next order-of-magnitude gain in serving performance. You will work across the serving stack to improve latency, throughput, and cost per token, and when it is the highest-leverage path, go deeper into runtimes, kernels, quantization, or speculative decoding. We’re looking for builders who are energized by owning these outcomes end to end, rather than staying in one layer.

We built TensorRT-LLM and other tools that have shaped the field. As a founding engineer, you will help set the next standard. Our team is flat, the problems are consequential, and you will carry work from first principles and prototype to production serving.

About you
  • You chase performance wherever it lives, from kernels to serving systems to the model itself, and you think in terms of the whole system rather than a single layer

  • You would rather own a metric end to end (latency, throughput, cost per token) than a single component

  • You are drawn to the risk and upside of an early-stage startup

  • You embrace ambiguity and have a strong bias for action

  • You operate with a tech-lead mentality regardless of title or years of experience

  • You treat AI as a force multiplier: fluent with coding agents and modern AI/ML workflows, you fold them into how you ship, and you are willing to push the boundary of what’s possible with agents

Requirements
  • Experience optimizing latency- and throughput-critical inference across the full stack, from kernels and systems to the model layer

  • Familiarity with modern GPU systems and deployment architectures, including HGX B300 and GB300 NVL
    72

  • Familiarity with serving infrastructure, gateways, and inference stacks such as Kubernetes, NVIDIA Dynamo, Kong, SMG, PyTorch, vLLM, SGLang, and TensorRT-LLM

  • Working knowledge of modern LLM and multimodal architectures, including the quantization and speculative-decoding tradeoffs behind them

  • Production proficiency in Python and at least one systems language:
    Rust, C++, or Go

Bonus
  • Hands-on experience authoring kernels with CUDA, Triton, or PTX

  • Experience profiling, tracing, and debugging across model, runtime, kernel, and network layers using tools such as Nsight Systems (nsys)

  • Experience with batching, KV-cache management, and scheduling strategies

  • Knowledge of quantization formats and algorithms, including NVFP4, FP8, PTQ, QAT, and QAD

  • Familiarity with system-level inference optimization for NVL
    72-scale deployments, including wide expert parallelism, disaggregated prefill and decode, and related high-bandwidth interconnect techniques

  • A track record of running inference workloads across 1,000+ GPUs

  • A background in HPC or other high-performance systems

  • Contributions to open-source ML or systems infrastructure

Compensation & Benefits
  • Highly competitive salary and equity

  • Comprehensive health coverage for you and your dependents

  • Strong 401(k) match

  • Meals and snacks on us + discretionary productivity budget

Strong preference for in-person in Palo Alto or Brooklyn.

About Inco

Inco AI (inco.ai) builds the agent-native inference platform for tomorrow’s token economics. We build the inference layer that makes advanced AI fast enough for real workflows and affordable enough to use nts are at the center of our work: when a single task chains dozens of model and tool calls, every gain in speed, cost, and reliability compounds.

We’re the researchers and…

To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction

Search for further Jobs Here:

SEARCH JOBS: 10+ Million Listings
(Try combinations for better Results! Or enter less keywords for broader Results)
Education Level
Experience Level (years)
Posted in last:
Salary