Sr AI Engineer; Emerging AI

DAT builds agentic solutions for freight and logistics, and this role helps power the platform behind them. As a Sr AI Engineer (Emerging AI), you will design, operate, and continuously improve the AI infrastructure that supports model training, serving, evaluation, monitoring, and governance, with a strong focus on AWS scalability, reliability, and cost-aware production delivery.

• Design, build, and operate shared services that enable engineers and business users to ship models, including a model gateway for LLM access, inference endpoints for real-time and batch scoring, feature storage, vector search, and a common SDK.
• Architect and support scalable AI systems by writing design documents and driving alignment across Product and Engineering and Business users on how models get built and shipped at DAT.

• Architect and run scalable, reliable AI infrastructure on AWS using services including Bedrock, Sage Maker, EKS, Redpanda, MSK (Kafka), Lambda, and S3.
• Provision and manage infrastructure using Pulumi.

• Build offline and online evaluation systems to determine whether a model or prompt change improves outcomes, including regression suites, LLM-as-judge pipelines, A/B and shadow testing, and drift detection.
• Own inference cost and latency as first-class metrics, right-sizing GPU and serverless capacity and tuning batching, caching, and quantization while providing teams visibility into workload costs.
• Implement guardrails, prompt and output logging, PII handling, access controls, and model and dataset lineage to support security and customer data commitments.

• Drive adoption of modern software development practices across AI work, including automated testing, code reviews, CI/CD pipelines, and infrastructure-as-code.
• Mentor engineers and data scientists on production ML and distributed systems, raising the operational bar across teams building on the platform.
• Lead response and resolution for complex production incidents involving AI services, perform root cause analysis, and implement preventative measures.

• 10+ years of experience in software engineering, including substantial time as a senior or staff engineer owning production infrastructure relied upon by other engineering teams.
• 4+ years building and operating machine learning or AI platforms, taking models from notebook to production traffic and keeping them healthy.

• Strong development language skills ideally using Type Script/js, Java, or Go, along with the discipline to create libraries other teams adopt.
• Extensive AWS experience with production systems, ideally including EKS/Kubernetes, Lambda, Redpanda, MSK/Kafka, S3, and Secrets Manager, plus infrastructure-as-code with Pulumi, Terraform, or similar.
• Hands-on experience serving models at scale, including containerized inference, GPU scheduling, autoscaling, batching, and tradeoffs between real-time, streaming, and batch scoring.
• Practical experience with LLM systems in production, including retrieval-augmented generation, vector databases, prompt and context management, tool-calling or agent frameworks, and managed model APIs such as Amazon Bedrock.
• Experience building evaluation and monitoring beyond services, including offline eval sets, online metrics, drift and data quality checks, and clear definitions of model regression.
• Background in event-driven and distributed systems using Redpanda, AWS MSK, or Kafka.
• Fluency with observability and on-call health, designing dashboards, APM traces, logs, and alerts, defining SLOs, and using them to reduce incident frequency and MTTR.
• Experience establishing engineering standards, including PR…