Principal AI​/ML Engineer; Full-Stack, Applied AI

About Sher Innovations

Sher Innovations builds enterprise software for complex, operations-heavy organizations. We focus on production reliability, governed security, and disciplined delivery. Our teams build platforms and applications that stay up, stay secure, and evolve without chaos.

Build Superior
· People-First
· Partner
· Impact.

Integrity, One Team, Empathy, Ownership, Craftsmanship, Drive.

At Sher Innovations, your work runs in production, not a lab. We’re expanding our engineering capacity to meet demand for reliable, governed systems in operations-heavy environments. We’re looking for senior engineers who take end-to-end ownership, ship with discipline, and raise the bar on production readiness.

We’re building a team that holds itself and each other to high accountability and a high bar for quality.

As a Principal AI/ML Engineer (Full-Stack, Applied AI) at Sher Innovations, you will build end-to-end applications that operators can trust in production. This is not a research-only role. You will ship complete systems across frontend, backend, data, and deployment, embedding AI where it creates real leverage inside the workflow.

You will lead the technical design of AI-enabled capabilities across the stack, including product integration, services, data pipelines, evaluation, model serving, and monitoring. You’ll partner closely with engineering leadership and customer stakeholders to define success metrics, validate performance, and deliver production-ready releases.

You’ll help set the standard for how we build and operate AI in enterprise and regulated environments: governed, observable, and built for operational reality.

We operate in a strategic hybrid model built for high-accountability teams. Most focused work is remote. We use intentional in-person time for architecture reviews, working sessions, and stakeholder alignment.

• Design and deploy LLM-enabled workflows that automate complex operational tasks with clear guardrails.
• Deliver production-grade model serving patterns with reliability, latency, and cost controls.

• Architect scalable data pipelines, feature/embedding workflows, and retrieval patterns where appropriate.
• Establish evaluation harnesses, regression tests, and quality gates for model changes.

• Implement monitoring for quality, drift, failure modes, and operational performance.
• Drive incident-ready practices: runbooks, rollback paths, and measurable SLOs.

• Set engineering standards for ML systems design, code quality, and operational readiness.
• Mentor through design reviews, code reviews, and pragmatic technical guidance.

• Work directly with customer and partner stakeholders to translate requirements into shipped outcomes.
• Communicate tradeoffs clearly, especially around risk, safety, cost, and reliability.

We are seeking a principal-level engineer who operates with a leadership mindset and strong ownership:

• Production-first ML mindset:
  You build with evaluation, monitoring, and failure modes in mind.
• Systems thinking:
  You design end-to-end architecture across data, models, services, and interfaces.
• Pragmatic execution:
  You choose approaches that are maintainable and operable, not just impressive.
• Ownership:
  You take responsibility through delivery and beyond.
• Communication:
  You can explain technical decisions clearly to engineers and non-engineers.

• 8+ years building production software systems, including ownership of system design and delivery.
• Proven experience deploying ML/LLM-enabled capabilities in production environments.
• Strong Python experience and comfort designing backend services and APIs.
• Experience with evaluation and monitoring (quality, drift, performance, cost) for ML systems.
• Strong engineering fundamentals: testing, observability, and operational readiness.

• Cloud/infrastructure production experience with GCP, AWS, Azure, or Linode
• Infrastructure and deployment experience with Terraform, Kubernetes, Docker, and VMs
• Backend framework experience with Phoenix, FastAPI, or Django (or equivalent)
• Data systems experience with relational and No
  
  SQL, plus Redis
• Streaming or real-time systems experience with Kafka, MQTT, or similar
• Network and traffic management familiarity (load balancing, HAProxy, etc.)
• Mobile experience (Swift, Kotlin, or React Native)
• Experience in regulated or operations-heavy domains

• Bachelor’s or Master’s in Computer Science (or equivalent professional experience)

• Competitive Compensation:
  Salary + Equity
• Health & Wellness:
  Medical, Dental, and Vision
• Future Planning: 401(k)
• Flexibility:
  Unlimited PTO and Strategic Hybrid work model
• Perks:
  Daily lunches and professional development opportunities

Sher Innovations is an equal opportunity employer.