Principal Software Engineer - Agentic SOC

Heavy use of LLMs and AI coding tools to build agentic systems; role centers on building and evaluating LLM-based agents and rapid prototyping with AI.

Lead the design and implementation of an Agentic SOC that uses LLM-based agents to investigate security signals end-to-end, augmenting SOC analysts by automating high-volume, repetitive work while preserving security, auditability, and human handoffs. Own system architecture, reliability, evaluation, and technical leadership for production-grade agent behavior and decision-making.

Huntress is hiring a Principal Software Engineer to lead the design and delivery of an Agentic Security Operations Center (SOC). The role owns the architecture and technical direction of an LLM-based agent system that investigates signals end-to-end, assembles evidence, and decides when to act autonomously versus hand off to a human analyst. You will work closely with SOC analysts, detection engineers, product, and other engineering teams and drive production-grade reliability, auditability, and tenant isolation.

• Own the architecture of agentic investigations: signal ingestion, investigation state management, tool surfaces for agents, and final investigation artifacts (verdict, timeline, scope, evidence).
• Design and build LLM-powered agents that replicate strong analyst behavior and make appropriate decisions about escalation and handoff.
• Build LLM-based pre-processing to summarize, correlate, and surface what matters from analyst tools and telemetry.
• Embed a security mindset in the agent design: trust boundaries, tenant isolation, attack resistance, and full audit trails.
• Define and own evaluation: labeled datasets, offline evaluation, regression gates, and metrics to measure non-deterministic system quality over time.
• Decide, based on data, what the system can act on autonomously and what requires human review; evolve that decisioning responsibly.
• Engineer for production: reliability, latency, cost at scale, and robustness to model failures or changes.
• Prototype and derisk uncertain approaches, then hand off production-ready designs to the team.
• Represent the team technically across SOC leadership, product, other engineering teams, and executives; drive alignment and protect team scope.
• Mentor engineers, perform thoughtful code review, and raise both the team’s baseline and ceiling through patterns and tooling.

• 15+ years building complex software products with significant time as a technical lead on production systems.
• Hands‑on experience designing, shipping, and operating LLM‑based agents in production (tool use, context management, structured output, guardrails, evaluation) and an understanding of failure modes.
• Experience building evaluation frameworks for non-deterministic systems (labeled datasets, offline evals, regression gates).
• Security mindset with experience handling untrusted input, multi‑tenant isolation, and auditable trails.
• Experience building systems that make automated decisions with real consequences and sound judgment about what to automate.
• Deep backend expertise in distributed systems, queues, durable workflows, and concurrency; designing for throughput and correctness under load.
• Strong skills in one or more backend languages and ability to learn new ones quickly; primary stack is Ruby on Rails and fluency is expected.
• Experience with public cloud environments (AWS, Azure, or similar) and data stores such as Postgres and Redis.
• Familiarity with AI coding tools (example: Claude Code).
• BS or MS in Computer Science/Engineering or equivalent experience.

• Experience in SOCs, incident response, threat hunting, or detection…