Principal AI Architect

CoAdvantage is an HCM company providing payroll, ASO, and PEO services to 16,000 clients. We deliver payroll, benefits, HR compliance, time/PTO, and risk management solutions, and we are building a governed AI platform that will become a primary source of differentiation versus AI-native competitors.

We are standing up a parallel AI architecture- a microservice plane that talks to legacy systems via ETL contracts- anchored on three substrates (engineering knowledge graph, analytics feature store, customer knowledge store) and a governed multi-agent harness. The Principal AI Architect owns the technical shape of that platform.

You are the most senior individual contributor on the AI program. You make the load-bearing technical decisions that everyone else builds against.

• Own the reference architecture for the AI plane end-to-end: data substrates, model layer, agent harness, orchestration, application surfaces, observability, and the ETL boundary to core platform and adjacent systems (payroll, benefits, claims, ADO).
• Drive build-vs-buy decisions on the substrate trio- graph (Neo4j vs. Tiger Graph vs. managed), vector (pgvector vs. Pinecone vs. Azure AI Search), feature store (Feast vs. Databricks Feature Store vs. Tecton), and warehouse direction (Azure Fabric vs. Snowflake)- with written, defensible recommendations.
• Specify the multi-agent harness: agent envelopes (planning, tools, reflection, memory), reward functions and hacking watchlists, HITL triggers, handoffs, capability gradients, and the deterministic Orchestrator that controls them.
• Define tenant isolation, identity resolution, and AuthZ for the Customer Knowledge Store across three access tiers (Client Admin, WSE, Internal CoAd)- cross-tenant leakage is a catastrophic-tier risk and you own the controls that prevent it.
• Set the assurance program: input/output guardrails, red-team plan, evals, circuit breakers, immutable lineage, and the production to KG feedback loop.
• Partner with the Head of AI on the 18-month roadmap and the platform consolidation thesis.
• Own the runtime execution architecture for the agentic platform, including orchestration topology, state management, workflow durability, retry semantics, and failure recovery strategies across distributed multi-agent systems.
• Define platform-wide evaluation and reliability standards, including groundedness metrics, hallucination detection, behavioral regression testing, drift monitoring, degraded operating modes, and production resiliency requirements.
• Establish governance standards for AI platform operations, including inference cost optimization, data contracts and lineage, tenant-safe memory architecture, and security controls for regulated multi-tenant environments.
• You will write specs, code, and prototypes. You will not be a deck-only architect.

• AI-first coding. Claude Code, Copilot, and successor tools are the default development surface. Hand-coding without AI assistance is the exception, not the norm.
• Build your own agentic workflows. You will compose and operate your own multi-step agent pipelines- for code generation, spec mining, design review, eval authoring, migration analysis. If a workflow doesn’t yet exist for a job you do twice, you build it.
• Every workflow is testable. Every agent, every chain, every prompt that touches production has an eval harness, a regression suite, and a defined success criterion before it ships.
• Ambiguity is the job. You will get problems framed at the strategy level and return a sequenced, costed, testable plan with a working prototype.
• You estimate. Every recommendation comes with a timeline, a headcount ask, a confidence interval, and a list of the assumptions that could blow it up.
• You suggest the tools. Strong opinions, loosely held, written down.

• Publish a v1.0 reference architecture for the AI plane that the Head of AI signs and the CIO endorses.
• Deliver a build-vs-buy recommendation, with cost model and 18-month TCO, for the substrate trio.
• Stand up a working prototype of one agent in the harness (Codi or Tespi from the v0.4 spec) with its eval suite, reward function, and HITL path- end-to-end on a real CoAd…