AI Architect

The Information is the go-to source of in-depth reporting for the most influential leaders in technology and business. Founded in 2013 and headquartered in San Francisco, our original, high-quality journalism has the power to inform the most consequential decisions shaping our future, and we’ve built a community of 700,000 active readers who depend on us to do just that. We have a financially healthy business, plenty of capital, and big ambitions to grow our team and business.

We're looking for an AI Architect to lead the design, evolution, and scaling of our AI pipeline infrastructure. You'll take ownership of a backend platform built to orchestrate AI workflows at scale, and extend it to support a growing set of high-fan-out AI features that are central to our product strategy.

Our AI infrastructure is built around an internal, API-only backend service responsible for orchestrating AI pipelines end to end. It runs on a modern web framework with a background job processing system, backed by a relational database, and is designed around durable, observable, idempotent jobs rather than ad hoc scripts. Some early ai workflows still run on a separate orchestration framework and are being progressively migrated into this platform.

• An internal, API-only backend application deployed on a cloud PaaS (staging and production environments)
• A background job processing system with multiple queues and a durable message broker
• A dedicated relational database for pipeline state and history
• A job base class/pattern that provides idempotency guards, status-transition state machines, retry-with-backoff, structured logging, and error reporting on failure
• Authenticated API access for inbound integrations, with secure credential management for outbound integrations
• Multiple LLM providers for generation, structured output, and embeddings
• A vector database for similarity search and matching at scale
• Integrations with adjacent internal systems (content/CMS, notifications, messaging/chat tooling, tracing and error-monitoring platforms)

• Maintain and operate all existing AI pipelines running on the platform
• Complete the migration of remaining workflows from the legacy orchestration framework to the primary platform
• Own on-call response for AI pipeline failures, including failed-job triage and retries
• Manage LLM provider relationships, API key rotation, cost tracking, and model upgrades

• Design and implement new high-fan-out AI pipelines on the existing platform — built to support future horizontal workflows (many generations across many users/items) without significant rework
• Establish patterns and conventions for onboarding new pipelines, including registry entries, job sub classing, batch fan-out, and automated reporting
• Drive architectural decisions around data storage strategy, queue partitioning, concurrency throttling, and cost controls as pipeline volume grows
• Evaluate and integrate new LLM providers and embedding models as the landscape evolves, while maintaining backward compatibility with existing vector data
• Build observability and operational tooling — extending the primary ops dashboard with custom reporting, cost tracking, and alerting as needed

• Partner with product, editorial/content, and growth teams to translate product requirements into pipeline designs
• Work across the engineering organization to integrate AI pipelines with the broader technical stack
• Mentor engineers on AI pipeline patterns, prompt engineering, and platform architecture
• Own the technical proposal process for new pipelines and major infrastructure changes

• Deep backend…