Gen AI Architect

Location: Fremont, CA (Onsite)
Job Type: C2C
Remote: No

We are seeking an experienced Senior Technical Lead / Gen AI Architect with strong expertise in Langfuse v3, Azure AI services, and GenAI lifecycle management. The role will involve architecting, implementing, and optimizing advanced GenAI evaluation frameworks, ensuring enterprise-grade reliability, scalability, and observability for AI-driven solutions.

• Provide technical leadership to development teams, ensuring best practices, coding standards, and performance optimization in complex GenAI projects.
• Architect and deploy Langfuse v3 in production environments, including upgrades from Langfuse v2 with backward compatibility.
• Design and build modular components for prompt management, tracing, metrics, evaluation, and playground features using Langfuse v3.

Leverage Langfuse capabilities for:

Tracing end-to-end visibility of GenAI workflows

Metrics performance, latency, and usage analytics

Evaluation automated/manual scoring of outputs

Playground interactive prompt testing/debugging

• Integrate Azure AI Evaluation SDK into enterprise pipelines for scalable evaluation of GenAI models.
• Build reusable components and templates for evaluation across diverse RAG, multi-agent, and GenAI lifecycle workflows.
• Ensure scalability, observability, and traceability of evaluation frameworks in both offline and online environments.
• Collaborate with cross-functional teams (product, data science, and engineering) to align evaluation metrics with business KPIs.
• Conduct feasibility studies, recommend technical alternatives, and ensure compliance with architecture best practices.
• Stay current with emerging GenAI evaluation tools and methodologies (e.g., Tru Lens, W&B, Helicone).

• Hands-on expertise in Langfuse (including v3 features) and integrations.
• Proficiency with Azure AI Services and Azure AI Evaluation SDK.
• Strong understanding of LLMOps, prompt engineering, and GenAI lifecycle management.
• Experience with Retrieval-Augmented Generation (RAG), fine-tuning, and multi-agent orchestration.
• Proficiency in Python (Type Script or other GenAI-related languages is a plus).
• Experience with cloud-native architectures (Azure preferred).
• Familiarity with observability tools, tracing frameworks, and evaluation metrics for GenAI systems.
• Excellent communication, documentation, and stakeholder collaboration skills.

• Experience with additional observability tools such as Tru Lens, Weights & Biases (W&B), Helicone.
• Exposure to evaluation standards for enterprise AI adoption.
• Knowledge of multi-cloud and hybrid deployment strategies.