Sr IT AI Software Engineer

Staff AI Software Engineer

Our Customer Support Tools team in IT at Palo Alto Networks is at the core of our business and is a true differentiator. We enable our employees and business to be successful, and our focus is on the next-generation of IT—cloud, mobile, and social. We are a team of disruptors, challenging the status quo and looking for new and better ways to solve problems.

We are a team of doers, getting things done and driving change from the ground up. We are a team of partners, collaborating with the business to help them achieve their goals. We are a team that is passionate about our work and our mission. We are looking for like-minded individuals to join our team and help us continue to make a difference.

We are seeking a dynamic Staff AI Software Engineer to lead the design and delivery of our next-generation scalable services. Leveraging Python, Django, FastAPI, React, and GraphQL within Google Cloud Platform, you will evolve our core architecture and drive our transition into AI-native engineering. As a technical leader, you will ensure our systems—including traditional microservices and emerging agentic workflows—are resilient, scalable, and innovative.

• Drive the strategic architectural vision and deployment of high-quality, scalable software assets using Python and React, ensuring long-term alignment with enterprise architecture.
• Architect and govern advanced agentic systems using the Vertex AI Agent SDK (or alternative frameworks like Lang Chain / Lang Graph) and A2A protocols to enable seamless interoperability between autonomous enterprise services.
• Establish enterprise-wide AI Evaluation (Evals) frameworks and automated benchmarking pipelines to rigorously measure, monitor, and optimize LLM/agent performance across metrics like accuracy, safety, latency, and cost.
• Define the organizational strategy and architecture for data retrieval pipelines, ensuring scalable and optimized utilization of vector embeddings, semantic search, and similarity-based retrieval patterns.
• Establish enterprise-wide prompt engineering standards and LLM security frameworks, safeguarding applications against vulnerabilities such as prompt injection through robust input/output validation models.
• Standardize and scale Model Context Protocol (MCP) servers to effectively bridge core enterprise IT data with LLM-driven applications across business units.
• Partner closely with technical executives, product management, and cross-functional engineering teams to align AI initiatives with overarching business objectives and drive high-impact results.
• Mentor and cultivate technical leaders and engineers across the organization, fostering a culture of excellence, applied learning, and accountability in full-stack and AI domains.
• Take ultimate accountability for cross-team project outcomes, ensuring AI and software solutions meet rigorous enterprise-grade standards for security, quality, and performance.

• Minimum of 9 years of related experience with a Bachelor's degree in Computer Science or a related field, or equivalent military experience.
• 9+ years of professional experience in backend development with Python, utilizing frameworks like Django or FastAPI to design and build fault-tolerant microservice architectures.
• Proven track record of architecting, deploying, and scaling production-grade Generative AI and LLM-powered applications.
• Hands-on experience with LLM orchestration and agentic frameworks (e.g., Lang Chain, Lang Graph, or native cloud Agent SDKs).
• Deep technical expertise in RAG architectures, including vector embeddings, semantic search, and working with vector databases.
• Demonstrated experience implementing LLM guardrails, prompt injection defenses, and robust AI evaluation (Evals) workflows.
• Demonstrated expertise in frontend development with JavaScript, Type Script, and React.
• Expert-level proficiency in containerization and orchestration using Docker and Kubernetes (GKE).
• Proven track record of architecting and delivering enterprise-grade software solutions from concept to production at scale.

• Deep, hands-on experience with Google Cloud Platform,…