Sr Staff IT AI Software Engineer

Job Summary

Our Customer Support Tools team in IT at Palo Alto Networks is at the core of our business and is a true differentiator. We enable our employees and business to be successful, and our focus is on the next‑generation of IT—cloud, mobile, and social. We are a team of disruptors, challenging the status quo and looking for new and better ways to solve problems.

We are a team of doers, getting things done and driving change from the ground up. We are a team of partners, collaborating with the business to help them achieve their goals. We are a team that is passionate about our work and our mission. We are looking for like‑minded individuals to join our team and help us continue to make a difference.

We are seeking a dynamic Staff AI Software Engineer to lead the design and delivery of our next‑generation scalable services. Leveraging Python, Django, FastAPI, React, and GraphQL within Google Cloud Platform, you will evolve our core architecture and drive our transition into AI‑native engineering. As a technical leader, you will ensure our systems—including traditional microservices and emerging agentic workflows—are resilient, scalable, and innovative.

• Drive the strategic architectural vision and deployment of high‑quality, scalable software assets using Python and React, ensuring long‑term alignment with enterprise architecture.
• Architect and govern advanced agentic systems using the Vertex AI Agent SDK (or alternative frameworks like Lang Chain / Lang Graph) and A2A protocols to enable seamless interoperability between autonomous enterprise services.
• Establish enterprise‑wide AI Evaluation (Evals) frameworks and automated benchmarking pipelines to rigorously measure, monitor, and optimize LLM/agent performance across metrics like accuracy, safety, latency, and cost.
• Define the organizational strategy and architecture for data retrieval pipelines, ensuring scalable and optimized utilization of vector embeddings, semantic search, and similarity‑based retrieval patterns.
• Establish enterprise‑wide prompt engineering standards and LLM security frameworks, safeguarding applications against vulnerabilities such as prompt injection through robust input/output validation models.
• Standardize and scale Model Context Protocol (MCP) servers to effectively bridge core enterprise IT data with LLM‑driven applications across business units.
• Partner closely with technical executives, product management, and cross‑functional engineering teams to align AI initiatives with overarching business objectives and drive high‑impact results.
• Mentor and cultivate technical leaders and engineers across the organization, fostering a culture of excellence, applied learning, and accountability in full‑stack and AI domains.
• Take ultimate accountability for cross‑team project outcomes, ensuring AI and software solutions meet rigorous enterprise‑grade standards for security, quality, and performance.

• Minimum of 9 years of related experience with a Bachelor’s degree in Computer Science or a related field, or equivalent military experience.
• 9+ years of professional experience in backend development with Python, utilizing frameworks like Django or FastAPI to design and build fault‑tolerant microservice architectures.
• Proven track record of architecting, deploying, and scaling production‑grade Generative AI and LLM‑powered applications.
• Hands‑on experience with LLM orchestration and agentic frameworks (e.g., Lang Chain, Lang Graph, or native cloud Agent SDKs).
• Deep technical expertise in RAG architectures, including vector embeddings, semantic search, and working with vector databases.
• Demonstrated experience implementing LLM guardrails, prompt injection defenses, and robust AI evaluation…