LLM Specialist

Job Overview

Pen Fed is hiring a (Hybrid) LLM Specialist at our Tysons, Virginia location. The LLM Specialist serves as Pen Fed's subject matter expert for Large Language Models (LLMs), Generative AI technologies, and emerging foundation models. This role is responsible for evaluating, selecting, implementing, securing, and optimizing LLM solutions that support business objectives, enhancing member experiences, improving operational efficiency, and drive employee productivity.

The incumbent partners closely with business leaders, Technology, Information Security, Risk Management, Compliance, Data, and AI Engineering teams to translate business requirements into effective AI solutions. The LLM Specialist evaluates technical, security, performance, and cost considerations across AI models and platforms while ensuring LLM deployments align with enterprise AI management, responsible AI principles, privacy requirements, and regulatory expectations. This role serves as a trusted advisor on LLM technologies, helping the organization understand model capabilities, limitations, risks, and opportunities while supporting the successful adoption of Generative AI across the enterprise.

Reasonable accommodation may be made to enable individuals with disabilities to perform the essential functions. This is not intended to be an all-inclusive list of job duties, and the position will perform other duties as assigned.

• Evaluate and recommend Large Language Models (LLMs), foundation models, and Generative AI platforms based on business requirements, technical capabilities, security requirements, and operational considerations.
• Assess models across criteria including accuracy, latency, context window, performance, scalability, explainability, integration requirements, and business fit.
• Analyze and communicate technical tradeoffs among commercial, open source, hosted, and enterprise AI solutions.
• Provide recommendations regarding model architecture, deployment strategy, and solution design for business use cases.

• Partner with business units and technology teams to identify, define, and evaluate AI use cases.
• Translate business requirements into technical specifications and AI solution recommendations.
• Design and evaluate prompting, retrieval-augmented generation (RAG), fine-tuning, and model orchestration approaches when appropriate.
• Support implementation of AI-enabled applications, workflows, copilots, agents, and knowledge management solutions.
• Collaborate with AI Engineers and solution teams to ensure effective deployment and adoption of AI solutions.

• Evaluate and communicate cost implications associated with AI platforms, models, and deployment approaches.
• Analyze token consumption, inference costs, hosting models, licensing structures, and operational expenses.
• Develop financial analyses and cost models that enable business stakeholders to make informed investment decisions.
• Recommend optimization strategies that balance business value, performance scalability, and cost efficiency.

• Assess security risks associated with LLM implementations, including prompt injection, data leakage, unauthorized access, model misuse, adversarial attacks, and other emerging threats.
• Collaborate with Information Security, Risk Management, Compliance, and Legal teams to ensure AI solutions operate within established enterprise standards and policies.
• Support the evaluation of AI solutions for privacy, security, and regulatory compliance requirements.
• Participate in risk identification, mitigation planning, monitoring activities, and remediation efforts related to AI technologies.
• Assist in establishing controls that support responsibility and secure AI deployment across the enterprise.

• Monitor AI model performance, effectiveness, reliability, and operational outcomes after deployment.
• Identify performance degradation, emerging vulnerabilities, and opportunities for improvement.
• Recommend model updates, technology enhancements, and alternative solutions as AI capabilities evolve.
• Maintain awareness of advancements in Generative AI, LLMs, model evaluation methodologies, and industry best practices.

• Develop guidance, standards, job aids, and documentation related to LLM usage and Generative AI best practices.
• Educate business and technical stakeholders on AI capabilities, limitations, risks, and responsible usage considerations.
• Support AI literacy efforts and enterprise training initiatives.
• Serve as a trusted advisor and technical resource for enterprise AI initiatives.

• Maintain knowledge of and ensure adherence to all applicable federal and state laws, regulations, and Pen Fed policies, procedures, and standards.
• Support compliance with enterprise AI management, information security, privacy, risk management, and data governance requirements.
• Assist…