AI & Decision Engineer

Description

The Systems Performance Analysis Group (KBS) at the Johns Hopkins University Applied Physics Laboratory is seeking a Generative AI & Decision Engineer to design, build, and deploy AI-enabled analytical and decision-support applications for Naval and Air Force sponsors. You will lead the development of human–LLM interaction capabilities, Retrieval-Augmented Generation (RAG) systems, and intelligent agents that operate at scale to support a variety of applications.

This role will collaborate closely with internal stakeholders and external sponsors, including DoD organizations, to demonstrate and transition cutting-edge AI technologies into operationally relevant environments.

As an
AI & Decision Engineer, you will...

Architect, develop, and evolve a Generative AI-powered analytical applications that support human–LLM interactions and LLM-driven decision-making at scale.

• Design and implement LLM agents that can autonomously plan, reason, and act using tools such as RAG, analytical tools, and relevant databases.
• Apply prompt engineering techniques with state-of-the-art models to produce data-driven recommendations based operational and developmental data for a variety of systems.
• Design, implement, and manage RAG pipelines using vector databases and frameworks to integrate relevant documentation, prior analytical results, and large information repositories into LLM decision-making.
• Develop and maintain agents capable of: (1) Interpreting database schemas and generating SQL queries to answer user questions (SQL agents). (2) Translating natural language inputs into structured actions and game events stored in a database for downstream simulation and adjudication.
• Experiment with and evaluate prompting strategies to improve reasoning quality, robustness, and transparency of model outputs in high-consequence decision contexts.
• Integrate orchestration and observability tools (e.g., Prefect) to monitor LLM pipelines, track outputs, and provide real-time insight into system behavior during wargame execution.
• Fine-tune and adapt foundation models (e.g., Llama, Mistral) using AWS Sage Maker, Hugging Face TRL, and synthetic data generation (e.g., GPT-4 series) to optimize performance on sponsor-specific tasks such as deductive coding, domain knowledge transfer, etc.
• Engage with internal leadership and external sponsors to demonstrate capabilities, collect requirements, and potentially transition systems to operational users.
• Document system architectures, experiments, evaluation results, and operational guidance; prepare and present technical briefings and reports to technical and non-technical stakeholders.
• Collaborate in multidisciplinary teams (AI/ML, software, human factors, operations analysts) to integrate AI capabilities into broader analytic and operational workflows.

You meet our minimum qualifications for the job if you have...

• Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, Applied Mathematics, Data Science, or a closely related field.
• Experience building production or prototype applications involving large language models (LLMs) or other generative models.
• Proficiency in Python and at least one modern web/backend framework (e.g., FastAPI, Flask, Django).
• Experience with relational databases, including schema design and query development (e.g., PostgreSQL).
• Hands-on experience with at least one of:
  • Retrieval-Augmented Generation (RAG) systems.
  • LLM agent frameworks (tool use, ReAct, chain-of-thought–style prompting).
  • Vector databases (e.g., Qdrant, ChromaDB).
• Demonstrated ability to comprehend and synthesize complex technical or scientific information and make timely, well-reasoned decisions
• Strong written and oral communication skills, including…