AI Engineer III

POSITION OVERVIEW

NBME is seeking a highly skilled AI Engineer to join our team and take ownership of building, maintaining, and scaling AI-driven systems. This role is critical for translating research innovations into production-ready pipelines and tools that support multiple programs, products, and services. The AI Engineer will work closely with research, IT, and product teams to operationalize AI solutions, ensuring they are reliable, secure, and optimized for performance.

This role has been designated as primarily remote, meaning it requires little onsite attendance. While this role can be designated as remote, you are offered the flexibility to select if you prefer to work primarily remotely, hybrid, or onsite.

At NBME, we continue to innovate and improve how we fulfill the evolving needs of the healthcare community. This commitment starts and ends with the people  recruiting and empowering talented individuals from various disciplines and backgrounds, which includes professionals with diverse life experiences, abilities, and perspectives, NBME can take a well-informed, robust approach to advancing medical education and assessment for years to come.

RESPONSIBILITIES:

* Design, implement, and maintain production-grade AI and NLP pipelines, including model training, deployment, monitoring, and updates.

* Collaborate with research teams on pre-production research and to operationalize prototypes, ensuring reproducibility, scalability, and robustness.

* Build and maintain infrastructure for embeddings, large language models, and other AI assets.

* Optimize AI systems for performance, reliability, and cost-efficiency.

* Implement and maintain monitoring, logging, and alerting for AI systems in production.

* Ensure compliance with data security, privacy, and organizational standards.

* Support multiple product teams by providing AI engineering expertise, tools, and best practices.

* Contribute to technical documentation, knowledge sharing, and internal training initiatives.

QUALIFICATIONS:

* Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field.

* A minimum of 3 years of experience in AI/ML engineering, including model deployment, MLOps, system monitoring, and production pipelines.

* Proficiency with Python and AI/ML libraries (e.g., PyTorch, Tensor Flow, Hugging Face Transformers).

* Experience with cloud platforms (AWS, GCP, Azure) and containerization (Docker, Kubernetes).

* Familiarity with vector databases, embeddings, and large-scale NLP applications.

* Strong problem-solving skills and ability to work collaboratively in a cross-functional team.

* Excellent communication skills for technical and non-technical stakeholders.

Preferred Qualifications:

* Experience in educational assessment or healthcare-related AI applications.

* Experience with Agentic LLM architectures for complex task completion.

About NBME

NBME is a not-for-profit organization that specializes in the creation of assessments and learning tools for physicians and health professionals. Our mission is to advance assessment of these professionals to achieve optimal care for all, supporting the development of a highly effective, diverse and compassionate health care workforce.

Founded in 1915, NBME develops and manages the United States Medical Licensing Examination with the Federation of State Medical Boards. We offer a comprehensive portfolio of assessment products for every stage of the medical school journey and provide assessment services to various health profession organizations.

We are dedicated to advancing innovative assessment approaches through research, collaboration and contributions to the medical education and assessment communities. In 2024, we expanded our assessment capabilities to include simulation through the acquisition of MedVR Education.

Learn more on NBME's website.

NBME's Community Collaborations and Contributions

NBME believes that a key path to meeting our mission is collaborating with and supporting our communities. Our Community Collaborations and Contributions programs provide resources and recognition to medical educators and researchers advancing the field of assessment throughout their careers. In addition, we invest in fee assistance, scholarships and pathway programs to aid learners and help support increased representation in health care.

Headquartered in Philadelphia, NBME also gives to local organizations that advance health equity and access and positively impact social determinants of health.

Compensation we are offering for this position is at 118,160 (min) -- $147,700 /year. Please note that the offered rate for this position typically aligns with the minimum to midrange of this grade, but it can vary based on the successful candidate's qualifications and experience, department budget, and an internal equity review.

The NBME offers competitive salaries, excellent benefits, and a rewarding work environment. Excellent Benefits include:
Healthcare, Dental, Prescription, and Vision…