Exam Development Manager

Description

The Examinations Development Manager is responsible for compliance with documented policies and procedures as it pertains to exam development and maintenance. This person will work with Subject Matter Experts (SMEs), stakeholders, vendors, and staff to develop and maintain examinations at standardized rigor for continued assurance of validity and reliability as well as fairness and impartiality and meeting relevant and required standards (e.g., ISO/ICE 17024, ASTM E2659).

The Examinations Development Manager is charged with meeting the needs of ISA’s current and future credential holders by managing the development and maintenance of all ISA’s credentialing examinations (including both written and performance-based examinations). This is accomplished by utilizing experience in exam development and maintenance, data analysis, psychometrics and testing, volunteer recruitment and engagement, and project management.

Examination Development & Maintenance

• Leads the design, development, revision, and maintenance of certification examinations.
• Manages examination development and publication projects from end to end through publication and implementation.
• Ensures examination questions are designed to measure competence based on, and consistent with, the scheme for each given credential.
• Ensures examination forms (including forms associated with performance-based examinations) have comparability of results for each single examination, both in content and difficulty, including validity of fail/pass decisions.
• Anticipates item and test development needs for all ISA examinations and build multi-year plans to meet those needs.
• Manages SME review and approval of examination items and exam forms in a timely and cyclical fashion according to documented policies and procedures, organization needs and industry best practices.
• Responsible for item bank management system, ensuring statistical data is available to inform and report on item and exam development activities and needs.
• Works with Publication and Documentation Lead and other key staff to ensure that relevant policies, procedures and communication is updated and accurate with regard to examinations (e.g., credentialing schemes, exam outlines, cut scores)
• Coordinates standard setting studies and implementation of passing scores.
• Verifies blueprint alignment, item accuracy, scoring integrity, and examination formatting.

Psychometrics, Quality & Continuous Improvement

• Collaborates with psychometric vendor(s) to ensure examinations meet accepted psychometric standards.
• Reviews item statistics and examination performance data to support continuous improvement. Monitors examination form quality using statistical analyses and candidate performance trends.
• Coordinates standard setting studies and implementation of passing score changes.
• Recommends improvements based on psychometric findings and stakeholder feedback.
• Maintains documentation supporting examination development decisions and audit readiness.

Subject Matter Expert (SME) & Committee Management

• Recruits, trains, and manages diverse panels of subject matter experts.
• Trains Subject Matter Experts (SMEs) on current best management practices for item development and review.
• Serves as main point of contact for translators, editors, and SMEs working on item development, review, and translation.
• Facilitates SME meetings (virtual and in-person) related to item and exam development and maintenance well as to ensure validity and reliability of both written and performance-based examinations.
• Facilitates item writing workshops and examination review meetings.
• Ensures SME activities are documented and follow established governance procedures.

Quality Assurance & Examination Security

• Establishes and implements safeguards for item banking, examination publication, and content handling.
• Complies with ISO/IEC 17024 to prevent fraudulent examination practices, monitor examination statistics to ensure the integrity of the examination process, and works with vendors (if applicable) to investigate security incidents, breaches, and other concerns related to exam security.
• Protects the…