Manager, Speaking Test Development

The Center for Applied Linguistics is a nonprofit organization promoting access,equity and mutual understanding for linguistically and culturally diverse people around the world.

The mission of the Center for Applied Linguistics (CAL) is to promote language learning and cultural understanding by serving as a trusted source for research, resources, and policy analysis. Through its work, CAL seeks solutions to issues involving language and culture as they relate to access and equity in education and society around the globe.

CAL is located in Washington, D.C. For this position, the employee may choose to work in their preferred combination of in person at the D.C. office in a hoteling capacity, or remotely (up to 100% remote). CAL does not compensate or reimburse candidates for relocation expenses. This position is not open to candidates residing in New York, New Jersey, Pennsylvania, Texas, California, or the U.S. Territories.

The Manager, Speaking Test Development, is responsible for overseeing the development of Speaking test items and ancillary materials for the WIDA suite of assessments, which are large-scale English language proficiency assessments for multilingual learners in grades 1-12. This position involves reviewing, revising, and approving test items (linguistic content, graphics, scripts, audio files) that align to the WIDA English Language Development Standards Framework, with a focus on the language domain of Speaking.

The Manager, Speaking Test Development, oversees the test development process from start to finish. This position develops test refreshment plans, timelines, and procedures; provides comprehensive feedback to test developers; and signs off on test items content at critical points in the development process. This position also collaborates with the Manager, Rater Training, to oversee the creation of scoring materials for the Speaking test.

ESSENTIAL DUTIES

Item Development

• Manage test development staff who write, review, revise, and select Speaking test items based on item specifications, test maps, and blueprints
• Evaluate test items for appropriate task setup and design, linguistic complexity, grade-level appropriateness, content correctness, universal accessibility, appropriate scaffolding, fairness, sensitivity, and adherence to style guides
• Give clear, constructive, and timely feedback to test developers on test item text, graphics, and layouts using test item specifications
• Sign off on test item content at the end of each phase of the item development cycle
• Work in close collaboration with managers and directors across functional teams and partner organizations to coordinate efforts
• Collaborate on the production of project plans and timelines for test item development in coordination with client and other vendors and oversee their implementation
• Oversee the development and revision of item specifications and sign off on revisions
• Oversee the creation of training materials to prepare external reviewers
• Ensure test development staff provide content, administrative, and technical support for external reviewers
• Lead meetings during which stakeholders review the content, fairness, and grade-appropriateness of test items and ensure stakeholder feedback on test items is resolved
• Work with test development staff and audio recording studio to cast actors and produce recordings for Speaking test items, overseeing recording sessions and quality monitoring
• Write, review, and edit written reports, procedures, and foundational documents for internal and external stakeholders
• Update and maintain style guides and item development checklists
• Ensure test development staff follow quality control and test security procedures

Field Testing and Research

• Coordinate, conduct, and sign off on quality checks of field test QTI packages and ancillary materials
• Review and update field test functional rules
• Oversee the qualitative analysis of Speaking field test responses and report findings to internal teams and external stakeholders
• Conduct research, e.g., pilot testing and cognitive lab interviews, in support of test development

Rater Training and Scoring

• Collaborate on research and development work for scoring materials related to the Speaking test, including rubrics, rater training presentations and courses, and other scoring guidance
• Support the selection of anchor and training responses
• Review score justifications and score appeals, as well as qualifying, recalibration, and validity sets
• Synthesize…