Automated Essay Scoring: Psychometric Guidelines and Practices
Assessing Writing Volume 18, Number 1, ISSN 1075-2935
In this paper, we provide an overview of psychometric procedures and guidelines Educational Testing Service (ETS) uses to evaluate automated essay scoring for operational use. We briefly describe the e-rater system, the procedures and criteria used to evaluate e-rater, implications for a range of potential uses of e-rater, and directions for future research. The description of e-rater includes a summary of characteristics of writing covered by e-rater, variations in modeling techniques available, and the regression-based model building procedure. The evaluation procedures cover multiple criteria, including association with human scores, distributional differences, subgroup differences and association with external variables of interest. Expected levels of performance for each evaluation are provided. We conclude that the "a priori" establishment of performance expectations and the evaluation of performance of e-rater against these expectations help to ensure that automated scoring provides a positive contribution to the large-scale assessment of writing. We call for continuing transparency in the design of automated scoring systems and clear and consistent expectations of performance of automated scoring before using such systems operationally. (Contains 1 figure and 1 table.)
Ramineni, C. & Williamson, D.M. (2013). Automated Essay Scoring: Psychometric Guidelines and Practices. Assessing Writing, 18(1), 25-39.
Cited ByView References & Citations Map
Jinlan Tang, School of Online and Continuing Education, Beijing Foreign Studies University, Beijing, China; Yi'an Wu, School of English and International Studies, Beijing Foreign Studies University, Beijing, China
International Journal of Computer-Assisted Language Learning and Teaching Vol. 7, No. 2 (April 2017) pp. 58–74
These links are based on references which have been extracted automatically and may have some errors. If you see a mistake, please contact firstname.lastname@example.org.