You are here:

Automated Essay Scoring: Psychometric Guidelines and Practices
ARTICLE

,

Assessing Writing Volume 18, Number 1, ISSN 1075-2935

Abstract

In this paper, we provide an overview of psychometric procedures and guidelines Educational Testing Service (ETS) uses to evaluate automated essay scoring for operational use. We briefly describe the e-rater system, the procedures and criteria used to evaluate e-rater, implications for a range of potential uses of e-rater, and directions for future research. The description of e-rater includes a summary of characteristics of writing covered by e-rater, variations in modeling techniques available, and the regression-based model building procedure. The evaluation procedures cover multiple criteria, including association with human scores, distributional differences, subgroup differences and association with external variables of interest. Expected levels of performance for each evaluation are provided. We conclude that the "a priori" establishment of performance expectations and the evaluation of performance of e-rater against these expectations help to ensure that automated scoring provides a positive contribution to the large-scale assessment of writing. We call for continuing transparency in the design of automated scoring systems and clear and consistent expectations of performance of automated scoring before using such systems operationally. (Contains 1 figure and 1 table.)

Citation

Ramineni, C. & Williamson, D.M. (2013). Automated Essay Scoring: Psychometric Guidelines and Practices. Assessing Writing, 18(1), 25-39. Retrieved July 6, 2022 from .

This record was imported from ERIC on April 18, 2013. [Original Record]

ERIC is sponsored by the Institute of Education Sciences (IES) of the U.S. Department of Education.

Copyright for this record is held by the content creator. For more details see ERIC's copyright policy.

Keywords

Cited By

View References & Citations Map
  • Impact of AWE Rubrics and Automated Assessment on EFL Writing Instruction

    Jinlan‎ Tang, School of Online and Continuing Education, Beijing Foreign Studies University, Beijing, China; Yi'an Wu, School of English and International Studies, Beijing Foreign Studies University, Beijing, China

    International Journal of Computer-Assisted Language Learning and Teaching Vol. 7, No. 2 (April 2017) pp. 58–74

These links are based on references which have been extracted automatically and may have some errors. If you see a mistake, please contact info@learntechlib.org.