Statistical Indexes for Monitoring Item Behavior under Computer Adaptive Testing Environment
American Educational Research Association Annual Meeting,
A computerized adaptive test (CAT) administration usually requires a large supply of items with accurately estimated psychometric properties, such as item response theory (IRT) parameter estimates, to ensure the precision of examinee ability estimation. However, an estimated IRT model of a given item in any given pool does not always correctly capture what underlies actual examinee responses. This so-called model-data deviation could seriously jeopardize the quality of a test in practice. Therefore, monitoring item behavior in a timely manner is extremely important in CAT for practitioners to take appropriate actions, such as blocking problematic items from active use or pulling the items from subsequent item pools. The purpose of this simulation study was to develop and test two statistical indexes for identifying problematic items with serious model-data deviations. Preliminary results from the simulation study suggested that the new indexes Z(2) and Z(3) could be applied to items with either uniform or nonuniform deviations. Also, results showed that the new indexes exhibited a desired feature. That is, the measured index value monotonically increased as the degree of model-data deviation increased. Further, index Z(3) was more stable across different ability distributions than index Z(2). However, the results indicated that both indexes were sensitive to the variation of examinee sample size. (Author/SLD)
Zhu, R., Yu, F. & Liu, S. (2002). Statistical Indexes for Monitoring Item Behavior under Computer Adaptive Testing Environment. Presented at American Educational Research Association Annual Meeting 2002.