You are here:

The common core conundrum: To what extent should we worry that changes to assessments will affect test-based measures of teacher performance?
ARTICLE

, , , American Institutes for Research ; , University of Missouri ; , University of Virginia ; , American Institutes for Research

Economics of Education Review Volume 62, Number 1, ISSN 0272-7757 Publisher: Elsevier Ltd

Abstract

Policies that require the use of information about student achievement to evaluate teacher performance are becoming increasingly common across the United States, but there is some question as to how or whether to use student test-based teacher evaluations when student assessments change. We bring empirical evidence to bear on this issue. Specifically, we examine how estimates of teacher value-added are influenced by assessment changes across 12 test transitions in two subjects and five states. In all of the math transitions we study, value-added measures from test change years and stable regime years are broadly similar in terms of their statistical properties and informational content. This is also true for some of the reading transitions; we do find, however, some cases in which an assessment change in reading meaningfully alters value-added measures. Our study directly informs contemporary policy debates about how to evaluate teachers when new assessments are introduced and provides a general analytic framework for examining employee evaluation policies in the face of changing evaluation metrics.

Citation

Backes, B., Cowan, J., Goldhaber, D., Koedel, C., Miller, L.C. & Xu, Z. (2018). The common core conundrum: To what extent should we worry that changes to assessments will affect test-based measures of teacher performance?. Economics of Education Review, 62(1), 48-65. Elsevier Ltd. Retrieved February 27, 2020 from .

This record was imported from Economics of Education Review on March 1, 2019. Economics of Education Review is a publication of Elsevier.

Full text is availabe on Science Direct: http://dx.doi.org/10.1016/j.econedurev.2017.10.004

References

View References & Citations Map
  1. Aaronson, D., Barrow, L., & Sander, W. (2007). Teachers and student achievement in the Chicago public high schools. Journal of Labor Economics, 25(1), pp. 95-135.
  2. AERA (American Educational Research Association) (2015). AERA statement on use of value-added models (VAM) for the evaluation of educators and educator preparation programs. Educational Researcher, 44(8), pp. 448-452.
  3. ASA (American Statistical Association) (2014). ASA statement on using value-added models for educational assessment. Retrieved from http://www.amstat.org/asa/files/pdfs/POL-ASAVAM-Statement.pdf.
  4. Bacher-Hicks, A., Kane, T. J., & Staiger, D. O. (2014). Validating teacher effect estimates using changes in teacher assignments in Los Angeles. NBER working paper No. 20657.
  5. Bazemore, M., Dyk, P.V., Kramer, L., Yelton, A., & Brown, R. (2006). North Carolina mathematics tests: Technical report. The office of curriculum and school reform services.
  6. Brown, E. (2016). Technical glitches plague computer-based standardized tests nationwide. The Washington Post Available online: https://www.washingtonpost.com/local/education/technical-glitches-plague-computer-based-standardized-tests-nationwide/2016/04/13/21178c7e-019c-11e6-9203-7b8670959b88_story.html?utm_term=.edce982aa6a8.
  7. Brown, E. (2014). D.C. public schools takes a hiatus from test-based teacher evaluations as city moves to common core exams. The Washington Post Available online: https://www.washingtonpost.com/local/education/dc-public-schools-takes-hiatus-from-test-based-teacher-evaluations-as-city-moves-to-common-core-exams/2014/06/19/184b8b44-f7c2-11e3-8aa9-dad2ec039789_story.html.
  8. Chetty, R., Friedman, J.N., & Rockoff, J.E. (2014). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), pp. 2593-2632.
  9. Chetty, R., Friedman, J.N., & Rockoff, J.E. (2014). Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood. American Economic Review, 104(9), pp. 2633-2679.
  10. Chetty, R., Friedman, J.N., & Rockoff, J.E. (2017). Measuring the impacts of teachers: Reply. American Economic Review, 107(6), pp. 1685-1717.
  11. Chin, M. (2016). A "jarring" experience? Exploring how changes to standardized tests impact teacher experience effects.
  12. Chute, E., & Niederberger, M. (2015). Pennsylvania gets waiver on using PSSA scores to assess schools. Retrieved from http://www.post-gazette.com/news/education/2015/09/08/Pennsylvania-granted-one-year-waiver-on-using-PSSA-test-scores-to-measure-school-teacher-performance/stories/201509080124.
  13. Condie, S., Lefgren, L., & Sims, D. (2014). Teacher heterogeneity, value-added and education policy. Economics of Education Review, 40(1), pp. 76-92.
  14. Corcoran, S. P., Jennings, J. L., & Beveridge, A. A. (2011). Teacher effectiveness on high-and low-stakes tests.
  15. Corcoran, S., & Goldhaber, D. (2013). Value added and its uses: Where you stand depends on where you sit. Education, 8(3), pp. 418-434.
  16. D'Agostino, J.V., Welsh, M.E., & Corson, N.M. (2007). Instructional sensitivity of a state's standards-based assessment. Educational Assessment, 12(1), pp. 1-22.
  17. Darling-Hammond, L., Amrein-Beardsley, A., Haertel, E., & Rothstein, J. (2012). Evaluating teacher evaluation. The Phi Delta Kappan, 93(6), pp. 8-15.
  18. Dee, T.S., & Wyckoff, J. (2015). Incentives, selection, and teacher performance: Evidence from IMPACT. Journal of Policy Analysis and Management, 34(2), pp. 267-297.
  19. Dimick, J.B., Staiger, D.O., Basur, O., & Birkmeyer, J.D. (2009). Composite measures for predicting surgical mortality in the hospital. Health Affairs, 28(4), pp. 1189-1198.
  20. Ehlert, M., Koedel, C., Parsons, E., & Podgursky, M.J. (2014). The sensitivity of value-added estimates to specification adjustments: Evidence from school-and teacher-level models in Missouri. Statistics and Public Policy, 1(1), pp. 19-27.
  21. Frölich, M. (2004). Finite-sample properties of propensity-score matching and weighting estimators. The Review of Economics and Statistics, 86(1), pp. 77-90.
  22. Glazerman, S., Protik, A., Teh, B., Bruch, J., & Max, J. (2013). Transfer incentives for high- performing teachers: Final results from a multisite experiment (NCEE 2014-4003). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
  23. Glazerman, S., & Protik, A. (2015). Validating value-added measures of teacher performance. Unpublished manuscript.
  24. Goldhaber, D., & Chaplin, D.D. (2015). Assessing the “Rothstein Falsification Test”: Does it really show teacher value-added models are biased?. Journal of Research on Educational Effectiveness, 8(1), pp. 8-34.
  25. Goldhaber, D., Cowan, J., & Walch, J. (2013). Is a good elementary teacher always good? Assessing teacher performance estimates across subjects. Economics of Education Review, 36(1), pp. 216-228.
  26. Goldhaber, D., & Hansen, M. (2013). Is it just a bad class? Assessing the long-term stability of estimated teacher performance. Economica, 80(319), pp. 589-612.
  27. Goldhaber, D., Walch, J., & Gabele, B. (2014). Does the model matter? Exploring the relationship between different student achievement-based teacher assessments. Statistics and Public Policy, 1(1), pp. 28-39.
  28. Guarino, C.M., Maxfield, M., Reckase, M.D., Thompson, P.N., & Wooldridge, J.M. (2015). An evaluation of empirical Bayes's estimation of value-added teacher performance measures. Journal of Educational and Behavioral Statistics, 40(2), pp. 190-222.
  29. Guarino, C.M., Reckase, M.D., & Wooldridge, J.M. (2015). Can value-added measures of teacher performance be trusted. Education Finance and Policy, 10(1), pp. 117-156.
  30. Innes, R. (2009). Federal tests show Kentucky's test scoring inflated again in 2009. Available online: http://www.bipps.org/federal-tests-show-kentucky%E2%80%99s-test-scoring-inflated-again-in-2009/.
  31. Kane, T.J., & Staiger, D.O. (2008). Estimating teacher impacts on student achievement: An experimental evaluation (No. w14607).
  32. Kane, T.J., & Staiger, D.O. (2011). Initial findings from the measures of effective teaching project.
  33. Kane, T.J., McCaffrey, D.F., Miller, T., & Staiger, D.O. (2013). Have we identified effective teachers? Validating measures of effective teaching using random assignment. Seattle, WA: Bill and Melinda Gates Foundation.
  34. Kane, T.J., Taylor, E.S., Tyler, J.H., & Wooten, A.L. (2011). Identifying effective classroom practices using student achievement data. Journal of Human Resources, 46(3), pp. 587-613.
  35. Koedel, C., & Betts, J.R. (2010). Value-added to what? How a ceiling in the testing instrument influences value-added estimation. Education Finance and Policy, 5(1), pp. 54-81.
  36. Koedel, C., Leatherman, R., & Parsons, E. (2012). Test measurement error and inference from value-added models. The B.E. Journal of Economic Analysis & Policy, 12(1).
  37. Koedel, C., Mihaly, K., & Rockoff, J. (2015). Value-added modeling: A review. Economics of Education Review, 47, pp. 180-195.
  38. Koretz, D., Waldman, M., Yu, C., Langi, M., & Orzech, A. (2014). Using the introduction of a new test to investigate the distribution of score inflation.
  39. Lacireno-Paquet, N., Morgan, C., & Mello, D. (2014). How states use student learning objectives in teacher evaluation systems: A review of state websites. Washington, DC: Institute of Education Sciences, United States Department of Education.
  40. Lefgren, L., & Sims, D. (2012). Using subject specific test scores efficiently to predict teacher value-added. Educational Evaluation and Policy Analysis, 34(1), pp. 109-121.
  41. Lockwood, J.R., McCaffrey, D.F., Hamilton, L.S., Stecher, B., Le, V., & Martinez, J.F. (2007). The sensitivity of value-added teacher effect estimates to different mathematics achievement measures. Journal of Educational Measurement, 44(1), pp. 47-68.
  42. Lockwood, J.R., & McCaffrey, D. (2014). Correcting for test score measurement error in ANCOVA models for estimating treatment effects. Journal of Educational and Behavioral Statistics, 39(1), pp. 22-52.
  43. Massachusetts Department of Elementary and Secondary Education (2004). Supplement to the Massachusetts english language arts curriculum framework.
  44. Massachusetts Department of Elementary and Secondary Education (2004). Supplement to the Massachusetts mathematics curriculum framework.
  45. McCaffrey, D.F. (2013). Will teacher value-added scores change when accountability tests change? Knowledge brief. Stanford, CA: Carnegie Foundation for the Advancement of Teaching.
  46. McCaffrey, D.F., Sass, T.R., Lockwood, J.R., & Mihaly, K. (2009). The intertemporal variability of teacher effect estimates. Education Finance and Policy, 4(4), pp. 572-606.
  47. NCTQ (2016). Teacher evaluation policy in Tennessee. Retrieved from http://www.nctq.org/dmsView/Evaluation_Timeline_Brief_Tennessee.
  48. Office of the Superintendent of Public Instruction (2004). Mathematics K-10 grade level expectations: A new level of specificity (No. 04-0006). Olympia, WA: Office of the Superintendent of Public Instruction.
  49. Office of the Superintendent of Public Instruction (2004). Reading K-10 grade level expectations: A new level of specificity (No. 04-0001). Olympia, WA: Office of the Superintendent of Public Instruction.
  50. Office of the Superintendent of Public Instruction (2008). Washington state K-12 mathematics learning standards. Olympia, WA: Office of the Superintendent of Public Instruction.
  51. Papay, J.P. (2011). Different tests, different answers: The stability of teacher value-added estimates across outcome measures. American Educational Research Journal, 48(1), pp. 163-193.
  52. Polikoff, M.S., & Porter, A.C. (2014). Instructional alignment as a measure of teacher quality. Educational Evaluation and Policy Analysis, 36(4), pp. 399-416.
  53. Rothstein, J. (2017). Measuring the impacts of teachers: Comment. American Economic Review, 107(6), pp. 1656-1684.
  54. Rothstein, J. (2015). Teacher quality policy when supply matters. American Economic Review, 105(1), pp. 100-130.
  55. Sass, T.R., Semykina, A., & Harris, D.N. (2014). Value-added models and the measurement of teacher productivity. Economics of Education Review, 38(1), pp. 9-23.
  56. Schmidt, W.H., & Houang, R.T. (2012). Curricular coherence and the common core state standards for mathematics. Educational Researcher, 41(8), pp. 294-308.
  57. Schochet, P.Z., & Chiang, H.S. (2013). What are error rates for classifying teacher and school performance using value-added models. Journal of Educational and Behavioral Statistics, 38(2), pp. 142-171.
  58. Simpson, K., & Torres, Z. (2014). Testing portion of Colorado teacher evaluations lagging behind. Retrieved from http://www.denverpost.com/2014/04/19/testing-portion-of-colorado-teacher-evaluations-lagging-behind/.
  59. Stacy, B., Guarino, C.M., Reckase, M.D., & Wooldridge, J. (2013). Does the precision and stability of value-added estimates of teacher performance depend on the types of students they serve? (No. 7676). Bonn, Germany: Institute for the Study of Labor.
  60. Staiger, D.O., & Rockoff, J.E. (2010). Searching for effective teachers with imperfect information. Journal of Economic Perspectives, 24(3), pp. 97-118.
  61. Steinberg, M., & Donaldson, M. (2016). The new educational accountability: Understanding the landscape of teacher evaluation in the post-NCLB era. Education Finance and Policy, 11(3), pp. 340-359.
  62. Steinberg, M., & Garrett, R. (2016). Classroom composition and measured teacher performance: What do teacher observation scores really measure. Educational Evaluation and Policy Analysis, 38(2), pp. 293-317.
  63. Whitehurst, G.J., Chingos, M.M., & Lindquist, K.M. (2014). Evaluating teachers with classroom observations: lessons learned from four districts. Washington DC: Brown Center on Education Policy. Brookings Institution.
  64. Winters, M.A., & Cowen, J.M. (2013). Would a value-added system of retention improve the distribution of teacher quality? A simulation of alternative policies. Journal of Policy Analysis and Management, 32(3), pp. 634-654.
  65. Xu, Z., Ozek, U., & Corritore, M. (2012). Portability of teacher effectiveness across school settings. CALDER working paper WP77.

These references have been extracted automatically and may have some errors. Signed in users can suggest corrections to these mistakes.

Suggest Corrections to References