Investigating Factors Affecting the Uptake of Automated Assessment Technology
Carl Dreher, Curtin University, Australia ; Torsten Reiners, University of Hamburg, Germany ; Heinz Dreher, Curtin University, Australia
JITE-Research Volume 10, Number 1, ISSN 1539-3585 Publisher: Informing Science Institute
Automated assessment is an emerging innovation in educational praxis, however its pedagogical potential is not fully utilised in Australia, particularly regarding automated essay grading. The rationale for this research is that the usage of automated assessment currently lags behind the capacity that the technology provides, thus restricting the pedagogical benefits for students, job satisfaction for staff, and quality assurance and financial benefits for universities. The exploration of the different perspectives of stakeholders regarding their needs and expectations of automated assessment shows the high-stake game of surviving and thriving. Inter alia, students value education as a means to gain employment, educators’ value education as their means of employment, and universities value education as their means of existence. The various facets of this interplay are described, including the value of knowledge and learning (pedagogical considerations); society’s system for measuring worth and exchanging value (economic considerations); sociotechnological evolution and emerging innovations in educational assessment, and the adoption and resistance thereof. We contribute to the field a national survey that investigated and reveals potential reasons for this sub-optimal utilisation by exploring the candid perceptions of Australian university students and teachers regarding automated assessment. Students and staff from Australian universities were invited to participate in an anonymous online survey. Two hundred and sixty five people completed the survey. The analysis utilised a mixed method approach, combining quantitative and qualitative research paradigms. We present a sub-set of the survey results; this sample’s use of automated assessment is mapped via quantitative data, and respondents’ perceptions of automated assessment are examined via a constructivist grounded theory analysis of short answer responses. Specific aims of this research were to survey the human and automated assessment practices in Australian universities (specifically, the educational roles of users (e.g., students, educators, management, IT-support, HR-administration), assessment types used, frequency and mode of use (individual vs. group), mode of awarding marks (individual vs. group), and mode of marking (human vs. computer)), examine what ‘automated assessment’ means to respondents, explore respondents’ impressions of automated essay grading, and survey respondents’ use of automated assessment technologies. This sample’s concept of ‘automated assessment’ is multifaceted but can be summarised by four categories of themes that describe automated assessment as involving computers or technology, electronic media, ‘marking’ or ‘assessing’, and degrees of automation. Understanding stakeholders’ perceptions of automated assessment is instrumental in promoting its use and benefits. The implications from the survey transfer the focal point of our research agenda to building a profound methodology for future automated assessment and, in particular, to delineate a context specific process of integration for the relevant stakeholders (e.g., students and staff). Most important, the marketing portfolio calls for the following, among others, to be demonstrated across various educational domains: robustness regarding marking accuracy and time (compared to human marking), ease of use for inexperienced users (students as well as teachers), and economic savings. The automated assessment has to meet pedagogical benefits first, but also has to provide commercial benefits. Especially the latter one is related to the reputation of the university, which is judged in society by successful graduations and post-graduation job performance. High quality education is the harbinger of new student enrolments and the attractor of research and business projects, including endowments and sponsorship, in total increasing the financial scope. In order to increase adoption of these technologies we have to learn more about stakeholders’ concerns, desires, and speak their language in order to have them move-in and feel at home using educational innovations such as automated essay grading. The contribution provides an inside view and understanding of where we are with respect to automated assessment and what is still part of the roadmap for a successful integration.
Dreher, C., Reiners, T. & Dreher, H. (2011). Investigating Factors Affecting the Uptake of Automated Assessment Technology. Journal of Information Technology Education: Research, 10(1), 161-181. Informing Science Institute.
ReferencesView References & Citations Map
These references have been extracted automatically and may have some errors. Signed in users can suggest corrections to these mistakes.Suggest Corrections to References