Trust in Automated Writing Evaluation (AWE): Multiple case study on feedback engagement and literacy

Authors

  • Nur Arifah Drajati Universitas Sebelas Maret
  • Anis Handayani Universitas Negeri Yogyakarta

DOI:

https://doi.org/10.26418/jeltim.v6i2.83172

Keywords:

academic writing, automated writing evaluation (AWE), feedback engagement, feedback literacy, feedback trust

Abstract

Artificial Intelligence (AI) for educational purposes has recently been a research interest. Specifically for academic writing, AI-based feedback, often called automated writing evaluation (AWE), is an essential aspect to support students"™ writing. However, despite its popularity, the trust in AWE, which potentially contributes to the students"™ writing improvement, has been explored on a limited basis. Hence, this multiple case study attempts to examine the trust in AWE and how this trust influences feedback engagement and literacy. Of 42 students joining a writing class, a total of 4 undergraduate students were involved to explore this issue. Considering the Indonesian government's policy to conduct online learning amidst the COVID-19 pandemic era, classroom observation, stimulated recalls, and semi-structured interviews were conducted virtually via Zoom meetings to gather the data. Results show that the students' trust in AWE varied, indicated by their varied responses to AWE. Furthermore, their trust in AWE only affects the behavioral aspect, excluding the cognitive and affective aspects. Meanwhile, in feedback literacy, their trust only affected the aspect of appreciating the feedback. These findings imply that educators should pay interest to students' trust in AWE since it, in some ways, influences improvement of their feedback engagement and literacy.

Author Biographies

Nur Arifah Drajati, Universitas Sebelas Maret

Nur Arifah Drajati is a lecturer of the English Language Education Department of Universitas Sebelas Maret (UNS).  She contributed to several reputable journals. Her research interests lie in technology in language learning, TPACK, Multimodal, and IDLE.

Anis Handayani, Universitas Negeri Yogyakarta

Anis Handayani is a lecturer of the English Language Education Department of Universitas Negeri Yogyakarta.  She contributed to several reputable journals. Her research interests lie in academic writing, educational technology, and teacher professional development.

References

Bai, L., & Hu, G. (2016). In the face of fallible AWE feedback: how do students respond? Educational Psychology, 37(1), 67–81. https://doi.org/10.1080/01443410.2016.1223275

Carless, D., & Boud, D. (2018). The development of student feedback literacy: enabling uptake of feedback. Assessment and Evaluation in Higher Education, 43(8), 1315–1325. https://doi.org/10.1080/02602938.2018.1463354

Carless, D., & Winstone, N. (2020). Teacher feedback literacy and its interplay with student feedback literacy. Teaching in Higher Education, 28(1), 150–163. https://doi.org/10.1080/13562517.2020.1782372

Chapelle, C. A., Cotos, E., & Lee, J. (2015). Validity arguments for diagnostic assessment using automated writing evaluation. Language Testing, 32(3), 385-405. https://doi.org/10.1177/0265532214565386

Cope, B., Kalantzis, M., Searsmith, D., Cope, B., Kalantzis, M., & Searsmith, D. (2020). Artificial intelligence for education : Knowledge and its assessment in AI-enabled learning ecologies. Educational Philosophy and Theory, 53(12), 1229–1245. https://doi.org/10.1080/00131857.2020.1728732

Gravett, K., & Winstone, N. E. (2019). ‘Feedback interpreters’: the role of learning development professionals in facilitating university students’ engagement with feedback. Teaching in Higher Education, 24(6), 723–738. https://doi.org/10.1080/13562517.2018.1498076

Han, Y. (2019). Written corrective feedback from an ecological perspective: The interaction between the context and individual learners. System, 80, 288–303. https://doi.org/https://doi.org/10.1016/j.system.2018.12.009

Han, Y., & Xu, Y. (2021). Student feedback literacy and engagement with feedback: a case study of Chinese undergraduate students. Teaching in Higher Education, 26(2), 181–196. https://doi.org/10.1080/13562517.2019.1648410

Hojeij, Z. (2022). Effectiveness of Using Digital Feedback on EFL Student Writing Skills. International Journal of Computer-Assisted Language Learning and Teaching, 12(1), 1-18. https://doi.org/10.4018/IJCALLT.291111

Huang, S., & Renandya, W. A. (2018). Exploring the integration of automated feedback among lower-proficiency EFL learners. Innovation in Language Learning and Teaching, 14(1), 15–26. https://doi.org/10.1080/17501229.2018.1471083

Jiang, L., & Yu, S. (2020). Appropriating automated feedback in L2 writing : experiences of Chinese EFL student writers. Computer Assisted Language Learning, 35(7), 1329–1353. https://doi.org/10.1080/09588221.2020.1799824

Koltovskaia, S. (2020). Student engagement with automated written corrective feedback (AWCF) provided by Grammarly: A multiple case study. Assessing Writing, 44(April 2020), 100450. https://doi.org/10.1016/j.asw.2020.100450

Lee, C. (2019). A study of adolescent English learners ’ cognitive engagement in writing while using an automated content feedback system. Computer Assisted Language Learning, 33(1–2), 26–57. https://doi.org/10.1080/09588221.2018.1544152

Li, Z., Link, S., Ma, H., Yang, H., & Hegelheimer, V. (2014). The role of automated writing evaluation holistic scores in the ESL classroom. System, 44, 66–78. https://doi.org/10.1016/j.system.2014.02.007

Li, Z., Zuo, T., Wei, X., & Ding, N. (2023). ICT Self-efficacy scale: the correlations with the age of first access to the internet, the age at first ownership of a personal computer (PC), and a smartphone. Medical Education Online, 28(1), 1-8. https://doi.org/10.1080/10872981.2022.2151068

Link, S., Mehrzad, M., & Rahimi, M. (2020). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer Assisted Language Learning, 35(4), 605–634. https://doi.org/10.1080/09588221.2020.1743323

Liu, S., & Yu, G. (2022). L2 learners ’ engagement with automated feedback : An eye-tracking study. Language Learning & Technology, 26(2), 78–105. https://doi.org/https://doi.org/10125/73480

Lyons, J. B., & Guznov, S. Y. (2018). Theoretical Issues in Ergonomics Science Individual differences in human – machine trust : A multi-study look at the perfect automation schema. Theoretical Issues in Ergonomics Science, 20(4), 440–458. https://doi.org/10.1080/1463922X.2018.1491071

Malecka, B., Boud, D., Tai, J., & Ajjawi, R. (2022). Navigating feedback practices across learning contexts: implications for feedback literacy. Assessment and Evaluation in Higher Education, 47(8), 1330–1344. https://doi.org/10.1080/02602938.2022.2041544

Ranalli, J. (2018). Automated written corrective feedback: how well can students make use of it? Computer Assisted Language Learning, 31(7), 653–674. https://doi.org/10.1080/09588221.2018.1428994

Ranalli, J. (2021). L2 student engagement with automated feedback on writing: Potential for learning and issues of trust. Journal of Second Language Writing, 52(June 2021), 1–16. https://doi.org/10.1016/j.jslw.2021.100816

Ranalli, J., Link, S., & Chukharev-hudilainen, E. (2016). Automated writing evaluation for formative assessment of second language writing : investigating the accuracy and usefulness of feedback as part of argument-based validation. Educational Psychology, 37(1), 8–25. https://doi.org/10.1080/01443410.2015.1136407

Saricaoglu, A. (2021). Voluntary use of automated writing evaluation by content course students. ReCALL, 33(3), 265–277. https://doi.org/10.1017/S0958344021000021

Sutton, P. (2012). Conceptualizing feedback literacy: knowing, being, and acting. Innovations in Education and Teaching International, 49(1), 31–40. https://doi.org/10.1080/14703297.2012.647781

Tian, L., & Zhou, Y. (2020). Learner engagement with automated feedback , peer feedback and teacher feedback in an online EFL writing context. System, 91(July 2020), 102247. https://doi.org/10.1016/j.system.2020.102247

Walker, R., Oliver, R., & Mackenzie, R. (2020). Interviews with secondary school students: Perceptions of feedback. Issues in Educational Research, 30(4), 1576–1595. http://www.iier.org.au/iier30/walker.pdf

Wilken, J. L. (2018). Perceptions of L1 Glossed Feedback in Automated Writing Evaluation : A Case Study. Computer Assisted Language Instruction Consortium, 35, 30–48. https://doi.org/https://doi.org/10.1558/cj.26383

Wood, J. (2021). Making peer feedback work: the contribution of technology-mediated dialogic peer feedback to feedback uptake and literacy. Assessment and Evaluation in Higher Education, 47(3), 327–346. https://doi.org/10.1080/02602938.2021.1914544

Yu, S., & Liu, C. (2021). Improving student feedback literacy in academic writing: An evidence-based framework. Assessing Writing, 48(April 2021), 100525. https://doi.org/10.1016/j.asw.2021.100525

Zhai, N., & Ma, X. (2021). Automated writing evaluation (AWE) feedback : a systematic investigation of college students ’ acceptance. Computer Assisted Language Learning, 35(9), 2817–2842. https://doi.org/10.1080/09588221.2021.1897019

Zhang, Z. (Victor). (2020). Engaging with automated writing evaluation (AWE) feedback on L2 writing: Student perceptions and revisions. Assessing Writing, 43(January 2020), 100439. https://doi.org/10.1016/j.asw.2019.100439

Zhang, Z. (Victor), & Hyland, K. (2018). Student engagement with teacher and automated feedback on L2 writing. Assessing Writing, 36, 90-102. https://doi.org/10.1016/j.asw.2018.02.004

Downloads

Published

2024-10-05

Issue

Vol. 6 No. 2 (2024): October 2024

Section

Articles

License

Copyright (c) 2024 by the authors

Authors retain copyright and grant the journal the right to publish the work, which is simultaneously made available under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence: https://creativecommons.org/licenses/by/4.0/