Developing and Validating a Critical Writing Assessment Rubric for Iranian EFL Learners in Computer-Mediated Communication
Sara Samadi
1
(
)
Mohammad Hashamdar
2
(
Department of English Language Teaching and Translation, Karaj Branch, Islamic Azad University, Karaj, Iran.
)
Gholam Hassan Famil Khalili
3
(
Department of English Language Teaching and Translation, Karaj Branch, Islamic Azad University, Karaj, Iran.
)
الکلمات المفتاحية: Computer-mediated communication, critical writing, intellectual standards, rubric,
ملخص المقالة :
Given the increasing prevalence of online learning and the absence of a validated instrument for assessing the critical writing skills of Iranian EFL learners in Computer-Mediated Communication (CMC) contexts, this study aimed to develop and validate an analytic rubric to enhance critical writing in CMC environments. The participants were 236 EFL students and 10 EFL/ESL teachers, representing a diverse range of demographics, experiences, and qualifications. The rubric development process was guided by a comprehensive literature review, the administration of a critical thinking questionnaire, expert feedback, thematic analysis of semi-structured interviews, integration of Paul and Elder’s (2019) Intellectual Standards, and the development and application of a Likert-scale critical writing questionnaire. Subsequent pilot testing facilitated refinements based on feedback. Accordingly, the finalized analytic rubric featured four principal components: (1) Clarity, Accuracy, and Precision (CAP); (2) Relevance and Logic (RL); (3) Depth and Significance (DS); and (4) Breadth and Fairness (BF). A rigorous validation process, including expert evaluation, factor analysis, and structural equation modeling (SEM), confirmed the rubric's reliability and validity. The findings demonstrated satisfactory internal consistency, as well as convergent and discriminant validity. This study offers valuable pedagogical implications for English as a Foreign Language (EFL) teachers, learners, researchers, and curriculum developers.
Alghizzi, T. M., & Alshahrani, T. M. (2024). Effects of grading rubrics on EFL learners’ writing in an EMI setting. Heliyon, 10(18).1-17. https://doi.org/10.1016/j.heliyon.2024.e36394
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
Brookhart, S. (2013). How to create and use rubrics for formative assessment and grading.
ASCD. https://doi.org/10.4135/9781452218649.n15
Cohen, L., Manion, L., & Morrison, K. (2002). Research methods in education. Routledge. https://doi.org/10.4324/9780203224342
Dadakoğlu, S. C., & Özdemir, A. (2021). Development of analytical rubric for evaluating the effect of mobile design applications on artistic creativity: Validity-reliability study. İnönü Üniversitesi Eğitim Bilimleri Enstitüsü Dergisi, 8(16), 64-84. https://doi.org/10.29129/inujgse.897929
Dappen, L., Isernhagen, J., & Anderson, S. (2008). A statewide writing assessment model: Student proficiency and future implications. Assessing Writing, 13(1), 45-60. https://doi.org/10.1016/j.asw.2008.04.001.
Elbow, P. (2022). The democratization of writing and the role of cheating. Composition Studies, 50(1), 67-180.
Farzana, T. (2023). The standard measurement in online learning: a rubric as a focus on teaching-learning practices to move up quality education. EIKI Journal of Effective Teaching Methods, 1(3). https://doi.org/10.59652/jetm.v1i3.37
Ferris, D. (2009). Response to Student Writing: Implications for Second Language Students. New York, NY: Routledge. https://doi.org/sid.ir/paper/614509/en
Field, A. (2013). Discovering statistics using IBM SPSS statistics. Sage.
Gupta, B. L., & Gupta, P. B. (2021). Rubrics as versatile educational tool for outcome-based education. Journal of Engineering Technology Education, 15(2), 13-24
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2013). Multivariate data analysis: Pearson new international edition PDF eBook. Pearson Higher Ed.
Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43, 115-135. https://doi.org/10.1007/s11747-014-0403-8
Jackson, D. O. (2024). The longitudinal development of argumentative writing in an English for academic purposes course in Japan. System, 126, 103482. https://doi.org/10.1016/j.system.2024.103482
Kaiser, H. F. (1970). A second generation little jiffy. Psychometrika, 35(4), 401–415. https://doi.org/10.1007/BF02291817
Keller, S. D., Trüb, R., Raubach, E., Meyer, J., Jansen, T. & Fleckenstein, J. (2023). Designing and validating an assessment rubric for writing emails in English as a foreign language. RISTAL, 6, 16–48. https://doi.org/10.2478/ristal-2023-0002
Kobylarek, A., Błaszczyński, K., Ślósarz, L., & Madej, M. (2022). Critical Thinking Questionnaire (CThQ)–construction and application of critical thinking test tool. Andragogy Adult Education and Social Marketing, 2(2), 1-1. https://doi.org/10.15503/andr2022.1
Kohn, A. (2006). Speaking my mind: The trouble with rubrics. English Journal, 95(4), 12-15. https://doi.org/10.58680/ej20064950
Kuehner, A. V., & Hurley, J. (2019). How integrating reading and writing supports student success. Journal of Developmental Education, 20-26.
Kwan, J. L. Y., & Chan, W. (2011). Comparing standardized coefficients in structural equation modeling: A model reparameterization approach. Behavior Research Methods, 43(3), 730–745. https://doi.org/10.3758/s13428-011-0088-6
Le, X. M., Phuong, H. Y., Phan, Q. T., & Le, T. T. (2023). Impact of using analytic rubrics for peer assessment on EFL students’ writing performance: an experimental study. Multicultural Education, 9(3), 41-53. https://doi.org/10.5281/zenodo.7750831
Maatuk, A. M., Elberkawi, E. K., Aljawarneh, S., Rashaideh, H., & Alharbi, H. (2022). The COVID-19 pandemic and E-learning: challenges and opportunities from the perspective of students and instructors. Journal of Computing in Higher Education, 34(1), 21-38. https://doi.org/10.1007/s12528-021-09274-2
Panadero, E., & Jonsson, A. (2020). A critical review of the arguments against the use of rubrics. Educational Research Review, 30, 100329. https://doi.org/10.1016/j.edurev.2020.100329
Panadero, E., Jonsson, A., Pinedo, L., & Fernández-Castilla, B. (2023). Effects of rubrics on academic performance, self-regulated learning, and self-efficacy: A meta-analytic review. Educational Psychology Review, 35(4), 113.https://doi.org/10.1007/s10648-023-09823-4
Panadero, E., & Romero, M. (2014). To rubric or not to rubric? The effects of self-assessment on self-regulation, performance, and self-efficacy. Assessment in Education: Principles, Policy & Practice, 21(2), 133-148. https://doi.org/10.1080/0969594X.2013.877872.
Paul, R., & Elder, L. (2006). Critical thinking competency standards. Dillon Beach: Foundation for critical thinking.
Paul, R., & Elder, L. (2014). Critical thinking: Intellectual standards essential to reasoning well within every domain of human thought, Part 4. Journal of Developmental Education, 37(3), 34.
Paul, R., & Elder, L. (2019). The thinker's guide to intellectual standards: The words that name them and the criteria that define them. Rowman & Littlefield.
Piaget, J. (2015). The grasp of consciousness (psychology revivals): action and concept in the young child. Psychology Press. https://doi.org/10.2307/3120744
Reynolds-Keefer, L. (2019). Rubric-referenced assessment in teacher preparation: An opportunity to learn by using. Practical Assessment, Research, and Evaluation, 15(1), 8. https://doi.org/doi:10.7275/psk5-mf68.
Reynders, G., Lantz, J., Ruder, S. M., Stanford, C. L., & Cole, R. S. (2020). Rubrics to assess critical thinking and information processing in undergraduate STEM courses. International Journal of STEM Education, 7, 1-15. https://doi.org/10.1186/s40594-020-00208-5
Samadi, S., & Ghaemi, F. (2016). The impact of embedded story structures versus sequential story structures on critical thinking of Iranian intermediate EFL learners. International Journal of Applied Linguistics and English Literature, 5(5), 171-178. https://doi.org/10.7575/aiac.ijalel.v.5n.5p.171
Saxton, E., Belanger, S., & Becker, W. (2012). The Critical thinking analytic rubric (CTAR): investigating intra-rater and inter-rater reliability of a scoring mechanism for critical thinking performance assessments. Assessing Writing, 17(4), 251-270. http://dx.doi.org/10.1016/j.asw.2012.07.002
Sosik, J. J., Kahal, S. S., & Piovoso, M. J. (2009). Silver bullet or voodoo statistics? A primer for using the partial least squares data analytic technique in group and organization research. Group and Organization Management, 34(1), 5–36. https://doi.org/10.1177/1059601108329198.
Sword, H. (2019). Snowflakes, splinters, and cobblestones: Metaphors for writing. Innovations in Narrative and Metaphor: Methodologies and Practices, 39-55. https://doi.org/10.1007/978-981-13-6114-2_4
Tashtoush, M. A., Shirawia, N., & Rasheed, N. M. (2024). Scoring rubrics method in performance assessment and its effect of mathematical achievement. Athens Journal of Education, 11(1), 1-22. https://doi.org/10.30958/aje.X-Y-Z
Taylor, B., Kisby, F., & Reedy, A. (2024). Rubrics in higher education: an exploration of undergraduate students’ understanding and perspectives. Assessment & Evaluation in Higher Education, 49(6), 799-809. https://doi.org/10.1080/02602938.2023.2299330
Torrance, H. (2012). Formative assessment at the crossroads: Conformative, deformative and transformative assessment. Oxford Review of Education, 38(3), 323-342. https://doi.org/10.1080/03054985.2012.689693
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes (Vol. 86). Harvard University Press.
Yaffe, P. (2022). Why is expository writing so undervalued---and what to do about it? Ubiquity, 2022(July), 1-9. https://doi.org/10.1145/3542739
Yamanishi, H., Ono, M., & Hijikata, Y. (2019). Developing a scoring rubric for L2 summary writing: A hybrid approach combining analytic and holistic assessment. Language Testing in Asia, 9, 1-22. https://doi.org/10.1186/s40468-019-0087-6
Yancey, K. B. (2021). Follow the sources: Notes toward WEC’s contribution to disciplinary writing. Writing-enriched curricula: Models of faculty-driven and departmental transformation. The WAC Clearinghouse, 73-93. https://doi.org/10.37514/PER-B.2021.1299.2.03
Yu, Z. (2021). The effects of gender, educational level, and personality on online learning outcomes during the COVID-19 pandemic. International Journal of Educational Technology in Higher Education, 18(1), 14. https://doi.org/10.1186/s41239-021-00252.
