AI Tools and Language Learning Achievement: A Quasi-Experimental Study in Pakistani EFL Classrooms

Abstract

English as a Foreign Language (EFL) classrooms' incorporation of AI tools has been largely neglected in Pakistan. The effect of AI grammar tools on students' language learning achievement and language learning motivation were examined in this study. Pre-test post-test control group (quasi-experimental) design was used. A total of 180 9/10th Grade students from four Public schools participated in Peshawar, Khyber Pakhtunkhwa. The experimental group (n = 90) received 8 weeks of writing exercises with ChatGPT and Grammarly, whereas the control group (n = 90) did their writing using traditional methods. Data was obtained by using a Grammar Achievement Test and a Language Learning Motivation Questionnaire (LLMQ) and analyzed using ANCOVA, paired-samples t-test, and AMOS: Language Structural Equation Modeling. The theoretical model chosen was the Input Hypothesis and the Technology Acceptance Model (TAM). The experimental group achieved significantly higher post-test grammar scores (M = 72.4, p < .001, partial η² = .31) and there was a significant increase in motivation scores as well. The findings of SEM showed that perceived usefulness and perceived ease of use had positive prediction for learning achievement with student engagement as a mediator. There were no significant ethical issues reported but teachers reported training needs. This study is one of the first experimental studies on the use of AI tools in EFL context in Pakistan. The results were conducive to the inclusion of AI tools in the Single National Curriculum along with teacher training. Public schools and the implications of SDG 4 and digital equity are discussed.