The Malaysian University English Test (MUET) presents persistent challenges for learners of English as a second language (ESL), particularly in the reading component, which requires them to comprehend complex academic texts under considerable time pressure. To address this challenge, MUETBot, a conversational tool supported by artificial intelligence (AI), was designed and developed to facilitate independent learning of MUET reading skills. This paper reports the evaluation phase within the Design and Development Research (DDR) framework, focusing on content validity, user acceptance, and the reliability of the evaluation instrument. Five subject matter experts assessed content validity using a structured validation form, from which item and scale level content validity indices were calculated. User acceptance was subsequently evaluated through a User Acceptance Test (UAT) administered to 30 undergraduate ESL students on two occasions separated by two weeks. The expert evaluation produced an average scale level content validity index of .92, indicating strong alignment between the MUETBot content and the constructs underlying MUET reading. The UAT scores demonstrated good internal consistency (Cronbach’s ? = .824) and strong temporal stability (r = .85, p < .001). Students also reported high levels of acceptance across all five constructs, with mean scores ranging from 4.33 to 4.47 on a five point scale. The findings indicate that MUETBot provides content aligned with the MUET reading construct and is positively accepted by its intended users. The study also presents an empirically grounded approach to validating language learning tools supported by AI before examining their effects on learning outcomes.
Bibauw, S., François, T., & Desmet, P. (2019). Discussing with a computer to practice a foreign language: Research synthesis and conceptual framework of dialogue-based CALL. Computer Assisted Language Learning, 32(8), 827–877. https://doi.org/10.1080/09588221.2018.1535508
Bower, M. (2019). Technology-mediated learning theory. British Journal of Educational Technology, 50(3), 1035–1048. https://doi.org/10.1111/bjet.12771
Chapelle, C. A. (2001). Computer applications in second language acquisition: Foundations for teaching, testing and research. Cambridge University Press.
Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). SAGE.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2012). How to design and evaluate research in education (8th ed.). McGraw-Hill.
Fryer, L. K., Ainley, M., Thompson, A., Gibson, A., & Sherlock, Z. (2017). Stimulating and sustaining interest in a language course: An experimental comparison of chatbot and human task partners. Computers in Human Behavior, 75, 461–468. https://doi.org/10.1016/j.chb.2017.05.045
Haenlein, M., & Kaplan, A. (2019). A brief history of artificial intelligence: On the past, present, and future of artificial intelligence. California Management Review, 61(4), 5–14. https://doi.org/10.1177/0008125619864925
Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning—Are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38(1), 237–257. https://doi.org/10.1111/jcal.12610
Karnine, M. (2022). Challenges in reading comprehension among ESL learners. Journal of Language Education, 8(2), 45–58.
Krashen, S. (1981). Second language acquisition and second language learning. Pergamon Press.
Lyu, B., Lai, C., & Chai, C. S. (2025). Effectiveness of chatbots in improving language learning: A meta-analysis of comparative studies. International Journal of Applied Linguistics. Advance online publication. https://doi.org/10.1111/ijal.12668
Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497. https://doi.org/10.1002/nur.20147
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. https://doi.org/10.1002/nur.20199
Richey, R. C., & Klein, J. D. (2007). Design and development research: Methods, strategies, and issues. Routledge.
?ahin K?z?l, A. (2025). A systematic review of the recent research on the usefulness of chatbots for language education. Journal of Computer Assisted Learning, 41(1), Article e70001. https://doi.org/10.1111/jcal.70001
Short, J., Williams, E., & Christie, B. (1976). The social psychology of telecommunications. John Wiley & Sons.
Sidek, M. N., & Jamaludin, A. (2008). Pembinaan modul: Bagaimana membina modul latihan dan modul akademik. Penerbit Universiti Putra Malaysia.
Siemens, G. (2005). Connectivism: A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning, 2(1), 3–10.
Taber, K. S. (2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2
Tang, T. Y. (2024). MUETBot: Designing and developing a chatbot to support the learning of MUET reading skills [Unpublished doctoral dissertation]. Universiti Pendidikan Sultan Idris.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
Yin, T. T., & Hanif, H. (2026). The Validity and Reliability of MUETBot in Supporting the Learning of MUET Reading Skills among ESL Students. International Journal of Academic Research in Progressive Education and Development, 15(3), 930–946.
Copyright: © 2026 The Author(s)
Published by Knowledge Words Publications (www.kwpublications.com)
This article is published under the Creative Commons Attribution (CC BY 4.0) license. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this license may be seen at: http://creativecommons.org/licences/by/4.0/legalcode
