Abstract :

Information services for student grant programmes require answers that are fast, consistent, and grounded in official documents, while the rules themselves change with every programme cycle. This study evaluates AIDA (Academic Digital Information Assistant), a university academic service chatbot, within the domain of an academic learning grant programme. Following a Design Science Research framework, the evaluation proceeded in two cycles. The first measured answer accuracy through the assessment of three programme administrators across 32 test questions on the dimensions of relevance, correctness, and completeness. The second measured usability through the Chatbot Usability Questionnaire administered to 76 student users. The first cycle yielded 90.6% relevance, 81.3% correctness, 90.6% completeness, and an overall pass rate of 81.3%, with inter-rater agreement between 82% and 85%. All six failing items shared a single root cause, namely timeline information that had not been updated to the current programme period; three of them were nonetheless judged relevant and complete, rendering the error difficult for users to detect. The second cycle produced a usability score of 67.72 (SD 11.93) with a reliability of 0.865, highest on ease of use and lowest on conversational naturalness and error handling. The study concludes that the principal constraint on administrative service chatbots lies not in language understanding but in the governance of content freshness, and proposes a knowledge base update mechanism tied to the programme calendar.

---

Keywords :

academic chatbot, BM25, chatbot usability questionnaire, knowledge base governance, retrieval-augmented generation

---

References :

1. Republic of Indonesia, Regulation of the Minister of Higher Education, Science, and Technology Number 39 of 2025 on Higher Education Quality Assurance. Jakarta, 2025.
2. Republic of Indonesia, Regulation of the Minister of Education, Culture, Research, and Technology Number 53 of 2023 on Higher Education Quality Assurance. Jakarta, 2023.
3. Republic of Indonesia, Decree of the Minister of Education and Culture Number 74/P/2021 on the Recognition of Semester Credit Units for Kampus Merdeka Programme Learning. Jakarta, 2021.
4. A. Kuhail, N. Alturki, S. Alramlawi, and K. Alhejori, “Interacting with educational chatbots: A systematic review,” Educ. Inf. Technol. (Dordr)., vol. 28, no. 1, pp. 973–1018, Jan. 2023, doi: 10.1007/s10639-022-11177-3.
5. Labadze, M. Grigolia, and L. Machaidze, “Role of AI chatbots in education: systematic literature review,” International Journal of Educational Technology in Higher Education, vol. 20, no. 1, p. 56, Oct. 2023, doi: 10.1186/s41239-023-00426-1.
6. Ramandanis and S. Xinogalos, “Investigating the Support Provided by Chatbots to Educational Institutions and Their Students: A Systematic Literature Review,” Multimodal Technologies and Interaction, vol. 7, no. 11, p. 103, Nov. 2023, doi: 10.3390/mti7110103.
7. Peyton, S. Unnikrishnan, and B. Mulligan, “A review of university chatbots for student support: FAQs and beyond,” Discover Education, vol. 4, no. 1, p. 21, Jan. 2025, doi: 10.1007/s44217-025-00397-7.
8. -J. Hwang and C.-Y. Chang, “A review of opportunities and challenges of chatbots in education,” Interactive Learning Environments, vol. 31, no. 7, pp. 4099–4112, Oct. 2023, doi: 10.1080/10494820.2021.1952615.
9. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” Apr. 2021, [Online]. Available: http://arxiv.org/abs/2005.11401
10. Gao et al., “Retrieval-Augmented Generation for Large Language Models: A Survey,” Mar. 2024.
11. Huang et al., “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions,” ACM Trans. Inf. Syst., vol. 43, no. 2, pp. 1–55, Mar. 2025, doi: 10.1145/3703155.
12. Wulandari, “Analisis kinerja Retrieval Augmented Generation (RAG) klasik dalam chatbot akademik berbasis multimodal [Performance analysis of classical Retrieval Augmented Generation (RAG) in multimodal-based academic chatbots],” Journal of Informatics and Computer Science (JINACS), vol. 7, no. 3, pp. 628–632, 2026.
13. Wollny, J. Schneider, D. Di Mitri, J. Weidlich, M. Rittberger, and H. Drachsler, “Are We There Yet? – A Systematic Literature Review on Chatbots in Education,” Front. Artif. Intell., vol. 4, Jul. 2021, doi: 10.3389/frai.2021.654924.
14. W. Okonkwo and A. Ade-Ibijola, “Chatbots applications in education: A systematic review,” Computers and Education: Artificial Intelligence, vol. 2, p. 100033, 2021, doi: 10.1016/j.caeai.2021.100033.
15. S. M. Pereira, F. Falcão, L. Costa, B. S. Lunn, J. M. Pêgo, and P. Costa, “Here’s to the future: Conversational agents in higher education- a scoping review,” Int. J. Educ. Res., vol. 122, p. 102233, 2023, doi: 10.1016/j.ijer.2023.102233.
16. Nicolescu and M. T. Tudorache, “Human-Computer Interaction in Customer Service: The Experience with AI Chatbots—A Systematic Literature Review,” Electronics (Basel)., vol. 11, no. 10, p. 1579, May 2022, doi: 10.3390/electronics11101579.
17. Robertson and H. Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond,” Foundations and Trends® in Information Retrieval, vol. 4, no. 1–2, pp. 1–174, Sep. 2009, doi: 10.1561/1500000019.
18. Karpukhin et al., “Dense Passage Retrieval for Open-Domain Question Answering,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 6769–6781. doi: 10.18653/v1/2020.emnlp-main.550.
19. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych, “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.” [Online]. Available: http://www.ukp.tu-darmstadt.de
20. Huang et al., “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions,” ACM Trans. Inf. Syst., vol. 43, no. 2, pp. 1–55, Mar. 2025, doi: 10.1145/3703155.
21. Ji et al., “Survey of Hallucination in Natural Language Generation,” ACM Comput. Surv., vol. 55, no. 12, pp. 1–38, Dec. 2023, doi: 10.1145/3571730.
22. Chen, H. Lin, X. Han, and L. Sun, “Benchmarking Large Language Models in Retrieval-Augmented Generation,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, pp. 17754–17762, Mar. 2024, doi: 10.1609/aaai.v38i16.29728.
23. Holmes, A. Moorhead, R. Bond, H. Zheng, V. Coates, and M. Mctear, “Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces?,” in Proceedings of the 31st European Conference on Cognitive Ergonomics, New York, NY, USA: ACM, Sep. 2019, pp. 207–214. doi: 10.1145/3335082.3335094.
24. Rapp, L. Curti, and A. Boldi, “The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots,” Int. J. Hum. Comput. Stud., vol. 151, p. 102630, Jul. 2021, doi: 10.1016/j.ijhcs.2021.102630.
25. Følstad and P. B. Brandtzaeg, “Users’ experiences with chatbots: findings from a questionnaire study,” Qual. User Exp., vol. 5, no. 1, p. 3, Dec. 2020, doi: 10.1007/s41233-020-00033-2.
26. Peffers, T. Tuunanen, M. A. Rothenberger, and S. Chatterjee, “A Design Science Research Methodology for Information Systems Research,” Journal of Management Information Systems, vol. 24, no. 3, pp. 45–77, Dec. 2007, doi: 10.2753/MIS0742-1222240302.
27. UNESCO, “Guidance for generative AI in education and research,” Paris, 2023.