Psychometric Evaluation of an Applied Mathematics Examination Instrument Based on The Rasch Model

Vol 9 No 8 (2026): Volume 09 Issue 08 August 2026

Article Date Published : 6 August 2026 | Page No.: 4282-4295 | Google Scholar | Crossref doi for this article

Abstract :

Administering the same examination to students from different programs can result in uneven measurement of competence and potentially flawed academic decisions. In many vocational settings, psychometric reviews of examination instruments are uncommon, increasing the risk of inaccurate evaluation. This study investigates the Applied Mathematics Final Semester Examination (FSE) instruments in the Mechanical Engineering Department, Bali State Polytechnic, through the Rasch model. The data came from 206 students who answered 40 multiple-choice questions. The test was delivered online via the institutional e-learning system but taken in the classroom. Winsteps 5.9 was used to assess construct validity, reliability, separation indices, item difficulty in relation to ability, and Differential Item Functioning (DIF) across programs. Demonstrates that the instrument meets all psychometric standards. Nine components that didn’t fit nonetheless showed the planned construct. The separation indices were 3.46 and 6.18, and the person and item reliability were both over 0.90. The major dimension explained more than 63% of the variation, which indicated that it was one-dimensional. Usually, the difficulty levels were right for the students, but not at the extremes. Eight things suggested that there might be bias. The result is a one-of-a-kind Rasch evaluation of a vocational mathematics FSE that combines reliability, unidimensionality, and DIF analysis.  It can be used as a model for other situations and to make vocational math tests fair, valid, and reliable.

Keywords :

Applied Mathematics, item fit, psychometric evaluation, Rasch Model, vocational education

References :

  1. Aarts, P., Van Huijstee, J. C., Ragamin, A., Reeves, J. L., Van Montfrans, C., Van Der Zee, H. H., & Prens, E. P. (2023). Validity and reliability of two digital wound measurement tools after surgery in patients with Hidradenitis Suppurativa. Dermatology, 239(1), 99–108. https://doi.org/10.1159/000525844
  2. AERA, APA, & NCME. (2014). Standards for educational and psychological testing (14 ed). American Educational Research Association. https://www.testingstandards.net/
  3. Agala, C. B., Fried, B. J., Thomas, J. C., Reynolds, H. W., Lich, K. H., Whetten, K., Zimmer, C., & Morrissey, J. P. (2020). Reliability, validity and measurement invariance of the Simplified Medication Adherence Questionnaire (SMAQ) among HIV-positive women in Ethiopia: A quasi-experimental study. BMC Public Health, 20(1), 1–16. https://doi.org/10.1186/s12889-020-08585-w
  4. Ajeigbe, T. O., & Afolabi, E. R. I. (2014). Assessing unidimensionality and differential item functioning in qualifying examination for senior secondary school students, Osun State, Nigeria. World Journal of Education, 4(4), 30–37. https://doi.org/10.5430/wje.v4n4p30
  5. Alhadabi, A., & Aldhafri, S. (2021). A rasch model analysis of the psychometric properties of the student-teacher relationship scale among middle school students. European Journal of Educational Research, 10(2), 957–973. https://doi.org/10.12973/EU-JER.10.2.957
  6. Amalina, I. K., & Vidákovich, T. (2022). An integrated STEM-based mathematical problem-solving test: Developing and reporting psychometric evidence. Journal on Mathematics Education, 13(4), 587–604. https://doi.org/10.22342/jme.v13i4.pp587-604
  7. Andrich, D., Marais, I., & Humphry, S. (2012). Using a theorem by Andersen and the dichotomous Rasch model to assess the presence of random guessing in multiple choice items. Journal of Educational and Behavioral Statistics, 37(3), 417–442. https://doi.org/10.3102/1076998611411914
  8. Avinç, E., & Doğan, F. (2024). Digital literacy scale: Validity and reliability study with the rasch model. Education and Information Technologies, 29(nd), 22895–22941. https://doi.org/10.1007/s10639-024-12662-7
  9. Azizah, N., Suseno, M., & Hayat, B. (2022). Item analysis of the Rasch model items in the final semester exam Indonesian language lesson. World Journal of English Language, 12(1), 15–26. https://doi.org/10.5430/wjel.v12n1p15
  10. Bannigan, K., & Watson, R. (2009). Reliability and validity in a nutshell. Journal of Clinical Nursing, 18(23), 3237–3243. https://doi.org/10.1111/j.1365-2702.2009.02939.x
  11. Biggs, J., & Tang, C. (2011). Teaching for quality learning at university. What the student does (4th Edn.). In Innovations in Education and Teaching International (4nd editio, Vol. 50, Issue 4). SRHE and Open University Press. https://cetl.ppu.edu/sites/default/files/publications/-John_Biggs_and_Catherine_Tang-_Teaching_for_Quali-BookFiorg-.pdf
  12. Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model : Fundamental measurement in the human sciences (2ed ed.). Lawrence Erlbaum Associates, Inc. https://researchonline.jcu.edu.au/9907/2/9907
  13. Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences, (3rd ed.). Routledge. https://doi.org/https://doi.org/10.1111/j.1745-3984.2003.tb01103.x
  14. Bond, T. G., Yan, Z., & Heene, M. (2020). Applying the Rasch model fundamental measurement in the human sciences (4th Editio). Routledge. https://doi.org/https://doi.org/10.4324/9780429030499
  15. Boone, W. J. (2016). Rasch analysis for instrument development: Why,when,and how? CBE Life Sciences Education, 15(4), 1–7. https://doi.org/10.1187/cbe.16-04-0148
  16. Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch Analysis in the human sciences. Springer. https://doi.org/10.1007/978-94-007-6857-4
  17. Borsboom, D., Mellenbergh, G. J., & Van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061
  18. Brown, A., Nidumolu, A., McConnell, M., Hecker, K., & Grierson, L. (2019). Development and psychometric evaluation of an instrument to measure knowledge, skills, and attitudes towards quality improvement in health professions education: The Beliefs, Attitudes, Skills, and Confidence in Quality Improvement (BASiC-QI) scale. Perspectives on Medical Education, 8(3), 167–176. https://doi.org/10.1007/s40037-019-0511-8
  19. Buntragulpoontawee, M., Khunachiva, J., Euawongyarti, P., Wongpakaran, N., Wongpakaran, T., Kaewma, A., & Ashford, S. (2021). Investigating psychometric properties of the arm activity measure – Thai version (ArmA-TH) sub‐scales using the Rasch model. BMC Medical Research Methodology, 21(1), 1–9. https://doi.org/10.1186/s12874-021-01238-5
  20. Creswell, J. W., & Creswell, J. D. (2017). Research design qualitative, quantitative, and mixed methods approaches (4 th ed). SAGE Publications, Inc.
  21. Curtin, M., Browne, J., Staines, A., & Perry, I. J. (2016). The Early development instrument: An evaluation of its five domains using Rasch analysis. BMC Pediatrics, 16(1), 1–14. https://doi.org/10.1186/s12887-016-0543-8
  22. Darman, D. R., Suhandi, A., Kaniawati, I., Samsudin, A., & Wibowo, F. C. (2024). Development and validation of Scientific Inquiry Literacy Instrument (SILI) using Rasch measurement model. Education Sciences, 14(3), 1–28. https://doi.org/10.3390/educsci14030322
  23. DeMars, C. (2010). Item Response Theory. Oxford University Press, Inc. https://api.pageplace.de/preview/DT0400.9780199703845_A23607784
  24. DeVellis, R. F. (2017). Scale development theory and applications. In SAGE Publication (4th ed). SAGE Publications, Inc. All.
  25. Drake, D., Kennedy, R., & Wallace, E. (2017). The validity and responsiveness of isometric lower body multi-joint tests of muscular strength: A systematic review. Sports Medicine – Open, 3(1), 1–11. https://doi.org/10.1186/s40798-017-0091-2
  26. Ekstrand, J., Westergren, A., Årestedt, K., Hellström, A., & Hagell, P. (2022). Transformation of Rasch model logits for enhanced interpretability. BMC Medical Research Methodology, 22(1), 1–10. https://doi.org/10.1186/s12874-022-01816-1
  27. Engelhard, G., & Wang, J. (2022). Rasch Models for Solving Measurement Problems: Invariant Measurement in the Social Sciences. SAGE Publications, Inc. https://doi.org/10.4135/9781071878675
  28. Fitrah, M., Sofroniou, A., Ofianto, Judijanto, L., & Widihastuti. (2024). Reliability and separation index analysis of mathematics questions integrated with the cultural architecture framework using the Rasch model. Journal of Education and E-Learning Research, 11(3), 499–509. https://doi.org/10.20448/jeelr.v11i3.5861
  29. Frick, H., Strobl, C., & Zeileis, A. (2015). Rasch mixture models for DIF detection: A comparison of old and new score specifications. Educational and Psychological Measurement, 75(2), 208–234. https://doi.org/10.1177/0013164414536183
  30. Gunawan, B., Rahayu, W., & Aziz, T. A. (2024). Rasch model analysis: Validity and reliability of context-based geometry performance assessment instruments Jakarta Cultural Heritage. EduMa: Mathematics Education Learning and Teaching, 13(2), 131. https://doi.org/10.24235/eduma.v13i2.18136
  31. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1992). Fundamentals of item response theory. American Sosiological Association, 21(2), 289–290. http://www.jstor.org/stable/2075521
  32. Hamzah, F. M., Rashid, M. N. A., Rahman, M. N. A., & Rasul, M. S. (2022). Evaluating the validity and reliability of authentic learning instruments using Rasch Model. International Journal of Global Optimization and Its Application, 1(3), 182–189. https://doi.org/10.56225/ijgoia.v1i3.69
  33. Heale, R., & Twycross, A. (2015). Validity and reliability in quantitative studies. Evidence-Based Nursing, 18(3), 66–67. https://doi.org/10.1136/eb-2015-102129
  34. Heaney, A., McKenna, S. P., Hagell, P., & Sengupta, R. (2020). Improving scoring precision and internal construct validity of the bath ankylosing spondylitis disease activity index using Rasch measurement theory. Journal of Rheumatology, 47(3), 354–361. https://doi.org/10.3899/jrheum.180943
  35. Hoeve, K. B. (2022). A validity framework for accountability: Educational measurement and language testing. Language Testing in Asia, 12(1), 1–14. https://doi.org/10.1186/s40468-021-00153-2
  36. Holden, L. T. R., & Tanenbaum, G. J. (2023). Modern assessments of intelligence must be fair and equitable. Journal of Intelligence, 11(6), 1–20. https://doi.org/10.3390/jintelligence11060126
  37. Hope, D., Adamson, K., McManus, I. C., Chis, L., & Elder, A. (2018). Using differential item functioning to evaluate potential bias in a high stakes postgraduate knowledge based assessment. BMC Medical Education, 18(1), 1–7. https://doi.org/10.1186/s12909-018-1143-0
  38. Huang, S. M., Fang, S. C., Hung, C. T., & Chen, Y. H. (2022). Psychometric evaluation of a nursing competence assessment tool among nursing students: A development and validation study. BMC Medical Education, 22(1), 1–9. https://doi.org/10.1186/s12909-022-03439-y
  39. Humphris, G. M., Watson, E., Sharpe, M., & Ozakinci, G. (2018). Unidimensional scales for fears of cancer recurrence and their psychometric properties: The FCR4 and FCR7. Health and Quality of Life Outcomes, 16(1), 1–12. https://doi.org/10.1186/s12955-018-0850-x
  40. Joo, S., Ali, U., Robin, F., & Shin, H. J. (2022). Impact of differential item functioning on group score reporting in the context of large-scale assessments. Large-Scale Assessments in Education, 10(1), 1–22. https://doi.org/10.1186/s40536-022-00135-7
  41. Kesmodel, U. S. (2018). Cross-sectional studies – what are they good for? Acta Obstetricia et Gynecologica Scandinavica, 97(4), 388–393. https://doi.org/10.1111/aogs.13331
  42. Lamoureux, E. L., Pesudovs, K., Thumboo, J., Saw, S. M., & Wong, T. Y. (2009). An evaluation of the reliability and validity of the visual functioning questionnaire (VF-11) using rasch analysis in an Asian population. Investigative Ophthalmology and Visual Science, 50(6), 2607–2613. https://doi.org/10.1167/iovs.08-2359
  43. Linacre, J. M. (2002). Optimizing rating scale category effectiveness optimizing rating scale category effectiveness university of Chicago. Journal of Applied Measurement, 3(1), 85–106. https://www.researchgate.net/publication/292264351
  44. Linacre, J. M. (2010). Predicting responses from rasch measures. Journal of Applied Measurement, 11(1), 1–10. https://www.winsteps.com/a/Linacre-Predicting.pdf
  45. Linacre, J. M. (2012). What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions, 6(2), 878. https://www.rasch.org/rmt/rmt162.htm
  46. Linacre, J. M. (2017). Rasch measurement transactions: Teaching Rasch measurement. Autumn., 31(2), 1630–1642. www.rasch.org/rmt/rmt111n.htm
  47. Linacre, J. M. (2024). Reasonable mean-square fit values revisited. Rasch Measurement Transactions, 36(3), 1–6. https://www.rasch.org/rmt/rmt83b.htm?utm_source=chatgpt.com
  48. Linacre, J. M., Boone, W., Green, R., Trevor, G., & Christine, M. (2023). A User’s Guide to WINSTEPS® / MINISTEP Rasch-model computer programs (5.10; pp. 2–4). Winsteps.com. www.winsteps.com/manuals.htm
  49. Maric, D., Fore, G. A., Nyarko, S. C., & Varma-Nelson, P. (2023). Measurement in STEM education research: A systematic literature review of trends in the psychometric evidence of scales. International Journal of STEM Education, 10(1), 1–22. https://doi.org/10.1186/s40594-023-00430-x
  50. Papini, N. M., Jung, M., Kang, M., Lopez, N. V., & Herrmann, S. D. (2023). Development and Rasch analysis of the 18-Item Health Resilience Profile (HRP). Nutrients, 15(4), 1–17. https://doi.org/10.3390/nu15040807
  51. Pilotti, M. A. E. (2021). What lies beneath sustainable education? Predicting and tackling gender differences in STEM academic success. Sustainability, 13(4), 1671. https://doi.org/10.3390/su13041671
  52. Ranyard, R., McNair, S., Nicolini, G., & Duxbury, D. (2020). An item response theory approach to constructing and evaluating brief and in-depth financial literacy scales. Journal of Consumer Affairs, 54(3), 1121–1156. https://doi.org/10.1111/joca.12322
  53. Rezai, A. (2022). Fairness in classroom assessment: development and validation of a questionnaire. Language Testing in Asia, 12(1), 1–27. https://doi.org/10.1186/s40468-022-00162-9
  54. Ridwan, M. R., Hadi, S., & Jailani, J. (2023). Measurement of Psychometric Pproperties Numerical Aptitude Assessment Scale for prospective high school students: A Rasch model analysis. TEM Journal, 12(4), 2416–2429. https://doi.org/10.18421/TEM124-54
  55. Schumacker, R. E., & Smith, E. V. (2007). Reliability a rasch perspective. Educational and Psychological Measurement, 67(3), 394–409. https://doi.org/10.1177/0013164406294776
  56. Sijtsma, K., & van der Ark, L. A. (2022). Advances in nonparametric item response theory for scale construction in quality-of-life research. Quality of Life Research, 31(1), 1–9. https://doi.org/10.1007/s11136-021-03022-w
  57. Supriatna, M., Suryana, D., Anzhali, M. N., & Noor, A. M. (2025). Evaluating career self-awareness instruments for victims of violence using the Rasch model. BMC Psychology, 13(1), 1–13. https://doi.org/10.1186/s40359-025-02762-5
  58. Tan, L. Y., McLean, S., Kim, Y. A., & Vitta, J. P. (2024). Rasch modelling vs. item facility: Implications on the validity of assessments of Asian EFL/ESL vocabulary knowledge and lexical sophistication modelling. Language Testing in Asia, 14(1), 1–22. https://doi.org/10.1186/s40468-024-00327-8
  59. Tennant, A., & Küçükdeveci, A. A. (2023). Application of the Rasch measurement model in rehabilitation research and practice: Early developments, current practice, and future challenges. Frontiers in Rehabilitation Sciences, 4(July), 01–17. https://doi.org/10.3389/fresc.2023.1208670
  60. Tennant, A., McKenna, S. P., & Hagell, P. (2004). Application of Rasch analysis in the development and application of quality of life instruments. Value in Health, 7(Suplement. 1), 22–26. https://doi.org/10.1111/j.1524-4733.2004.7s106.x
  61. Tutz, G. (2024). Invariance of comparisons: Separation of item and person parameters beyond Rasch models. Journal of Mathematical Psychology, 122(August), 102876. https://doi.org/10.1016/j.jmp.2024.102876
  62. Vincent, J. I., MacDermid, J. C., King, G. J. W., & Grewal, R. (2015). Rasch analysis of the Patient Rated Elbow Evaluation questionnaire. Health and Quality of Life Outcomes, 13(1), 1–13. https://doi.org/10.1186/s12955-015-0275-8
  63. Wiley, J. A., Martin, J. L., Herschkorn, S. J., & Bond, J. (2015). A new extension of the binomial error model for responses to items of varying difficulty in educational testing and attitude surveys. PLoS ONE, 10(11), 1–16. https://doi.org/10.1371/journal.pone.0141981
  64. Wilson, M. (2023). Constructing Measures: An Item Response Modeling Approach (Second Edi). Routledge Falmer Taylor & Francis Group. https://doi.org/10.4324/9781003286929
  65. Woudstra, A. J., Meppelink, C. S., Pander Maat, H., Oosterhaven, J., Fransen, M. P., & Dima, A. L. (2019). Validation of the short assessment of health literacy (SAHL-D) and short-form development: Rasch analysis. BMC Medical Research Methodology, 19(1), 1–10. https://doi.org/10.1186/s12874-019-0762-4
  66. Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370. https://www.rasch.org/rmt/rmt83b.htm
  67. Wright, B. D., & Masters, G. N. (1982). Rating scale analysis item response modeling approach. Mesa Press.
  68. Wright, B. D., & Stone, M. H. (1999). Measurement essentials. In Measurement (2 Ed). WIDE RANGE, INC. https://www.rasch.org/measess/me-all.pdf
  69. Yang, H., Su, J., & Bradley, K. D. (2020). Applying the Rasch model to evaluate the Self-Directed Online Learning Scale (SDOLS) for graduate students. International Review of Research in Open and Distributed Learning, 21(3), 99–120. https://doi.org/https://doi.org/10.19173/irrodl.v21i3.4654
  70. Yusop, S. R. M., Rasul, M. S., Yasin, R. M., & Hashim, H. U. (2023). Identifying and validating vocational skills domains and indicators in classroom assessment practices in TVET. Sustainability (Switzerland), 15(6), 1–27. https://doi.org/10.3390/su15065195
  71. Zumbo, B. D. (2006). Psychometric methods for investigating DIF and test bias during test adaptation across languages and cultures (pp. 1–85). Lawrence Erlbaum Associates. https://faculty.educ.ubc.ca/zumbo/talks/Zumbo_ITC.PDF
  72. Zumbo, B. D., & Gelin, M. N. (2005). A matter of test bBias in educational policy research: Bringing the context into picture by iInvestigating sociological / community moderated (or mediated) test and item bias. Journal of Educational Research & Policy Studies, 5(1), 1–23. https://faculty.educ.ubc.ca

 

Author's Affiliation

   I Ketut Darma
Department of Mechanical Engineering, Politeknik Negeri Bali, Indonesia

   I Gde Nyoman Sangka
Department of Electrical Engineering, Politeknik Negeri Bali, Indonesia

Copyrights & License

I Ketut Darma, I Gde Nyoman Sangka, 2026

This work is licenced under a Creative Commons Attribution 4.0 International License.

Article Details

How to Cite :

Psychometric Evaluation of an Applied Mathematics Examination Instrument Based on The Rasch Model. I Ketut Darma, I Gde Nyoman Sangka, 9(8), 4282-4295. Retrieved from https://ijcsrr.org/single-view/?id=27175&pid=27136

Most read articles by the same author(s)

I Ketut Darma, I Gde Nyoman Sangka,

View

23

Citation

Downloads

Article Details