Articles

Psychometric Evaluation of an Applied Mathematics Examination Instrument Based on The Rasch Model

/0 Comments/in /by

Administering the same examination to students from different programs can result in uneven measurement of competence and potentially flawed academic decisions. In many vocational settings, psychometric reviews of examination instruments are uncommon, increasing the risk of inaccurate evaluation. This study investigates the Applied Mathematics Final Semester Examination (FSE) instruments in the Mechanical Engineering Department, Bali State Polytechnic, through the Rasch model. The data came from 206 students who answered 40 multiple-choice questions. The test was delivered online via the institutional e-learning system but taken in the classroom. Winsteps 5.9 was used to assess construct validity, reliability, separation indices, item difficulty in relation to ability, and Differential Item Functioning (DIF) across programs. Demonstrates that the instrument meets all psychometric standards. Nine components that didn’t fit nonetheless showed the planned construct. The separation indices were 3.46 and 6.18, and the person and item reliability were both over 0.90. The major dimension explained more than 63% of the variation, which indicated that it was one-dimensional. Usually, the difficulty levels were right for the students, but not at the extremes. Eight things suggested that there might be bias. The result is a one-of-a-kind Rasch evaluation of a vocational mathematics FSE that combines reliability, unidimensionality, and DIF analysis.  It can be used as a model for other situations and to make vocational math tests fair, valid, and reliable.

Unidimensionality Analysis and Differential Item Functioning in the Applied Mathematics Final Examination Test of Politeknik Negeri Bali Students Using the Rasch Model

/in /by

The objectives of this research are (1) to analyze the fulfillment of unidimensionality assumptions on Applied Mathematics FSE instruments by the Rasch Model, (2) to identify and analyze the existence of items that have differential item functioning (DIF) among study programs in the polytechnic, and (3) to interpret as well as recommend improvements to instruments based on results from analysis about unidimensionality and DIF. This survey is quantitative research with a cross-sectional design. The sample consists of 206 students coming from three study programs: the Mechanical Engineering, the Refrigeration and Air Conditioning Engineering, and the Utility Engineering Technology, Politeknik Negeri Bali (PNB), selected by the total sampling method. The research instrument comprises five questions constructed according to an outcome-based education (OBE) curriculum where content validity was checked through Aiken’s V method, giving a value equal to 0.91. Data were analyzed using Winsteps 5.9 software based on the Rasch model. Results of the analysis indicated that the test has outstanding reliability and separation indices both on respondents and items. Most items fall within the fitting criteria of the Rasch Model. The unidimensionality test proved that it can consistently measure a single construct in applied mathematics competency. Based on DIF analysis, most items work well across different learning categories, but a few items have large and significant DIFs. These imply that this instrument is suitable to be used as an assessment tool for learning outcomes where several parts still need improvement to enhance fairness in measuring. It raises awareness about conducting comprehensive evaluations on assessment tools within the OBE context in vocational education.

Validation of Applied Mathematics Exam Test Using Rasch Model Approach: Case Study in Diploma 3 Study Program of Refrigeration and Air Conditioning Engineering Politeknik Negeri Bali

/in /by

This study validates the Applied Mathematics exam (AME) test on Diploma 3 (D3) Study Program of Refrigeration and Air Conditioning Engineering (RACE), Politeknik Negeri Bali (PNB) to determine the validity, reliability, unidemensional, level of difficulty, and discriminatory power of the test. Validation uses the modern test theory approach of the Rasch model. Data were collected during the online final exam of the even semester of the 2023/2024 academic year. The instrument uses a multiple-choice test form with 5 answer options. The sample involved 73 even semester students of the D3 RACE study program who took applied mathematics courses. The data collected were analyzed using the Rasch Model assisted by the Winsteps application. The results of the analysis show that the AME test has an adequate level of validity, most of the questions meet the fit criteria for the Rasch Model. The level of test reliability is categorized as very good with a person and item reliability value of 0.90. Several questions still show misfits that require improvement. Item difficulty and person ability show a proportional distribution between the level of difficulty of the questions and the students’ abilities. Overall, the test’s discriminatory power is categorized as good, although there is one question that needs to be reviewed further for improvement. The implication is that the use of the Rasch Model in validating online test instruments can help teachers in compiling questions that are more valid, reliable, and in accordance with the level of student ability. The implication is that the application of the Rasch Model in validating test instruments can help lecturers in constructing more valid and reliable tests. The results of this study can be used as an empirical example of the application of the Rasch model theory to produce more valid and reliable measurements. It is recommended that the development of future test tests really needs to pay attention to the balance between the level of difficulty of the items and the abilities of the students, and ensure that the measurements are more valid and reliable, especially in the context of polytechnic education.