Application of the RASCH model to analyze Critical Thinking Skills Instruments (CTSI) on static fluid concept

Lingga Thursina Fajriyati; Duden Saepuzaman; Lina Aviyanti; Winny Liliawati

doi:10.21067/mpej.v10i1.12442

Authors

Lingga Thursina Fajriyati Universitas Pendidikan Indonesia, Indonesia
Duden Saepuzaman Universitas Pendidikan Indonesia, Indonesia
Lina Aviyanti Universitas Pendidikan Indonesia, Indonesia
Winny Liliawati Universitas Pendidikan Indonesia, Indonesia

DOI:

https://doi.org/10.21067/mpej.v10i1.12442

Keywords:

critical thinking skills, instrument, Rasch model, static fluid

Abstract

This study aims to analyze the reliability, validity of item analysis, and estimation of respondents' abilities in assessing students' critical thinking skills (CTS) instrument on static fluid material using the Rasch model. There are five indicators of the CTS instrument, namely Basic Clarification (BCL), Decision Basis (TBD), Inference (INF), Advanced Clarification (ACL), and Assumption and Integration (SIN), with a total of 20 items. The Rasch model was chosen because it provides an in-depth analysis of item characteristics, respondents' abilities, and allows the identification of imprecise or biased items. Although the theoretical background is presented briefly due to the methodological focus of this study, previous studies have primarily emphasized improving CTS through learning interventions rather than developing standardized measurement instruments. The research method used is quantitative with a descriptive design using Win steps 3.73 software for data analysis. The sample consisted of 215 high school students in grades XI and XII majoring in science, consisting of 120 female students and 95 male students in West Java. The results of the study showed that the test instrument had high reliability with an individual reliability of 0.73 and an item reliability of 0.92, as well as good internal consistency with a Cronbach's Alpha of 0.78. The validity of this instrument also met the acceptance criteria of the Rasch model, with MNSQ infit and MNSQ outfit values ranging from 0.5 to 1.5, indicating acceptable model fit. Therefore, the CTS test instrument is suitable for use in high school physics education. This study contributes to addressing the limited availability of validated CTSCTS instruments specifically designed for static fluid material at the high school level using Rasch analysis.

Downloads

Download data is not yet available.

References

Abdulridah Dhyaaldian, S. M., Hasan Al-Zubaidi, S., A Mutlak, D., Raheem Neamah, N., ALI ALBEER, A. A. M., A Hamad, D., ... & Ghaleb Maabreh, H. (2022). Psychometric evaluation of cloze tests with the Rasch model. International Journal of Language Testing, 12(2), 95-106.

Afifa, M., Khoirunnisa, R., Pratiwi, S. M. V., & Meitaza, D. (2024). Utilizing Rasch Model to Analyze A Gender Gap in Students’ Scientific Literacy on Energy. Jurnal Pendidikan Fisika Indonesia, 20(1), 85-95.

Alameddine, M. A., & Bashir, M. M. (2024). Investigating Strategies for Teaching Critical Thinking in Physics Classrooms. American J Sci Edu Re: AJSER-202.

Altun, S. A., Büyüköztürk, Ş., & Seheryeli, M. Y. (2021). Validity and Reliability Evidence of Professional Obsolescence Scale According to Different Test Theories. International Journal of Assessment Tools in Education, 8(2), 257-278.

Amirzadeh, S., Rasouli, D., & Dargahi, H. (2024). Assessment of validity and reliability of the Feedback Quality Instrument. BMC Research Notes, 17(1), 227.

Andrich, D. (2023). Person–item distribution and the quality of measurement. Measurement: Interdisciplinary Research and Perspectives, 21(3), 145–163.

Antino, M., Alvarado, J. M., Asún, R. A., & Bliese, P. (2020). Rethinking the exploration of dichotomous data: Mokken scale analysis versus factorial analysis. Sociological Methods & Research, 49(4), 839–867. https://doi.org/10.1177/0049124118769090

Avinç, E., & Doğan, F. (2024). Digital literacy scale: Validity and reliability study with the rasch model. Education and information Technologies, 1-47.

Ayub, M. R. S. S. N., Istiyono, E., Munadi, S., Permadi, C., Pattiserlihun, A., & Sudjito, D. N. (2020). Analisa Penilaian Soal Fisika Menggunakan Model Rasch Dengan Program R:-. Jurnal Sains Dan Edukasi Sains, 3(2), 46-52.

Balta, E., & Dogan, C. D. (2024). Investigation of Preknowledge Cheating via Joint Hierarchical Modeling Patterns of Response Accuracy and Response Time. SAGE Open, 14(4), 21582440241297946.

Bichi, A. A., & Talib, R. (2018). Item Response Theory: An Introduction to Latent Trait Models to Test and Item Development. International Journal of Evaluation and Research in Education, 7(2), 142-151.

Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch analysis in the human sciences. Springer.

Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge. https://doi.org/10.4324/9781315814698

Bond, T. G., Yan, Z., & Heene, M. (2021). Applying the Rasch model: Fundamental measurement in the human sciences (4th ed.). Routledge. https://doi.org/10.4324/9781003035861

Chien, C. Y., Lin, C. C., & Wang, W. C. (2022). Visual diagnosis of person–item targeting using person–item maps. Applied Psychological Measurement, 46(6), 454–470.

Choi, S. W. (2014). A review of the Rasch measurement model in psychometrics. Frontiers in Psychology, 5, 1077.

Creswell, J. W. (2014). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (4th ed.). SAGE Publications.

Dijkers, M. P., & Millis, S. R. (2020). The template for intervention description and replication as a measure of intervention reporting quality: Rasch analysis. Archives of rehabilitation research and clinical translation, 2(3), 100055.

Dirlik, E. M., & Kartal, S. (2022). The comparison of the dimensionality results provided by the automated item selection procedure and DETECT analysis. International Journal of Assessment Tools in Education, 9(4), 808–830. https://doi.org/10.21449/ijate.1059200

DeVellis, R. F., & Thorpe, C. T. (2021). Scale development: Theory and applications. Sage publications.

Embretson, S. E., & Reise, S. P. (2013). Item response theory for psychologists. Psychology Press.

Ennis, R. H. (2023). Critical Thinking Across the Disciplines. Inquiry: Critical Thinking Across the Disciplines, 38(2), 1–10.

Etikan, I., & Bala, K. (2017). Sampling and sampling methods. Biometrics & Biostatistics International Journal, 5(6), 00149.

Facione, P. A. (2020). Critical Thinking: What It Is and Why It Counts (2020 update). Insight Assessment.

Fadillah, S. M., Ha, M., Nuraeni, E., & Indriyanti, N. Y. (2023). Exploring Confidence Accuracy and Item Difficulty in Changing Multiple-Choice Answers of Scientific Reasoning Test. Malaysian Journal of Learning and Instruction (MJLI), 20(2), 319-341.

Putra, P. D. A., Sulaeman, N. F., Supeno, & Wahyuni, S. (2023). Exploring students' critical thinking skills using the engineering design process in a physics classroom. The Asia-Pacific Education Researcher, 32(1), 141-149.

Fergadiotis, G., Casilio, M., Dickey, M. W., Steel, S., Nicholson, H., Fleegle, M., ... & Hula, W. D. (2023). Item response theory modeling of the verb naming test. Journal of Speech, Language, and Hearing Research, 66(5), 1718-1739.

Giguère, G., Brouillette-Alarie, S., & Bourassa, C. (2023). A look at the difficulty and predictive validity of LS/CMI items with Rasch modeling. Criminal Justice and Behavior, 50(1), 118-138.

Hagquist, C., & Andrich, D. (2017). Recent advances in analysis of differential item functioning in health research using the Rasch model. Health and quality of life outcomes, 15(1), 181.

Halpern, D. F. (2014). Thought and Knowledge: An Introduction to Critical Thinking (5th ed.). Psychology Press.

Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory (Vol. 2). Sage.

Handayani, S., Sukmawati, E., & Rahmawati, D. (2023). Analysis of Students' Responses Using Rasch Model in Physics Learning. Journal of Educational Measurement and Evaluation, 15(2), 123–132.

Hudha, M. N., & Batlolona, J. R. (2017). How are the physics critical thinking skills of the students taught by using inquiry-discovery through empirical and theorethical overview?. Eurasia Journal of Mathematics, Science and Technology Education, 14(2), 691-697.

Humphry, S., Montuoro, P., & Maxwell, C. (2024). Cumulative ordering as evidence of construct validity for assessments of developmental attributes. Journal of Psychoeducational Assessment, 42(1), 60-73.

Holland, P. W., & Wainer, H. (2012). Differential item functioning. Routledge.

Irwanto, I. (2023). Improving Preservice Chemistry Teachers' Critical Thinking and Science Process Skills Using Research-Oriented Collaborative Inquiry Learning. Journal of Technology and Science Education, 13(1), 23-35.

Ismail, S. N., Muhammad, S., Omar, M. N., & Shanmugam, K. S. (2022). THE PRACTICE OF CRITICAL THINKING SKILLS IN TEACHING MATHEMATICS: TEACHERS’PERCEPTION AND READINESS. Malaysian Journal of Learning and Instruction, 19(1), 1-30.

Jamil, M., Hafeez, F. A., & Muhammad, N. (2024). Critical thinking development for 21st century: Analysis of Physics curriculum. Journal of Social & Organizational Matters, 3(1), 1-10.

Juandi, T., Kaniawati, I., Samsudin, A., & Riza, L. (2024). Prospective teachers’ perception of critical and reflective thinking skills on modern physics: Rasch Analysis. Journal for the Education of Gifted Young Scientists, 12(3), 137-150.

Kaltakci-Gurel, D., Eryilmaz, A., & McDermott, L. C. (2017). Development and application of a four-tier test to assess pre-service physics teachers’ misconceptions about geometrical optics. ReseaRch in science & Technological educaTion, 35(2), 238-260.

Kamilah, D. S., Muki, B. G., Aviyanti, L., & Suhandi, A. (2025). Review of misconceptions in physics among Indonesian high school students: Diagnosis, causes, and remediation. Momentum: Physics Education Journal, 9(1), 144–162. https://doi.org/10.21067/mpej.v9i1.11056

Kelsey Hall, E. D., & Starzec, K. (2024). Using an Interrupted Case Study to Engage Undergraduates’ Critical Thinking Style and Enhance Content Knowledge. Journal on Empowering Teaching Excellence, Spring 2024, 46.

Killip, S. C., MacDermid, J. C., Wouters, R. M., Sinden, K. E., Gewurtz, R. E., Selles, R. W., & Packham, T. L. (2022). Rasch analysis of the brief Michigan Hand Questionnaire in patients with thumb osteoarthritis. BMC Musculoskeletal Disorders, 23(1), 551.

Köhler, C., & Hartig, J. (2017). Practical significance of item misfit in educational assessments. Applied Psychological Measurement, 41(5), 388-400.

Laliyo, L. A. R., Tangio, J. S., Sumintono, B., Jahja, M., & Panigoro, C. (2020). Analytic Approach of Response Pattern of Diagnostic Test Items in Evaluating Students' Conceptual Understanding of Characteristics of Particle of Matter. Journal of Baltic Science Education, 19(5), 824-841.

Lin, J., Li, H., & Wang, Y. (2021). Analyzing item difficulty and test reliability in educational measurement: A Rasch model approach. Journal of Educational Measurement, 58(2), 120-135.

Linacre, J. M. (2020). A user's guide to Winsteps: Rasch-model computer programs. Winsteps.com.

Liu, J., Sun, M., Liu, Z., & Xu, Y. (2023). Pre-Service Teachers’ Instructional Innovation Capabilities: A Many-Faceted Rasch Model Analysis. SAGE Open, 13(2), 21582440231218802. https://doi.org/10.1177/21582440231218802

Lei, K., and Kathleen, M. (2019). The gap in research on critical thinking skills in physics education. Physics Education Research, 15(3), 234-245.

Loverude, M. E., Kautz, C. H., & Heron, P. R. (2003). Helping students develop an understanding of Archimedes’ principle. I. Research on student understanding. American Journal of Physics, 71(11), 1178-1187.

Mukhibin, A., Rusyid, H. K., Lutfi, A., Herman, T., & Utomo, D. A. S. (2023). An Analysis of Students’ Mathematical Self-Efficacy Instruments Using Rasch Model. Indonesian Journal of Mathematics Education, 6(2), 72–80. https://doi.org/10.31002/ijome.v6i2.994

Müller, M. (2020). Item fit statistics for Rasch analysis: can we trust them? Journal of Statistical Distributions and Applications, 7(5).

Meijer, R. R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25(2), 107–135.

Moore, M., & Gordon, P. C. (2015). Reading ability and print exposure: Item response theory analysis of the author recognition test. Behavior research methods, 47, 1095-1109.

Natanael, Y., Salsabilla, R., Aulia, D., Khoirunnisa, D., Munawar, H. N., & Hidayat, N. S. (2023). Rasch Rating Scale Model: Bias Detection and Validation Test of Indonesian-Adolescent Life Satisfaction Scale. ResearchGate.

Navarro-González, M. C., Padilla, J. L., & Benítez, I. (2024). Analyzing measurement invariance for studying the gender gap in educational testing: A mixed studies systematic review. European Journal of Psychological Assessment.

Paul, R., and Elder, L. (2020). Critical Thinking: Tools for Taking Charge of Your Learning and Your Life (4th ed.). Rowman & Littlefield.

Pereira, V. V., Samsudin, A., & Utama, J. A. (2023). STUDENT WORKSHEETS OF PBL AND PROBING PROMPTING TECHNIQUE ON CRITICAL THINKING SKILLS. Journal of Teaching and Learning Physics, 8(2), 89-98.

Planinic, M., Boone, W. J., Susac, A., & Ivanjek, L. (2019). Rasch analysis in physics education research: Why measurement matters. Physical Review Physics Education Research, 15(2), 020111.

Plouffe, R. A., Kowalski, C. M., Tremblay, P. F., Saklofske, D. H., Rogoza, R., Di Pierro, R., & Chahine, S. (2021). Gender Differences or Gender Bias?. European Journal of Psychological Assessment.

Prafitasari, F., Sukarno, S., & Muzzazinah, M. (2021). Integration of critical thinking skills in science learning using blended learning system. International Journal of Elementary Education, 5(3), 434-445.

Samsudin, A., Cahyani, P. B., Rusdiana, D., Efendi, R., Aminudin, A. H., & Costu, B. (2021). Development of a Multitier Open-Ended Work and Energy Instrument (MOWEI) Using Rasch Analysis to Identify Students' Misconceptions. Cypriot Journal of Educational Sciences, 16(1), 16-32.

Sumintono, B., & Widhiarso, W. (2015). Aplikasi model Rasch untuk penelitian ilmu-ilmu sosial. Trim Komunikata.

Setiawan, D., & Faoziyah, N. (2020). Development of a five-tier diagnostic test to reveal the student concept in fluids. Physics Communication, 4(1), 6-13.

Setyawarno, D., Maryati, & Natadiwijaya, I. F. (2025). Promoting a valid question model for measuring computational thinking skills based on confirmatory factor analysis and Rasch model. Cogent Education, 12(1), 2505339.

Scott, I. A., Hubbard, R. E., Crock, C., Campbell, T., & Perera, M. (2021). Developing critical thinking skills for delivering optimal care. Internal Medicine Journal, 51(4), 488-493.

Skjølberg, K. H., Trysnes, I., & Furrebø, E. F. (2023). Is the Coronavirus Created by the Government to Control Us? Critical Thinking and Conspiracy Beliefs among Norwegian Youth in Upper Secondary Schools. Journal of Social Science Education, 22(4), n4.

Student, S. R. (2022). Appraising Traditional and Purpose-built Person Fit Statistics’ Power to Detect Cheating. Chinese/English Journal of Educational Measurement and Evaluation| 教育测量与评估双语期刊, 3(1), 3.

Stemler, S. E., & Naples, A. (2021). Rasch measurement v. item response theory: Knowing when to cross the line. Practical Assessment, Research & Evaluation, 26, 11.

Swiecki, Z., Ruis, A. R., Gautam, D., Rus, V., & Williamson Shaffer, D. (2019). Understanding when students are active‐in‐thinking through modeling‐in‐context. British journal of educational technology, 50(5), 2346-2364.

Taherdoost, H. (2016). Sampling methods in research methodology; how to choose a sampling technique for research. International Journal of Academic Research in Management, 5(2), 18–27.

Tasçi, G. (2024). Development of a Protein Concept Inventory: A Proposal for Item Scoring and Responding. Science Insights Education Frontiers, 23(2), 3755-3777.

Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd

Tiruneh, D. T., De Cock, M., Weldeslassie, A. G., Elen, J., & Janssen, R. (2017). Measuring critical thinking in physics: Development and validation of a critical thinking test in electricity and magnetism. International Journal of Science and Mathematics Education, 15, 663-682.

Turan, U., Fidan, Y., & Yıldıran, C. (2019). Critical thinking as a qualified decision-making tool.

Tiruneh, D. T., De Cock, M., & Elen, J. (2018). Designing learning environments for critical thinking: examining effective instructional approaches. International journal of science and mathematics education, 16, 1065-1089.

Tutz, G. (2023). Unidimensionality in Rasch Models: Efficient item selection and hierarchical clustering methods based on marginal estimates. arXiv preprint arXiv:2309.00553.

Van der Linden, W. J. (Ed.). (2018). Handbook of item response theory: Three volume set. CRC press.

Walsh, C., Quinn, K. N., Wieman, C., & Holmes, N. G. (2019). Quantifying critical thinking: Development and validation of the physics lab inventory of critical thinking. Physical Review Physics Education Research, 15(1), 010135.

Wang, X., & Chen, L. (2020). Validity and reliability of assessment tools using Rasch analysis in healthcare research. Nursing Research, 69(3), 213-221.

Wang, J., & Tam, T. (2025). Bringing generalized status back in: Cross-national evidence for a unidimensional measure. Social Indicators Research. https://doi.org/10.1007/s11205-025-03595-w

Wieman, C., & Holmes, N. G. (2015). Measuring the impact of an instructional laboratory on the learning of introductory physics. American Journal of Physics, 83(11), 972-978.

Embretson, S. E., & Reise, S. P. (2025). Item response theory: Foundations for psychologists and social scientists. Routledge.

Wilson, M. (2023). Constructing measures: An item response modeling approach. Routledge.

Wilson, K., & Defianty, M. (2024). The critical challenge for ELT in Indonesia: Overcoming barriers in fostering critical thinking in testing-oriented countries. TESOL in Context, 33(1), 82-96.

Wind, S., & Hua, C. (2022). Rasch measurement theory analysis in R. Chapman and Hall/CRC.

Wei, S., Liu, X., Wang, Z., & Wang, X. (2012). Using rasch measurement to develop a computer modeling-based instrument to assess students’ conceptual understanding of matter. Journal of Chemical Education, 89(3), 335–345.

Wolfs, Z. G., Brand-Gruwel, S., & Boshuizen, H. P. (2023). Assessing Tonal Abilities in Elementary School Children: Testing Reliability and Validity of the Implicit Tonal Ability Test Using Rasch Measurement Model. SAGE Open, 13(3), 21582440231199041.

Wright, B. D., & Stone, M. H. (1999). Measurement essentials. Wide Range, Inc.

Xie, Q., Zhong, X., Wang, W. C., & Lim, C. P. (2014). Development of an item bank for assessing generic competences in a higher-education institute: a rasch modelling approach. Higher Education Research & Development, 33(4), 821-835.

Yalinkilic, F., & Gul, S. (2023). Development an achievement test on the subject of “Basic Compounds in the Structure of Living Things”. Science Insights Education Frontiers, 18(2), 2905-2925.

Yamada, Y., Kobayashi, N., Wagman, P., & Håkansson, C. (2025). Validity and reliability of the Japanese version of the occupational balance questionnaire. British Journal of Occupational Therapy, 03080226251329771.

Yıldırım Hoş, H., & Uysal Saraç, M. (2023). A Mixture Rasch Model Analysis of Mathematics Achievement. Kastamonu Education Journal, 31(1), 133–142. https://doi.org/10.24106/kefdergi.1246453

Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF). Ottawa: National Defense Headquarters, 160, 53.

Zou, T., & Bolt, D. M. (2023). Person misfit and person reliability in rating scale measures: The role of response styles. Measurement: Interdisciplinary Research and Perspectives, 21(3), 167-180.

Zhang, Y., Li, Z., & Zhao, X. (2019). Evaluating internal consistency of instruments: Cronbach's alpha and beyond. Measurement and Evaluation in Counseling and Development, 52(1), 42-56. https://doi.org/10.1080/07481756.2018.1486686

Zhang, M., Heffernan, N., & Lan, A. (2023). Modeling and Analyzing Scorer Preferences in Short-Answer Math Questions. arXiv preprint arXiv:2306.00791.