Psychometric validation on the heat and temperature-critical thinking skills test (HnT-CTST) using the rasch model

Raden Hanna Rifani; Duden Saepuzaman; Winny Liliawati; John Richard Huwae; Muhammad Zahran; Asep Sopandi

doi:10.21067/mpej.v10i1.13481

Authors

Raden Hanna Rifani Universitas Pendidikan Indonesia, Indonesia https://orcid.org/0009-0001-1645-7473
Duden Saepuzaman Universitas Pendidikan Indonesia, Indonesia https://orcid.org/0000-0002-7810-2328
Winny Liliawati Universitas Pendidikan Indonesia, Indonesia https://orcid.org/0000-0002-9938-618X
John Richard Huwae Universitas Pendidikan Indonesia, Indonesia https://orcid.org/0009-0009-3101-5559
Muhammad Zahran Okayama University, Japan https://orcid.org/0000-0002-6816-3899
Asep Sopandi Junior High School One Of Cianjur, Indonesia

DOI:

https://doi.org/10.21067/mpej.v10i1.13481

Keywords:

Critical thinking, Heat and temperature, Rasch model, Two-tier test

Abstract

Critical thinking is an essential competency in physics learning, particularly in temperature and heat topics, which are often associated with misconceptions and difficulties in conceptual understanding. Therefore, a valid and reliable instrument is required to accurately assess students’ critical thinking skills. This study aimed to develop and validate the Heat and Temperature–Critical Thinking Skills Test (HnT-CTST), a two-tier multiple-choice instrument constructed based on Facione’s critical thinking framework. A quantitative approach with a cross-sectional survey design was employed. The instrument consisted of 18 items that were reviewed by five experts and administered to 88 eighth-grade students at a public junior high school in Cianjur Regency, Indonesia. Rasch analysis was used to evaluate the instrument’s psychometric properties, including content validity, reliability, model fit, unidimensionality, item difficulty, and gender fairness through Differential Item Functioning (DIF) analysis. The content validity results showed Aiken’s V values ranging from 0.96 to 1.00, indicating excellent validity. Item reliability reached 0.94, person reliability was 0.71, and KR-20 was 0.74, demonstrating satisfactory internal consistency. Infit and outfit MNSQ values for both persons and items were within the acceptable range, confirming good model fit. Principal Component Analysis (PCA) of residuals revealed that 37.4% of the variance was explained by the primary construct, supporting unidimensionality. The Wright Map indicated that the instrument effectively represented varying levels of student ability. DIF analysis found no significant gender bias across all items. These findings demonstrate that the HnT-CTST is a valid, reliable, and fair instrument for measuring junior high school students’ critical thinking skills in temperature and heat concepts and can support future assessment and instructional practices.

Downloads

Download data is not yet available.

References

A Uyanah, D., & U. I, N. (2023). The theoretical and empirical equivalence of Cronbach alpha and Kuder–Richardson Formula 20 reliability coefficients. International Research Journal of Innovations in Engineering and Technology, 7(5), 17–23. https://doi.org/10.47001/IRJIET/2023.705003

Azizah, N., Suseno, M., & Hayat, B. (2022). Item analysis of Rasch model items in the final semester examination of Indonesian language lessons. World Journal of English Language, 12(1), 15–26. https://doi.org/10.5430/wjel.v12n1p15

Bailin, S. (2002). Critical thinking and science education. Science & Education, 11(4), 361–375. https://doi.org/10.1023/A:1016042608621

Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge.

Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch analysis in the human sciences. Springer.

Boone, W. J. (2016). Rasch analysis for instrument development: Why, when, and how? CBE Life Sciences Education, 15(4), rm4. https://doi.org/10.1187/cbe.16-04-0148

Brundage, M. J., Meltzer, D. E., & Singh, C. (2024). Investigating introductory and advanced students’ difficulties with change in internal energy, work, and heat transfer using a validated instrument. Physical Review Physics Education Research, 20(1), 010115. https://doi.org/10.1103/PhysRevPhysEducRes.20.010115

Caleon, I., & Subramaniam, R. (2010). Development and application of a three-tier diagnostic test to assess secondary students’ understanding of waves. International Journal of Science Education, 32(7), 939–961. https://doi.org/10.1080/09500690902890130

Ekstrand, J., Westergren, A., Årestedt, K., Hellström, A., & Hagell, P. (2022). Transformation of Rasch model logits for enhanced interpretability. BMC Medical Research Methodology, 22(1), Article 335. https://doi.org/10.1186/s12874-022-01816-1

Facione, P. A. (2020). Critical thinking: What it is and why it counts (2020 update). Insight Assessment.

Frost, M. H., Reeve, B. B., Liepa, A. M., Stauffer, J. W., Hays, R. D., & Sloan, J. A. (2007). What is sufficient evidence for the reliability and validity of patient-reported outcome measures? Value in Health, 10(Suppl. 2), S94–S105. https://doi.org/10.1111/j.1524-4733.2007.00272.x

García-Carmona, A. (2025). Scientific thinking and critical thinking in science education: Two distinct but symbiotically related intellectual processes. Science & Education, 34(1), 227–245. https://doi.org/10.1007/s11191-023-00460-5

Gurel, D. K., Eryılmaz, A., & McDermott, L. C. (2015). A review and comparison of diagnostic instruments to identify students’ misconceptions in science. Eurasia Journal of Mathematics, Science and Technology Education, 11(5), 989–1008. https://doi.org/10.12973/eurasia.2015.1369a

Giri, V., & Paily, M. U. (2020). Effect of scientific argumentation on the development of critical thinking. Science & Education, 29(3), 673–690. https://doi.org/10.1007/s11191-020-00120-y

Setyaedhi, H. S. (2024). Comparative test of Cronbach’s alpha reliability coefficient, KR-20, KR-21, and split-half method. Journal of Education Research and Evaluation, 8(1), 47–57. https://doi.org/10.23887/jere.v8i1.68164

Hidayah, N. (2023). Validity and reliability test of teaching materials using Aiken’s V formula and SPSS 22. Schola: Journal of Education and Learning, 1(2), 62–68.

Huebner, A., & Skar, G. B. (2021). Conditional standard error of measurement: Classical test theory, generalizability theory and many-facet Rasch measurement with applications to writing assessment. Practical Assessment, Research & Evaluation, 26(1), Article 22. https://doi.org/10.7275/1ef7-rm16

Kassiavera, S., Suparmi, S., Cari, C., & Sukarmin, S. (2024). Application of Rasch model in two-tier test for assessing critical thinking in physics education. Journal of Baltic Science Education, 23(6), 1227–1242. https://doi.org/10.33225/jbse/24.23.1227

Kurniawan, A., Istiyono, E., & Naba, S. D. (2024). Item quality analysis of physics concept understanding test with Rasch model. Jurnal Ilmu Pendidikan Fisika, 9(3), 474–484. https://doi.org/10.26737/jipf.v9i3.5692

Kusumadani, A. I., Afandy, H., Agustina, L., Astuti, R., & Waluyo, M. (2025). Evaluation of higher-order thinking skills of middle school students on vibration and wave topics using Rasch measurement. Jurnal Penelitian Pendidikan IPA, 11(5), 74–84. https://doi.org/10.29303/jppipa.v11i5.10900

Ku, K. Y. L. (2009). Assessing students’ critical thinking performance: Urging for measurements using multi-response format. Thinking Skills and Creativity, 4(1), 70–76. https://doi.org/10.1016/j.tsc.2009.02.001

Lestari, N., Nisa, C., Mutazam, M., & Soeharto, S. (2023). Developing two-tier test of critical thinking ability using Rasch model. Jurnal Ilmu Pendidikan Fisika, 8(2), 259–268. https://doi.org/10.26737/jipf.v8i2.3737

Linacre, J. M. (1994). Sample size and item calibration stability. Rasch Measurement Transactions, 7(4), 328.

Maulidiansyah, D., Meutia, I., & Istiyono, E. (2021). Computer-based two-tier diagnostic test to identify critical thinking skills in optical instruments. Jurnal Penelitian dan Evaluasi Pendidikan, 25(2), 168–181.

Mohamad, M. M., Sulaiman, N. L., Sern, L. C., & Salleh, K. M. (2015). Measuring the validity and reliability of research instruments. Procedia - Social and Behavioral Sciences, 204, 164–171. https://doi.org/10.1016/j.sbspro.2015.08.129

Müller, M. (2020). Item fit statistics for Rasch analysis: Can we trust them? Journal of Statistical Distributions and Applications, 7(1), Article 14. https://doi.org/10.1186/s40488-020-00108-7

Nurazizah, A., Natsir, N., & Zulkhaeriyah, Z. (2023). Development of innovative assessment methods in the context of modern learning. International Journal of Education, Language, Literature, Arts, Culture, and Social Humanities, 1(4), 84–93. https://doi.org/10.59024/ijellacush.v1i4.424

Nurjanah, S., Istiyono, E., Widihastuti, W., Iqbal, M., & Kamal, S. (2023). The application of Aiken’s V method for evaluating the content validity of instruments that measure the implementation of formative assessments. Journal of Research and Educational Research Evaluation, 12(2), 125–135.

Ramadani, S., Sinaga, P., & Liliawati, W. (2025). Critical thinking in science learning: Systematic literature review. Jurnal Pemberdayaan Masyarakat, 4(1), 183–196. https://doi.org/10.46843/jpm.v4i1.378

Romadloni, N. M. A., Amaruddin, H., Adi, H. M. M., & Qurroti, I. A. L. (2022). Validity and reliability analysis using the Rasch model to measure the quality of mathematics test items of vocational high schools. Journal of Research and Educational Research Evaluation, 11(1), 93–103.

Salame, I. I., Fadipe, O., & Akter, S. (2025). Examining difficulties, challenges, and alternative conceptions students exhibit while learning about heat and temperature concepts. Interdisciplinary Journal of Environmental and Science Education, 21(2), e2510. https://doi.org/10.29333/ijese/15997

Sarifah, I., Kuswati, E., Suniasih, W., Fahrurrozi, F., & Susilo, G. (2024). Cognitive test instruments to measure student mathematics ability in elementary school: Rasch analysis. Educational Administration: Theory and Practice, 30(4), 1557–1568. https://doi.org/10.53555/kuey.v30i4.1557

Song, H., & Cai, L. (2024). Interactive learning environment as a source of critical thinking skills for college students. BMC Medical Education, 24(1), Article 683. https://doi.org/10.1186/s12909-024-05247-y

Stemler, S. E., & Naples, A. (2021). Rasch measurement v. item response theory: Knowing when to cross the line. Practical Assessment, Research & Evaluation, 26(1), 1–16. https://doi.org/10.7275/v2gd-4441

Ekolu, S. O., & Quainoo, H. (2019). Reliability of assessments in engineering education using Cronbach’s alpha, KR and split-half methods. Global Journal of Engineering Education, 21(1), 24–29.

Sumintono, B., & Widhiarso, W. (2015). Aplikasi model Rasch untuk penelitian ilmu-ilmu sosial (Edisi revisi). Trim Komunikata Publishing House.

Sunni, F. M., Dawana, I. R., Arrahmaniyyah, R., Sunarti, T., Madlazim, M., Wasis, W., & Rizki, I. A. (2024). Research trend of Rasch model in the development of scientific literacy assessment through bibliometrics. Jurnal IPA dan Pembelajaran IPA, 8(4), 413–429. https://doi.org/10.24815/jipi.v8i4.42307

Şahin, A., & Anıl, D. (2017). The effects of test length and sample size on item parameters in item response theory. Educational Sciences: Theory & Practice, 17(1), 321–335. https://doi.org/10.12738/estp.2017.1.0270

Taber, K. S. (2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2

Tugirin, T., Masak, M., Husen, M., & Kuswanto, H. (2025). A systematic literature review on critical thinking skills in physics education. International Journal of Education in Mathematics, Science and Technology, 13(5), 1268–1282. https://doi.org/10.46328/ijemst.5709

Utama, R., & Toifur, M. (2025). Investigating senior high school students’ conceptual understanding of heat and temperature: A qualitative analysis in physics education. Jurnal Pendidikan Fisika, 13(2), 244–258. https://doi.org/10.24127/jpf.v13i2.14137

Zivanka, S. R., Silitonga, T. H., & Silitonga, M. (2022). The application of the Rasch model to develop a two-tier multiple-choice test to measure higher-order thinking skills on motion and force. Journal of Innovative Science Education, 11(1), 36–47. https://doi.org/10.21009/JISAE.111.05