Developing authentic assessment instrument based on multiple representations to measure students' critical thinking skills
Abstract
The purpose of this study was to produce an instruments test used to measure students' critical thinking skills in natural science learning. This research uses a 4-D development model (define, design, develop, and disseminate) involving 118 students at the develop stage and 60 students at the disseminate stage. The instrument developed was an essay test based on multiple representations. Validity was proved by using CVI, and reliability was estimated by using the Item Response Theory. The results showed that the instrument had a very good foreign exchange value. This is reflected in the Aiken V scores on the aspects of substance, construct, language and appearance, respectively, about 1.00, 1.00, 1.00, and 1.00. According to Rasch analysis, the instruments has meet the assumption test for 14 items which is unidimensional, the local independence assumption test, and the parameter invariance assumption test. According to OUTFIT MNSQ Value, the items are fit with PCM 1-PL which functions normally in making measurements. Reliability estimated for items shows a very high consistency of measurement of 0.97 and for person shows a high consistency of measurement of 0.86. The results of the student's CTS measurement showed that the average score was 65.50, with a distribution of high, medium, and low abilities, respectively, about 16.67%, 63.33%, and 20.00%. Thus, according to these results, an authentic assessment based on multiple representations is suitable to measure students' critical thinking skills.
References
Abdurrahman, A., Setyaningsih, C. A., & Jalmo, T. (2019). Implementating multiple representation-based worksheet to develop critical thinking skills. Journal of Turkish Science Education, 16(1), 138–155. https://doi.org/10.12973/tused.10271a
Amalia, S., & Wuryandani, W. (2020). Socio-cultural based learning module for critical thinking ability in elementary school: systematic search. Acta Educationis Generalis, 10(2), 180–187. https://doi.org/10.2478/atd-2020-0017
Arends, R. I., & Kilcher, A. (2010). Teaching for student learning: Becoming an accomplished teacher. In Teaching for Student Learning: Becoming an Accomplished Teacher. https://doi.org/10.4324/9780203866771
Boone, W. J., & Noltemeyer, A. (2017). Rasch analysis: A primer for school psychology researchers and practitioners. Cogent Education, 4(1), 1–13. https://doi.org/10.1080/2331186X.2017.1416898
Boone, W. J., Townsend, J. S., & Staver, J. (2011). Using Rasch theory to guide the practice of survey development and survey data analysis in science education and to inform science reform efforts: An exemplar utilizing STEBI self-efficacy data. Science Education, 95(2), 258–280. https://doi.org/10.1002/sce.20413
Boone, W. J., Yale, M. S., & Staver, J. R. (2014). Rasch analysis in the human sciences. Springer. https://doi.org/10.1007/978-94-007-6857-4
Brookhart, S. M., & Nitko, A. J. (2019). Educational assessment of students (8th ed.). Pearson.
Burhanuddin. (2015). Evaluasi keterlaksanaan penilaian otentik (authentic assessment) pada pembelajaran IPA SMP Negeri di Pasangkayu Kabupaten Mamuju Utara [Universitas Negeri Yogyakarta]. https://eprints.uny.ac.id/25593/
Chan, S. W., Ismail, Z., & Sumintono, B. (2014). A Rasch model analysis on secondary students’ statistical reasoning ability in descriptive statistics. Procedia - Social and Behavioral Sciences, 129, 133–139. https://doi.org/10.1016/j.sbspro.2014.03.658
Chusni, M. M., Saputro, S., Raharjo, S. B., & Suranto, S. (2021). Student’s critical thinking skills through discovery learning model using e-learning on environmental change subject matter. European Journal of Educational Research, 10(3), 1123–1135. https://doi.org/10.12973/eu-jer.10.3.1123
Chusni, M. M., Saputro, S., Suranto, S., & Rahardjo, S. B. (2020). Review of critical thinking skill in indonesia: Preparation of the 21st century learner. Journal of Critical Reviews, 7(09), 1230–1235. https://doi.org/10.31838/jcr.07.09.223
Doleck, T., Bazelais, P., Lemay, D. J., Saxena, A., & Basnet, R. B. (2017). Algorithmic thinking, cooperativity, creativity, critical thinking, and problem solving: exploring the relationship between computational thinking skills and academic performance. Journal of Computers in Education, 4(4), 355–369. https://doi.org/10.1007/s40692-017-0090-9
Fariña, P., González, J., & San MartÃn, E. (2019). The use of an identifiability-based strategy for the interpretation of parameters in the 1PL-G and rasch models. Psychometrika, 84(2), 511–528. https://doi.org/10.1007/s11336-018-09659-w
Fonna, M., & Mursalin, M. (2018). Role of self-efficacy toward students’ achievement in mathematical multiple representation ability (MMRA). Jurnal Ilmiah Peuradeun, 6(1), 31. https://doi.org/10.26811/peuradeun.v6i1.174
Fu, J., & Feng, Y. (2018). A comparison of score aggregation methods for unidimensional tests on different dimensions. ETS Research Report Series, 2018(1), 1–16. https://doi.org/10.1002/ets2.12194
Gardner, A. K., & Dunkin, B. J. (2018). Evaluation of validity evidence for personality, emotional intelligence, and situational judgment tests to identify successful residents. JAMA Surgery, 153(5), 409–415. https://doi.org/10.1001/jamasurg.2017.5013
Gebre, E. (2018). Learning with multiple representations: Infographics as cognitive tools for authentic learning in science literacy. Canadian Journal of Learning and Technology, 44(1), 1–24. https://doi.org/10.21432/cjlt27572
Hambleton, R. K., & Swaminathan, H. (1985). Item response theory. Springer Netherlands. https://doi.org/10.1007/978-94-017-1988-9
Hopfenbeck, T. N., Lenkeit, J., El Masri, Y., Cantrell, K., Ryan, J., & Baird, J.-A. (2018). Lessons learned from PISA: A systematic review of peer-reviewed articles on the programme for international student assessment. Scandinavian Journal of Educational Research, 62(3), 333–353. https://doi.org/10.1080/00313831.2016.1258726
Kartianom, K., & Ndayizeye, O. (2017). What‘s wrong with the Asian and African Students’ mathematics learning achievement? The multilevel PISA 2015 data analysis for Indonesia, Japan, and Algeria. Jurnal Riset Pendidikan Matematika, 4(2), 200–210. https://doi.org/10.21831/jrpm.v4i2.16931
Kristidhika, D. C., Cendana, W., Felix-Otuorimuo, I., & Müller, C. (2020). Contextual teaching and learning to improve conceptual understanding of primary students. Teacher in Educational Research, 2(2), 71–78. https://doi.org/10.33292/ter.v2i2.84
Kvale, S. E. (1989). Issues of validity in qualitative research. Studentlitteratur.
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. http://caepnet.org/~/media/Files/caep/knowledge-center/lawshe-content-validity.pdf
Lester, P. E., Inman, D., & Bishop, L. K. (2014). Handbook of tests and measurement in education and the social sciences. Rowman & Littlefield.
Linacre, J. M. (2012). A user guide to Winsteps Ministep Rasch model computer programs: Program manual 3.75.0.
Maba, W. (2017). Teacher’s perception on the implementation of the assessment process in 2013 curriculum. International Journal of Social Sciences and Humanities, 1(2), 1–9. https://doi.org/10.29332/ijssh.v1n2.26
Menteri Pendidikan dan Kebudayaan Republik Indonesia. (2016). Peraturan Menteri Pendidikan dan Kebudayaan Republik Indonesia Nomor 23 Tahun 2016 tentang standar penilaian pendidikan. Kementerian Pendidikan dan Kebudayaan Republik Indonesia.
Menteri Pendidikan dan Kebudayaan Republik Indonesia. (2019). Peraturan Menteri Pendidikan dan Kebudayaan Republik Indonesia Nomor 43 Tahun 2019 tentang penyelenggaraan ujian yang diselenggarakan satuan pendidikan dan ujian nasional. Kementerian Pendidikan dan Kebudayaan Republik Indonesia.
Moore, T. J., Brophy, S. P., Tank, K. M., Lopez, R. D., Johnston, A. C., Hynes, M. M., & Gajdzik, E. (2020). Multiple representations in computational thinking tasks: A clinical study of second-grade students. Journal of Science Education and Technology, 29(1), 19–34. https://doi.org/10.1007/s10956-020-09812-0
Mujib, N. R., Toenlioe, A. J. ., & Praherdhiono, H. (2018). Analisis butir soal ujian nasional IPA SD/MI tahun 2015 sampai 2017 berdasarkan taksonomi Bloom. Jktp, 1(2), 149–158. http://journal2.um.ac.id/index.php/jktp/article/view/3731
Mutia, N. B., & Prasetyo, Z. K. (2018). The effectiveness of students’ worksheet based on multiple representations to increase creative thinking skills. Journal of Education and Learning (EduLearn), 12(4), 631–637. https://doi.org/10.11591/edulearn.v12i4.8487
Namdar, B., & Shen, J. (2016). Intersection of argumentation and the use of multiple representations in the context of socioscientific issues. International Journal of Science Education, 38(7), 1100–1132. https://doi.org/10.1080/09500693.2016.1183265
Ngin, C. S. (2018). Examining a teacher’s use of multiple representations in the teaching of percentages : A commognitive perspective. Proceedings of the 41st Annual Conference of the Mathematics Education Research Group of Australasia, 591–598.
Nugrahanto, S., & Zuchdi, D. (2019). Indonesia PISA result and impact on the reading learning program in Indonesia. Proceedings of the International Conference on Interdisciplinary Language, Literature and Education (ICILLE 2018), 297(Icille 2018), 373–377. https://doi.org/10.2991/icille-18.2019.77
Paul, R., Elder, L., & Bartell, T. (1997). A brief history of the idea of critical thinking. California Commission on Teacher Credentialing. http://www.criticalthinking.org/pages/a-brief-history-of-the-idea-of-critical-thinking/408
Permatasari, A. K., Istiyono, E., & Kuswanto, H. (2019). Developing assessment instrument to measure physics problem solving skills for mirror topic. International Journal of Educational Research Review, 358–366. https://doi.org/10.24331/ijere.573872
Prahani, B. K., Limatahu, I., W.W, S., Yuanita, L., & Nur, M. (2016). Effectiveness of physics learning material through guided inquiry model to improve student’s problem solving skills based on multiple representation. International Journal of Education and Research, 4(12), 231–242. https://www.ijern.com/journal/2016/December-2016/17.pdf
Rachman, D., Soviyah, S., Fajaruddin, S., & Pratama, R. A. (2020). Reading engagement, achievement and learning experiences through kahoot. LingTera, 7(2). https://doi.org/10.21831/lt.v7i2.38457
Suprapto, N. (2016). What should educational reform in Indonesia look like? - Learning from the PISA science scores of East-Asian countries and Singapore. Asia-Pacific Forum on Science Learning and Teaching, 17(2), 1–22.
Susilaningsih, E., Supartono, S., Kristanto, T., Sariana, E., Azizah, P., & Natasukma, M. (2019). The effectiveness of multiple representation oriented learning material with project based learning to improve students’ chemistry learning outcomes. Proceedings of the 6th International Conference on Educational Research and Innovation (ICERI 2018), 330(Iceri 2018), 87–90. https://doi.org/10.2991/iceri-18.2019.18
Wijayanti, M. D., Rahardjo, S. B., Saputro, S., & Mulyani, S. (2019). Item analysis of critical thinking skills instrument to measure effectiveness of scientific group inquiry learning (SGIL) model. Jurnal Pendidikan IPA Indonesia, 8(4), 538–546. https://doi.org/10.15294/jpii.v8i4.20794
Wright, B. D., & Stone, M. H. (1988). Reliability in Rasch measurement. In Research Memorandum No. 53. MESA.
Yanti, H., Distrik, I. W., & Rosidin, U. (2019). The effectiveness of students’ worksheets based on multi-representation in improving students’ metacognition skills in static electricity. Journal of Physics: Conference Series, 1155(1), 012083. https://doi.org/10.1088/1742-6596/1155/1/012083
Yanto, F. (2019). Development of problem-based student worksheet with authentic assessment to improve student’s physics problem solving ability. Journal of Physics: Conference Series, 1185, 012075. https://doi.org/10.1088/1742-6596/1185/1/012075
Yusrizal, Y. (2016). Analysis of difficulty level of physics national examination’s questions. Jurnal Pendidikan IPA Indonesia, 5(1), 140–149. https://doi.org/10.15294/jpii.v5i1.5803
Amalia, S., & Wuryandani, W. (2020). Socio-cultural based learning module for critical thinking ability in elementary school: systematic search. Acta Educationis Generalis, 10(2), 180–187. https://doi.org/10.2478/atd-2020-0017
Arends, R. I., & Kilcher, A. (2010). Teaching for student learning: Becoming an accomplished teacher. In Teaching for Student Learning: Becoming an Accomplished Teacher. https://doi.org/10.4324/9780203866771
Boone, W. J., & Noltemeyer, A. (2017). Rasch analysis: A primer for school psychology researchers and practitioners. Cogent Education, 4(1), 1–13. https://doi.org/10.1080/2331186X.2017.1416898
Boone, W. J., Townsend, J. S., & Staver, J. (2011). Using Rasch theory to guide the practice of survey development and survey data analysis in science education and to inform science reform efforts: An exemplar utilizing STEBI self-efficacy data. Science Education, 95(2), 258–280. https://doi.org/10.1002/sce.20413
Boone, W. J., Yale, M. S., & Staver, J. R. (2014). Rasch analysis in the human sciences. Springer. https://doi.org/10.1007/978-94-007-6857-4
Brookhart, S. M., & Nitko, A. J. (2019). Educational assessment of students (8th ed.). Pearson.
Burhanuddin. (2015). Evaluasi keterlaksanaan penilaian otentik (authentic assessment) pada pembelajaran IPA SMP Negeri di Pasangkayu Kabupaten Mamuju Utara [Universitas Negeri Yogyakarta]. https://eprints.uny.ac.id/25593/
Chan, S. W., Ismail, Z., & Sumintono, B. (2014). A Rasch model analysis on secondary students’ statistical reasoning ability in descriptive statistics. Procedia - Social and Behavioral Sciences, 129, 133–139. https://doi.org/10.1016/j.sbspro.2014.03.658
Chusni, M. M., Saputro, S., Raharjo, S. B., & Suranto, S. (2021). Student’s critical thinking skills through discovery learning model using e-learning on environmental change subject matter. European Journal of Educational Research, 10(3), 1123–1135. https://doi.org/10.12973/eu-jer.10.3.1123
Chusni, M. M., Saputro, S., Suranto, S., & Rahardjo, S. B. (2020). Review of critical thinking skill in indonesia: Preparation of the 21st century learner. Journal of Critical Reviews, 7(09), 1230–1235. https://doi.org/10.31838/jcr.07.09.223
Doleck, T., Bazelais, P., Lemay, D. J., Saxena, A., & Basnet, R. B. (2017). Algorithmic thinking, cooperativity, creativity, critical thinking, and problem solving: exploring the relationship between computational thinking skills and academic performance. Journal of Computers in Education, 4(4), 355–369. https://doi.org/10.1007/s40692-017-0090-9
Fariña, P., González, J., & San MartÃn, E. (2019). The use of an identifiability-based strategy for the interpretation of parameters in the 1PL-G and rasch models. Psychometrika, 84(2), 511–528. https://doi.org/10.1007/s11336-018-09659-w
Fonna, M., & Mursalin, M. (2018). Role of self-efficacy toward students’ achievement in mathematical multiple representation ability (MMRA). Jurnal Ilmiah Peuradeun, 6(1), 31. https://doi.org/10.26811/peuradeun.v6i1.174
Fu, J., & Feng, Y. (2018). A comparison of score aggregation methods for unidimensional tests on different dimensions. ETS Research Report Series, 2018(1), 1–16. https://doi.org/10.1002/ets2.12194
Gardner, A. K., & Dunkin, B. J. (2018). Evaluation of validity evidence for personality, emotional intelligence, and situational judgment tests to identify successful residents. JAMA Surgery, 153(5), 409–415. https://doi.org/10.1001/jamasurg.2017.5013
Gebre, E. (2018). Learning with multiple representations: Infographics as cognitive tools for authentic learning in science literacy. Canadian Journal of Learning and Technology, 44(1), 1–24. https://doi.org/10.21432/cjlt27572
Hambleton, R. K., & Swaminathan, H. (1985). Item response theory. Springer Netherlands. https://doi.org/10.1007/978-94-017-1988-9
Hopfenbeck, T. N., Lenkeit, J., El Masri, Y., Cantrell, K., Ryan, J., & Baird, J.-A. (2018). Lessons learned from PISA: A systematic review of peer-reviewed articles on the programme for international student assessment. Scandinavian Journal of Educational Research, 62(3), 333–353. https://doi.org/10.1080/00313831.2016.1258726
Kartianom, K., & Ndayizeye, O. (2017). What‘s wrong with the Asian and African Students’ mathematics learning achievement? The multilevel PISA 2015 data analysis for Indonesia, Japan, and Algeria. Jurnal Riset Pendidikan Matematika, 4(2), 200–210. https://doi.org/10.21831/jrpm.v4i2.16931
Kristidhika, D. C., Cendana, W., Felix-Otuorimuo, I., & Müller, C. (2020). Contextual teaching and learning to improve conceptual understanding of primary students. Teacher in Educational Research, 2(2), 71–78. https://doi.org/10.33292/ter.v2i2.84
Kvale, S. E. (1989). Issues of validity in qualitative research. Studentlitteratur.
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. http://caepnet.org/~/media/Files/caep/knowledge-center/lawshe-content-validity.pdf
Lester, P. E., Inman, D., & Bishop, L. K. (2014). Handbook of tests and measurement in education and the social sciences. Rowman & Littlefield.
Linacre, J. M. (2012). A user guide to Winsteps Ministep Rasch model computer programs: Program manual 3.75.0.
Maba, W. (2017). Teacher’s perception on the implementation of the assessment process in 2013 curriculum. International Journal of Social Sciences and Humanities, 1(2), 1–9. https://doi.org/10.29332/ijssh.v1n2.26
Menteri Pendidikan dan Kebudayaan Republik Indonesia. (2016). Peraturan Menteri Pendidikan dan Kebudayaan Republik Indonesia Nomor 23 Tahun 2016 tentang standar penilaian pendidikan. Kementerian Pendidikan dan Kebudayaan Republik Indonesia.
Menteri Pendidikan dan Kebudayaan Republik Indonesia. (2019). Peraturan Menteri Pendidikan dan Kebudayaan Republik Indonesia Nomor 43 Tahun 2019 tentang penyelenggaraan ujian yang diselenggarakan satuan pendidikan dan ujian nasional. Kementerian Pendidikan dan Kebudayaan Republik Indonesia.
Moore, T. J., Brophy, S. P., Tank, K. M., Lopez, R. D., Johnston, A. C., Hynes, M. M., & Gajdzik, E. (2020). Multiple representations in computational thinking tasks: A clinical study of second-grade students. Journal of Science Education and Technology, 29(1), 19–34. https://doi.org/10.1007/s10956-020-09812-0
Mujib, N. R., Toenlioe, A. J. ., & Praherdhiono, H. (2018). Analisis butir soal ujian nasional IPA SD/MI tahun 2015 sampai 2017 berdasarkan taksonomi Bloom. Jktp, 1(2), 149–158. http://journal2.um.ac.id/index.php/jktp/article/view/3731
Mutia, N. B., & Prasetyo, Z. K. (2018). The effectiveness of students’ worksheet based on multiple representations to increase creative thinking skills. Journal of Education and Learning (EduLearn), 12(4), 631–637. https://doi.org/10.11591/edulearn.v12i4.8487
Namdar, B., & Shen, J. (2016). Intersection of argumentation and the use of multiple representations in the context of socioscientific issues. International Journal of Science Education, 38(7), 1100–1132. https://doi.org/10.1080/09500693.2016.1183265
Ngin, C. S. (2018). Examining a teacher’s use of multiple representations in the teaching of percentages : A commognitive perspective. Proceedings of the 41st Annual Conference of the Mathematics Education Research Group of Australasia, 591–598.
Nugrahanto, S., & Zuchdi, D. (2019). Indonesia PISA result and impact on the reading learning program in Indonesia. Proceedings of the International Conference on Interdisciplinary Language, Literature and Education (ICILLE 2018), 297(Icille 2018), 373–377. https://doi.org/10.2991/icille-18.2019.77
Paul, R., Elder, L., & Bartell, T. (1997). A brief history of the idea of critical thinking. California Commission on Teacher Credentialing. http://www.criticalthinking.org/pages/a-brief-history-of-the-idea-of-critical-thinking/408
Permatasari, A. K., Istiyono, E., & Kuswanto, H. (2019). Developing assessment instrument to measure physics problem solving skills for mirror topic. International Journal of Educational Research Review, 358–366. https://doi.org/10.24331/ijere.573872
Prahani, B. K., Limatahu, I., W.W, S., Yuanita, L., & Nur, M. (2016). Effectiveness of physics learning material through guided inquiry model to improve student’s problem solving skills based on multiple representation. International Journal of Education and Research, 4(12), 231–242. https://www.ijern.com/journal/2016/December-2016/17.pdf
Rachman, D., Soviyah, S., Fajaruddin, S., & Pratama, R. A. (2020). Reading engagement, achievement and learning experiences through kahoot. LingTera, 7(2). https://doi.org/10.21831/lt.v7i2.38457
Suprapto, N. (2016). What should educational reform in Indonesia look like? - Learning from the PISA science scores of East-Asian countries and Singapore. Asia-Pacific Forum on Science Learning and Teaching, 17(2), 1–22.
Susilaningsih, E., Supartono, S., Kristanto, T., Sariana, E., Azizah, P., & Natasukma, M. (2019). The effectiveness of multiple representation oriented learning material with project based learning to improve students’ chemistry learning outcomes. Proceedings of the 6th International Conference on Educational Research and Innovation (ICERI 2018), 330(Iceri 2018), 87–90. https://doi.org/10.2991/iceri-18.2019.18
Wijayanti, M. D., Rahardjo, S. B., Saputro, S., & Mulyani, S. (2019). Item analysis of critical thinking skills instrument to measure effectiveness of scientific group inquiry learning (SGIL) model. Jurnal Pendidikan IPA Indonesia, 8(4), 538–546. https://doi.org/10.15294/jpii.v8i4.20794
Wright, B. D., & Stone, M. H. (1988). Reliability in Rasch measurement. In Research Memorandum No. 53. MESA.
Yanti, H., Distrik, I. W., & Rosidin, U. (2019). The effectiveness of students’ worksheets based on multi-representation in improving students’ metacognition skills in static electricity. Journal of Physics: Conference Series, 1155(1), 012083. https://doi.org/10.1088/1742-6596/1155/1/012083
Yanto, F. (2019). Development of problem-based student worksheet with authentic assessment to improve student’s physics problem solving ability. Journal of Physics: Conference Series, 1185, 012075. https://doi.org/10.1088/1742-6596/1185/1/012075
Yusrizal, Y. (2016). Analysis of difficulty level of physics national examination’s questions. Jurnal Pendidikan IPA Indonesia, 5(1), 140–149. https://doi.org/10.15294/jpii.v5i1.5803
Authors
Chusni, M. M., & Suherman, S. (2021). Developing authentic assessment instrument based on multiple representations to measure students’ critical thinking skills. Momentum: Physics Education Journal, 5(2), 194–208. https://doi.org/10.21067/mpej.v5i2.5790
Momentum: Physisc Education Journal allows readers to read, download, copy, distribute, print, search, or link to the full texts of its articles and allow readers to use them for any other lawful purpose. 
This work is licensed under a Creative Commons Attribution 4.0 International License. The Authors submitting a manuscript do so with the understanding that if accepted for publication, copyright of the article shall be assigned to Momentum: Physics Education Journal