Evaluation of cognitive load and scoring procedures of concept map-based tests

Document Type : Original Article

Authors

1 PhD Candidate in Educational Psychology at Āzarbāijān Shahid Madani University

2 (PhD), Āzarbāijān Shahid Madani University

Abstract

Using concept map-based tests as assessment tools is one of the new assessment methods that can be an alternative for traditional assessment methods. So, the goal of the present study was evaluating cognitive load and scoring procedures of these tests. The research sample consisted of students of four biology classrooms from four different high schools (totally 109 students) who were selected via a multi-stage cluster sampling method. The research tools were a multiple-choice test (researcher-made), four concept map-based tests (researcher-made) and the NASA-TLX scale (Hart & Staveland, 1988). The study was done with some type of evaluation design. The statistical methods included Pearson's correlation coefficient, independent samples t-test and one-way ANOVA (Welch test and Games-Howell post hoc). Findings showed that construct-a-map tests imposed significantly more cognitive load on the subjects than fill-in-a-map tests. Also, relational, structural and similarity to expert map scoring procedures had acceptable and significant criterion validity, discriminant validity and reliability. All in all, relational, structural and similarity scoring procedures can be used appropriately for scoring the concept map-based tests. Also, using the fill-in-a-map tests can be an appropriate strategy to overcome the cognitive overload during the concept map-based tests. Thus, the results of the present research can be useful to design tests based on the cognitive rules and to bridge the gap between academic evaluation and cognitive sciences.

Highlights

احدی، فاطمه و سلیمانی، محسن. (1393). مقایسة تأثیر دو روش تدریس به شیوة ارائة مثال آموزشی حل شدة کامل و ارائة مثال به شیوة حل شدة ناقص بر بار شناختی دانشجویان در درس زبان تخصصی پزشکی. آموزش در علوم پزشکی، 14(4)، 291 – 302.

سیف، علی‌اکبر. (1387). اندازه‌گیری، سنجش و ارزشیابی آموزشی (چاپ بیست و هفتم). تهران: انتشارات دوران.

محمدی، محسن.، مظلومی، عادل.، نسل‌سراجی، جبرائیل و زراعتی، حجت. (1392). طراحی پرسش‌نامة ارزشیابی بار کاری ذهنی و تعیین روایی و پایایی آن در پرستاران بخش‌های ویژه یکی از بیمارستان‌های دانشگاه علوم پزشکی تهران. دانشکده بهداشت و انستیتو تحقیقات بهداشتی، 11(2)، 87-96.

مصرآبادی، جواد. (1390). بررسی روایی و پایایی شیوه‌های نمره‌گذاری آزمون‌های مبتنی بر نقشة مفهومی. اندازه‌گیری تربیتی، 2(5)، 1-24.

Amadieu, F., Tricot, A., & Mariné, C. (2010). Interaction between prior knowledge and concept-map structure on hypertext comprehension, coherence of reading orders and disorientation. Interacting with computers, 22(2), 88-97. doi:10.1016/j.intcom.2009.07.001.

Arneson, B. T. (2005). On the role of concept mapping assessment in todayʼs constructivist classroom (Doctoral dissertation, The University of Texas, Austin). Retrieved from https://repositories.lib.utexas.edu/handle/2152/23435

Ausubel, D. P. (1960). The use of advance organizersin learning and retention of meaningful information. Educational Psychology, 51(5), 267-272.

Berry, J. L. (2011). The effects of concept mapping and questioning on students organization and retention of science knowledge while using interactive informational read-alouds (Doctoral Dissertation, Texas A&M University). Retrieved from https://oaktrust.library.tamu.edu/handle/1969.1/ETD-TAMU-2011-08-10164.

Chang, K-E., Sung, Y-T., Chang, R.-B., & Lin, S-C. (2005). A New Assessment for Computer-based Concept Mapping. Educational Technology & Society, 8(3), 138-148.

Chang, K. E., Sung, Y. T., & Chen, S. F. (2001). Learning through computer-based concept mapping with scaffolding aid. Computer Assisted Learning, 17, 21-33.

Cline, B. E., Brewster, C. C., & Fell, R. D. (2010). A rule-based system for automatically evaluating student concept maps. Expert systems with applications, 37(3), 2282-2291.

De Jong, T. (2010). Cognitive load theory, educational research, and instructional design: some food for thought. Instructional science, 38(2), 105-134.

Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In P. Hancock, and N. Meshkati (Eds), Human Mental Workload (Vol. 52, pp. 139-183). North Holland, Amsterdam: Elsevier Science Publishers B.V.

Ho, V. W., Harris, P. G., Kumar, R. K., & Velan, G. M. (2018). Knowledge maps: a tool for online assessment with automated feedback. Medical education online, 23(1), 1457394.

Kapuza, A. (2020). How Concept Maps with and without a List of Concepts Differ: The Case of Statistics. Education Sciences, 10(4), 91. https://doi.org/10.3390/educsci10040091

Kinchin, I. M., Hay, D. B., & Adams, A. (2000). How a qualitative approach to concept map analysis can be used to aid learning by illustrating patterns of conceptual development. Educational research, 42(1), 43-57.

Klepsch, M., & Seufert, T. (2020). Understanding instructional design effects by differentiated measurement of intrinsic, extraneous, and germane cognitive load. Instructional Science, 48(1), 45-77.

Leighton, J. P. & Gierl, M. J. (2007). Cognitive diagnostic assessment for education: Theory and practices. New York, NY: Cambridge University Press.

Lim, Y. K. (2008). The effect of concept mapping with different levels of generativity and learners self-regulated learning skills on knowledge acquisition and representation (Doctoral dissertation, Pennsylvania State University). Retrieved from http://gateway.proquest.com/openurl?url_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:dissertation&res_dat=xri:pqdiss&rft_dat=xri:pqdiss:3431422

Lin, S. S. (2014). Cognitive load for configuration comprehension in computer-supported geometry problem solving: An eye movement perspective. International Journal of Science and Mathematics Education, 12(3), 605-627.

Liu, T. C., Lin, Y. C., & Tsai, C. C. (2009). Identifying senior high school students, misconceptions about statistical correlation, and their possible causes: an exploratory study using concept mapping with interviews. International Journal of Science and Mathematics Education. 7(4), 791-820.

Maker, C. J., & Zimmerman, R. H. (2020). Concept maps as assessments of expertise: Understanding of the complexity and interrelationships of concepts in science. Journal of Advanced Academics, 31(3), 254-297.

McClure, J. R., & Bell, P. E. (1990). Effects of an Environmental Education-Related STS Approach Instruction on Cognitive Structures of Preservice Science Teachers. (ERIC Number: ED341582).

McClure, J. R., Sonak, B., & Suen, H. K. (1999). Concept map assessment of classroom learning: Reliability, validity, and logistical practically. Research in Science Teaching, 36(4), 475–492.

Mesa, J. C. (2010). The development and use of a concept mapping assessment tool with young children on family visits to a live butterfly exhibit (Doctoral dissertation, University of Florida). Retrieved from https://ufdc.ufl.edu/UFE0042067/00001/pdf

Mukhopadhyay, A., Sur, S., Ghosal, A., & Acharya, A. (2018). A New Score Generation Algorithm for Student Concept Map Evaluation. Educational Technology Systems, 47(2), 267-286.

Novak, J. D., & Gowin, B. (1984). Learning how to learn. New York: Cambridge University Press.

Oda, K. (2011). College students GIS spatial concept knowledge assessed by concept maps (Doctoral dissertation. Texas A & M University). Retrieved from http://oaktrust.library.tamu.edu/handle/1969.1/ETD-TAMU-2011-05-9213

Plummer, K. J. (2008). Analysis of the psychometric properties of two different concept-map assessment tasks (Doctoral dissertation, Brigham Young University). Retrieved from https://scholarsarchive.byu.edu/etd/1352

Rubio, S., Díaz, E., Martín, J., & Puente, J. M. (2004). Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA-TLX, and Workload Profile Methods. Applied Psychology: An international review, 53(1), 61-86.

Ruiz-Primo, M. A. (2000). On the use of concept maps as an assessment tool in science: What we have learned so far. REDIE. Revista Electrónica de Investigación Educativa, 2(1), 29-53.

Ruiz-Primo, M. A., & Shavelson, R. J. (1996). Problems and issues in the use of concept maps in science assessment. Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching, 33(6), 569-600.

Rye, J. A., & Rubba, P. A. (2002). Scoring concept maps: An expert map-based scheme weighted for relationships. School Science and Mathematics, 102(1), 33-44.

Saglam, Y. (2006). A phenomenographic case study: concpt maps from the perspectives of midlle school students (Doctoral dissertation, Purdue University, West Lafayette, Indiana). Retrieved from https://docs.lib.purdue.edu/dissertations/AAI3232233/

Schau, C., & Mattern, N. (1997). Use of map techniques in teaching applied statistics courses. The American Statistician, 51(2), 171-175.

Schwendimann, B. A. (2011). Mapping biological ideas: Concept maps as knowledge integration tools for evolution education (Ductoral Dissertation, University of California, Berkeley).Retrieved from https://infoscience.epfl.ch/record/211031/files/Dissertation%20Beat%20Schwendimann.pdf

Schwendimann, B. A. (2015). Concept maps as versatile tools to integrate complex ideas: From kindergarten to higher and professional education. Knowledge Management & E-Learning: An International Journal, 7(1), 73-99.

Skulmowski, A., & Rey, G. D. (2020). Subjective cognitive load surveys lead to divergent results for interactive learning media. Human Behavior and Emerging Technologies, 2(2), 149-157.

Shavelson, R. J., Ruiz-Primo, M. A., & Wiley, E. W. (2005). Windows into the mind. Higher Education, 49(4), 413-430.

Snelson, L. J. (2010). Estimating the Reliability of Concept Map Ratings Using a Scoring Rubric Based on Three Attributes of Propositions (Doctoral dissertation, Brigham Young University). Retrieved from https://scholarsarchive.byu.edu/cgi/viewcontent.cgi?article=3283&context=etd

Stoddart, T., Abrams, R., Gasper, E., & Canaday, D. (2000). Concept maps as assessment in science inquiry learning: A report of methodology. International Journal of Science Education, 22(12), 1221-1246.

Strautmane, M. (2012). concept map-based knowledge assessment tasks and their scoring criteria: an overview. In A. J. Cañas, J. D. Novak and J. Vanhear (Eds), Concept Maps: Theory, Methodology, Technology Proceedings of the Fifth International Conference on Concept Mapping (pp. 80-88). Valletta, Malta: University of Malta.

Thain, D. (2012). A study of semi-hierarchical organization in the construction of concept maps using the framework of cognitive load theory (Doctoral Dissertation, Queen’s University, Kingston, Ontario, Canada). Retrieved from https://qspace.library.queensu.ca/handle/1974/7239.

Van Merriënboer, J. J. (1990). Strategies for programming instruction in high school: Program completion vs. program generation. Journal of educational computing research, 6(3), 265-285.

Vekiri, I. (2002). What is the value of graphical displays in learning? Educational Psychology Review, 14(3), 261-312.

Vodovozov, V & Raud, Z. (2015). Concept Maps for Teaching, Learning, and Assessment in Electronics. Education Research International, Article ID 849678, 9 pages. http://dx.doi.org/10.1155/2015/849678.

West, D. C., Park, J. K., Pomeroy, J. R., & Sandoval, J. (2002). Concept mapping assessment in medical education: a comparison of two scoring systems. Medical education, 36(9), 820-826.

Yin, Y., Vanides, J., Ruiz-Primo, M. A., Ayala, C. C., & Shavelson, R. J. (2005). Comparison of Two Concept-Mapping Techniques: Implications for Scoring, Interpretation, and Use. Research In Science Teaching., 42(2), 166–184.

Keywords