Since it was first published 30 years ago, Chi et al.'s seminal paper on expert and novicecategorization of introductory problems led to a plethora of follow-up studies within andoutside of the area of physics [Chi et al. Cognitive Science 5, 121 - 152 (1981)]. These studiesfrequently encompass "card-sorting" exercises whereby the participants group problems. Thestudy firmly established the paradigm that novices categorize physics problems by "surfacefeatures" (e.g. "incline," "pendulum," "projectile motion," . . . ), while experts use "deepstructure" (e.g. "energy conservation," "Newton 2," . . . ).While this technique certainly allows insights into problem solving approaches, simpledescriptive statistics more often than not fail to find significant differences between expertsand novices. In most experiments, the clean-cut outcome of the original study cannot bereproduced.Given the widespread implications of the original study, the frequent failure to reproduceits findings warrants a closer look. We developed a less subjective statistical analysis methodfor the card sorting outcome and studied how the "successful" outcome of the experimentdepends on the choice of the original card set.Thus, in a first step, we are moving beyond descriptive statistics, and develop a novel microscopicapproach that takes into account the individual identity of the cards and uses graphtheory and models to visualize, analyze, and interpret problem categorization experiments.These graphs are compared macroscopically, using standard graph theoretic statistics, andmicroscopically, using a distance metric that we have developed. This macroscopic sortingbehavior is described using our Cognitive Categorization Model. The microscopic comparisonallows us to visualize our sorters using Principal Components Analysis and compare theexpert sorters to the novice sorters as a group.In the second step, we ask the question: Which properties of problems are most importantin problem sets that discriminate experts from novices in a measurable way? We aredescribing a method to characterize problems along several dimensions, and then study theeectiveness of differently composed problem sets in differentiating experts from novices,using our analysis method.Both components of our study are based on an extensive experiment using a large problemset, which known physics experts and novices categorized according to the original experimentalprotocol. Both the size of the card set and the size of the sorter pool were largerthan in comparable experiments.Based on our analysis method, we nd that most of the variation in sorting outcome isnot due to the sorter being an expert versus a novice, but rather due to an independentcharacteristic that we named "stacker" versus "spreader." The fact that the expert-novicedistinction only accounts for a smaller amount of the variation may partly explain the frequentnull-results when conducting these experiments.In order to study how the outcome depends on the original problem set, our problemset needed to be large so that we could determine how well experts and novices could bediscriminated by considering both small subsets using a Monte Carlo approach and largersubsets using Simulated Annealing. This computationally intense study relied on our objectiveanalysis method, as the large combinatorics did not allow for manual analysis of theoutcomes from the subsets.We found that the number of questions required to accurately classify experts and novicescould be surprisingly small so long as the problem set was carefully crafted to be composedof problems with particular pedagogical and contextual features. In order to discriminateexperts from novices in a categorization task, it is important that the problem sets carefullyconsider three problem properties: The chapters that problems are in (the problems need tobe from a wide spectrum of chapters to allow for the original "deep structure" categorization),the processes required to solve the problems (the problems must required different solvingstrategies), and the diffculty of the problems (the problems must be "easy"). In other words,for the experiment to be "successful," the card set needs to be carefully "rigged" across threeproperty dimensions.
Read
