A COMPARISON OF SEVERAL METHODS OF SCORING THE .KUDER OCCUPATIONAL INTEREST SURVEY Thesis for the Degree of Ph. D. MICHIGAN STATE UNIVERSITY ‘ WILLIAM EARL LOADMAN‘II 1971 yaw!" \I This is to certify that the I thesis entitled A COMPARISON OF SEVERAL METHODS OF SCORING THE KUDER OCCUPATIONAL INTEREST SURVEY presented by William Earl Loadman II has been accepted towards fulfillment of the requirements for Ph.D. degree in Counseling, Personnel Services, and Educational Psychology F7 Major professor v "_-_l)ate l") “Lg/7] 0-7 639 ABSTRACT A COMPARISON OF SEVERAL METHODS OF SCORING THE KUDER OCCUPATIONAL INTEREST SURVEY BY William Earl Loadman II The ability of an occupational interest survey to accurately predict occupational classification is related to the method of scoring the instrument. The present re- search compared four different methods of scoring the Kuder Occupational Interest Survey (Form DD). The purpose was to discover more accurate methods of scoring the Kuder for use in predictive situations. Three scoring procedures used the response pattern of each item. These were: (a) pattern analysis, (b) Chi—square, and (c) Kuder lambda.. A fourth method of scoring, discriminant function, was based upon one of the first three, viz. the Kuder lambda. Method The basic data for the present study were the same response scores of subjects from nine occupations origi- nally used by Kuder to empirically derive nine lambda scoring keys. In this research, these occupations were divided into two groups of five similar and five dissimilar William Earl Loadman II occupations. This split provided the researcher with the opportunity to investigate the potential of each scoring method in classifying respondents from (a) similar occu- pations, and (b) dissimilar occupations. The data from both the similar and dissimilar occupational groups were randomly split in half to allow for cross-validation pro— cedures. Using a dichotomous (hit or miss) classification score for each subject of the cross-validation sample as the dependent variable, a comparison among the accuracies of the scoring methods was made for each set of data. This comparison was accomplished through the use of a re— peated measures analysis of variance model. The five occupations (between subjects) and the four methods of scoring (within subjects) variables were defined as fixed factors in the analysis, while subjects nested within groups was treated as a random factor. Conservative tests of the sources of variation were employed by applying the Greenhouse and Geisser reduction in degrees of freedom. A significant groups by methods disordinal interaction was found in both sets of data. Secondary analyses were employed to determine the significant simple effects of the groups and methods variables. All significant simple effects were subjected to Scheffé post hoc procedures to determine the significant pair-wise contrasts among means. William Earl Loadman II Results It was concluded that none of the proposed scoring methods functioned best for all situations. Rather, the post hoc procedures suggested that the scoring procedure which was best in classifying persons within a particular group was dependent upon the set of groups under consider- ation, the persons within the groups, and in certain cases, the number of persons within the group. The discriminant function, Chi-square and Kuder lambda methods of scoring all performed significantly better than the pattern analy- tic method. The pattern analytic method classified subs jects at about the chance probability level while the remaining three methods each functioned well above the chance level. Of these three methods, no method was con— sistently superior to the other two methods. For all scoring methods, the classification of subjects was more accurate in the set of dissimilar than in the similar set of occupations. The Kuder lambda scoring keys (keys now used com- mercially) were used in this research and the results of the classification based on these keys were compared with those of the three proposed methods of scoring the Kuder. Unfortunately, a true cross-validation sample was not available; this rendered the Kuder lambda keys slightly more predictive of occupational classification than otherwise would be true. The Kuder lambda method was most accurate in classifying five of the ten groups. William Earl Loadman II The discriminant function method of scoring the Kuder was based on the classification score from each Kuder lambda scoring key. Since there was no true cross- validation for the Kuder lambda technique, it follows that there was no true cross-validation for the discrimi— nant function method. Quasi cross-validation was used and resulted in the most accurate classification by the dis- criminant function method for two of the ten groups. On the set of similar occupations, the discriminant function method yielded excellent predictions for four of the five groups. For some unknown reason, the correct classifi— cation of persons in the fifth group broke down. When considering all of the persons within the set of dis— similar occupations, the discriminant function method accurately classified 82% of the persons on the quasi cross-validation. A Chi-square method of scoring the Kuder yielded results comparable to those of the Kuder lambda and dis- criminant function methods for the set of similar occu— pations. The Chi-square method was slightly less pre- dictive than the Kuder lambda and discriminant function methods on the dissimilar occupational data. Of the four proposed methods of scoring, the Chi-square method was most accurate in classifying persons in three of ten groups. The pattern analytic scoring method did not function well. This method was found to be significantly William Earl Loadman II worse than the other three methods across all conditions investigated in this research. A proposed alteration in the procedure for classify- ing subjects using the pattern analytic method was com- pared with the original pattern analytic method. Neither method exceeded chance probability in correctly classify- ing individuals on the cross-validation. The present research was not an adequate test for this comparison. Conclusions The results of this research indicated that there was no single best method to be used for scoring the Kuder when either a select set of similar or dissimilar occu— pations was considered. A significant groups by methods disordinal interaction suggested that the best scoring method for the Kuder was dependent upon the groups under consideration, the computer resources available, and the required degree of accuracy. Because true cross-vali- dation was not used in all aspects of the study, more explicit statements of conclusions are not warranted. A COMPARISON OF SEVERAL METHODS OF SCORING THE KUDER OCCUPATIONAL INTEREST SURVEY BY William Earl Loadman II A THESIS Submitted to Michigan State University in partial fulfillment of the requirement for the degree of DOCTOR OF PHILOSOPHY Department of Counseling Personnel Services and Educational Psychology 1971 ACKNOWLEDGMENTS A dissertation is not developed and written in a vacuum. There are many persons who have been instrumental in the initial develOpment and subsequent presentation of this thesis. First I wish to thank the members of my com— mittee. Dr. Terrence Allen and Dr. William Mehrens pro— vided encouragement and helpful comments. Dr. Andrew C. Porter generously shared not only his data, but his time and ideas in offering commentary and constructive criti- cism. He was most helpful in guiding me through several rocky stages in the development of this research.. Most of all, I am grateful to the chairman, Dr. Robert C. Craig. His patience, guidance, and encouragement in addition to his valuable suggestions were appreciated in the formative stages and his skillful instruction and insight were highly valued in improving the presentation. I would also like to extend my appreciation to Science Research Associates. At this point, I wish to say a special thank you to a very good friend, Dr. Donald J. Freeman. It was his initial encouragement that led me to pursue doctoral study. I wish I could find the words to adequately express my gratitude for the positive influence his continued ii friendship, interest, and encouragement have been for me. Finally, I must express my sincere thanks to my wife, Gail. She has doggedly persisted in keeping me on target. I will never be able to repay her many sacrifices in this dissertation effort. It is to her, to our son, William III, and to my parents, who have always been close by with their own special encouragement, that this disser- tation is affectionately dedicated. iii Chapter I. II. III. IV. TABLE OF CONTENTS THE PROBLEM OF SCORING AN OCCUPATIONAL INTEREST SURVEY . . . . . . . . Introduction . . . . . . Description of the Kuder Instrument. Kuder Weighting Technique . . . . REVIEW OF THE LITERATURE . . . . . General History . . . . . Pattern Analytic Technique. . Discriminant Function Technique Chi—Square Weighting Technique Kuder Related Research . . . Summary . . . . . . . DATA AND METHOD . . . . . . . . Statement of Problem. . . . Overview of Procedure . . . Occupational Data on the Kuder Kuder Scoring System. . . . Pattern Analytic Scoring System . . Discriminant Function Scoring System Chi-Square Scoring System . . . . Cross-Validation Technique. . . . Classification Comparisons Among Various Systems. . . . . . . Summary . . . . . . . . . . RESULTS AND DISCUSSION. . . . . . General Overview . . . Proportions of Subjects Correctly Classified . . . . . Repeated Measures Analyses. Testing for Simple Effects. Scheffé Post Hoc Analyses . iv Page \lUlH H 13 13 14 19 26 28 30 32 32 32 33 34 35 43 46 47 50 51 52 52 53 61 69 Chapter Comparisons of Methods Means . . . . Relationships Among Scoring Methods. . Kuder Lambda Scoring Results . . . . Pattern Analytic Scoring Results. . . Comparison of the Original and Modified Pattern Analytic Scoring Procedures Discriminant Function Scoring Results . Chi-Square Scoring Results. . . . Comparison of Methods in Quasi Cross— Validation . . . . . . . . . V. SUMMARY AND CONCLUSIONS . . . . . . BIBLIOGRAPHY . . . . . . . . . . . . APPENDICES Appendix A. Discriminant Function Analyses of Set I and Set II Data . . . . . . . . . B. A Presentation of the Non Cross-Validated Classification Prediction for the Chi- Square, Kuder Lambda and Discriminant Function Techniques. . . . . . . . C. Variance Covariance Matrices of the Dichotomous Cross-Validation Scores for Set I and Set II Data . . . . . . . Page 76 79 81 81 86 87 92 93 96 103 112 114 115 Table 1. 10. LIST OF TABLES The Proportions of Males Correctly Classified on the Cross-Validation Sample for Set I Data . . . . . . The Proportions of Males Correctly Classified on the Cross-Validation Sample for Set II Data. . . . . . Repeated Measures Analyses of Variance on the Dichotomous Classification Data for Set I and Set II. . . . . . . . Analysis of Variance: Tests of Simple Effects on the Classification Scores of Set I Data Using Pooled Estimates of the Variance . . . . . . . . . Analysis of Variance: Tests of Simple Effects on the Classification Scores of Set II Data Using Pooled Estimates of the Variance . . . . . . . . . Scheffé Post Hoc Analyses of Set I Groups Data at Each Level of the Methods Variable . . . . . . . . . . Scheffé Post Hoc Analyses of Set II Groups Data at Each Level of the Methods Variable . . . . . . . . . . Scheffé Post Hoc Analyses of Set I and Set II Data at Each Level of the Groups Variable . . . . . . . . The O Coefficient: The Relationships Among the Four Repeated Measures . . Comparison of the Proposed and Original Pattern Analytic Scoring Procedures for Set I and Set II Data . . . . . . vi Page 56 57 59 67 68 70 71 72 80 82 LIST OF FIGURES Figure Page 1. A Flow Chart for Performing Criterion Pattern Analysis . . . . . . . . . 37 2. An Illustration of the Output from the Prediction Program Associated with Com— puter Pattern Analysis . . . . . . . 41 3. The Graphic Relationship Among Groups, Measures, and the Proportion of Correctly Classified Individuals for Set I Data; Groups Variable Placed on the Abscissa. . 62 4. The Graphic Relationship Among Groups, Measures, and the Proportion of Correctly Classified Individuals for Set I Data; Methods Variable Placed on the Abscissa . 63 S. The Graphic Relationship Among Groups, Measures, and the Proportion of Correctly Classified Individuals for Set II Data; Groups Variable Placed on the Abscissa. . 64 6. The Graphic Relationship Among Groups, Measures, and the Proportion of Correctly Classified Individuals for Set II Data: Methods Variable Placed on the Abscissa . 65 vii CHAPTER I THE PROBLEM OF SCORING AN OCCUPATIONAL INTEREST SURVEY Introduction It has been repeatedly demonstrated that an indi- vidual possesses certain interests which are related to occupational preference. The measurement of these inter- ests has stimulated psychologists for several decades; and, in spite of the considerable time and effort expended in attempting to determine reliable and valid measurement (whether stated, observed, or covertly detected by unob- trusive measures), there still exists marked imprecision in the methodological procedures. During the past twenty to thirty years, one par— ticular methodological technique has been consistently used; it attempts to quantify the measurement of reported interests and subsequently employ these scores in a pre— diction model. The quantification of interests through a self-report to a series of items, is called an interest survey. The survey, usually referred to as an instrument, requires the subject to respond to a carefully generated list of items. His responses are compared with responses of one or more reference groups and/or with the responses of a group of men in general. The reference or criterion group generally consists of individuals employed in specific occupations. For example, one reference group may be forest rangers and the responses to the survey items of persons employed as forest rangers serve as the pattern for identifying persons with like interests. Thus, occupations or areas of occupation can be identi- fied and an individual can be "classified" as having re- ported interests similar to those in the reference group and different from men in the other reference groups or the men in general group. If this is accomplished, dis- crimination between occupational interests is said to have occurred. In vocational counseling situations, the derived results of the interest survey are used in conjunction with additional pertinent data (i.e., subject's abilities and current functioning levels) to identify potential occupational courses of action and possible alternatives which are consistent with and palatable to the individual. The ultimate decision regarding any of the suggested actions obviously rests with the individual. The desired end result of Conducting the survey is to provide the individual and/or employer with pre- dictive information regarding prospective occupational satisfaction and job-individual compatability. Clearly, it is imperative that the predictions be as accurate and precise as possible. Kuder (1961, p. 3) stated: More recently, studies of certain jobs have revealed that a worker's preferences among well- known activities are related to his job satis- faction and to the length of time he is likely to stay on a specific job. There is also some evidence that interest is related to achievement. According to Berdie and Campbell (Whitla, 1968) interest may be a better indicator of satisfaction in a given occu- pational pursuit than ability. In View of the evidence suggesting that interest plays a primary role in occu— pational satisfaction, the present study considered only the problem of measuring interests. The objective of the present study was to build a predictive model based solely on interest. There were several limiting factors which affected the construction and subsequent use of this model. The selection of the sample and the chosen occupations influenced the con— struction. Actual use of the model is affected by the fact that the interest measure is not the sole criterion upon which a vocational decision is based. A subject may exhibit dramatic interest in a particular job or job field, but be lacking a vital physical or mental capacity to allow successful performance. Consider the occupation of an airline pilot; a man may have significant and com- patible interests to pursue this vocation, but have faulty depth perception. Obviously if the occupational prediction were derived solely from the interest criteria, a disastrous situation could result. An additional and rather serious problem associ- ated with use of a survey is the absence of assurance that the measured interests and the actual interests are the same. The ramifications of this problem are twofold. First, a given subject may report interests of a given nature which may not be reflected in his actual behavior. Thus, the self-report instrument may not yield a valid determination of a subject's interests. Second, there is no assurance that subjects within a particular cri- terion group espousing particular kinds of interests are in fact satisfied with their present occupation. However, Kuder (1965) provided validation evidence which supported the basic assumption that persons within a given occu— pation are satisfied and that this satisfaction is posi- tively related to interests. This study focused on one of the most widely used instruments, the Kuder Occupational Interest Survey, Form DD (1966), which will hereafter be referred to as the Kuder. The Kuder is employed by many counselors and is relied upon in many vocational counseling settings. Therefore, there is an express need for the information supplied by the Kuder to be as accurate as possible when used in predictive situations. To accomplish this, the survey must be able to identify the subject's interests and further, these interests must be applicable in accurately classifying the subject with respect to selected criterion groups. The purpose of this research was to determine more accurate scoring procedures for the Kuder employing selected criterion groups. As pointed out by Gaier and Lee (1953), the discriminatory power (i.e., the ability of the instrument to accurately differentiate between persons in two or more groups) of a predictive instrument involves the method of scoring as well as the instrument itself. Taking the Kuder instrument as a given, this study con- sidered alternate methods of scoring the Kuder. The pro- posed methods were compared with the presently employed Kuder system and with a system employed by another author. Description of the Kuder Instrument The primary purpose of the Kuder is to classify males into vocational occupations using their interests (as determined by their responses to the items on the survey) as predictors of a specific occupation. The instrument consists of 100 triadic items. Each item is comprised of three sentences or phrases and each of these deals with a common activity which is generally under— stood and does not directly reflect any particular occu- pation. The vocabulary is pitched at the sixth grade. level, and Kuder claims that the tool is applicable to persons at all levels of educational and vocational saphistication. Responding to each item, the subject indi- cates the activity which he most prefers and the activity which he least prefers. The directions are easily under- stood and there is no set time limit for administration. Kuder reports that this time typically ranges from 20 to 30 minutes. The instrument can be applied to an individual or a group with equal ease; the appropriate target pOpu- lation is males from ninth grade through adult. The scoring keys developed by Kuder are empirically based upon the occupation under consideration. There are approxi- mately 77 scoring keys presently available (Kuder, 1967). Further elaboration on the Kuder is unnecessary and not pertinent to the development of the procedures investi- gated. Science Research Associates has the c0pyright on the original instrument and additional information can be obtained from that source. Since Kuder does not provide a "correct" answer for each item, the problem becomes one of weighting each response and obtaining a composite score that will result in a discrimination of subjects along a continuum of interest. Porter (1966, p. 3), stated: A solution to the weighting is dependent upon the groups which are to be discriminated among, the equipment available, and the desired limits as to the complexity of the weighting procedure. Kuder (1957) asserted that there are many potential possibilities for scoring and predicting group membership, and theoretically the very best method would consider all possible combinations of responses, "an astronomical number." Dismissing this alternative as impractical, Kuder proceeded to generate scoring keys based upon the pair~wise comparisons of responses of all occupational groups across each of the 100 items (Kuder, 1966). The keys were derived by determining a lambda coefficient for each response alternative for each occupational group. The occupational category yielding the largest sum of lambda coefficients was considered the classification category for a given subject. This procedure will be discussed in detail in the following section. Kuder Weighting Technique When Kuder was originally devising his instrument he encountered a major hurdle in the development of scoring keys. Kuder (1957) devoted considerable time to this problem of scoring and analysis and concluded that it was not possible to specify the one best scoring system; the procedures were dependent upon a host of factors including the number of cases, the composition of the survey, the content and type of item, the range of item validities, the homogenity of the group used, and the distribution of item characteristics. It would appear that the best possible scoring system must be devised for the specific situation. This alternative was thought to be entirely too limiting and Kuder, on the basis of cer— tain empirical evidence, proceeded in another direction. Reasoning that the research on item weighting was not definitive, a system of weighting was selected which yielded acceptable rather than optimal results; it was computationally simple and easy to interpret. For these reasons a unit weighting system was endorsed for the scale. The unit scoring keys for each occupation were developed and used to differentiate persons in the given occupation from persons in the norm group. The norm group consisted of 1000 men obtained in a stratified random telephone survey of 138 cities and towns which were assumed to represent sections and seg— ments of the United States population (Kuder, 1961). Kuder stated that this group was not the most representa- tive sample that could be obtained, but that it was adequate for his purposes. The Specific characteristics of this norm group had a pervasive and unknown effect upon all the scoring keys constructed by Kuder. The building of a key via the Kuder method* was straight- forward and although no specific sample size was insisted upon, a "rule of thumb" was 200 subjects. However, Kuder was quite insistent regarding adequate cross-validation samples to determine the proficiency of the key. With additional research, improved scoring methods were proposed. *Since the Kuder unit weighting system was not used in this study, discussion of the derivation of the original scoring procedure is limited. Derivations of the scoring methods used in this study are presented in the methodology section of the thesis. Kuder subsequently revised the original procedure and began using a fractional (non-unit) weighting tech- nique. This new system resulted in a more accurate method of prediction (Kuder, 1966), and the Kuder Occupational Interest Survey (Form DD) was originated. The items and the format of the instrument were identical to those em- ployed in Form D; only the method of scoring was altered. The new scoring system was based upon the work of Findley (1956) and Clemans (1958), and was developed in the following manner. Each item (triad) consists of three statements; a given subject is required to choose one of the statements as the most appropriate for himself and one of the remaining statements as the least appro- priate for himself. For a given criterion group, the proportion of subjects that selected each one of the three statements as most liked and the pr0portion that chose each one of the three statements as least liked were calculated. Six prOportions were determined for each item for each criterion group. The following example is help— ful in understanding the procedure: Item XX Most Least Statement 1 P(l) X P(4) X Statement 2 P(2) X P(5) X X P X Statement 3 P (6) (3) 10 P(l) represents the proportion of subjects in group j selecting statement 1 as the most liked alternative; P(2) represents the proportion of subjects in group j selecting. statement 2 as the most liked, etc.; and P(4) represents the proportion of subjects in group j selecting statement 1 as the least liked alternative, etc. This procedure was continued until the proportions for each of the 100 items were calculated (a total of 600 proportions for each group). Any given subject within the criterion group would necessarily have chosen a subset of 200 of the 600 possible response alternatives. A single score for each item for each subject was determined by combining the pro- portions from the two selected responses of the subject in. the following fashion, yielding one of six posSible re— sponse pattern scores for a given subject: 1 = P(1) + P(5) 2 = Pm + Pm 3=P(2) +P(4) 4=P(2) +P(6) 5=P(3) +P(4) 6=P(3) +P(5) The sum of all proportions for a given triad was 4.00 as the sum of P(l)' P ), and P(3) was equal to 1.00 (2 and the sum of P(4), P(5), and P(6) was equal to 1.00, and each proportion appears twice in the above diagram. For each subject, the sum of the proportions of the 100 11 endorsed response patterns served as the continuous vari- able indicating the subject's similarity to the rest of the group. A dichotomous variable was defined by the subject's selection or non-selection of the particular response pattern alternative. By generating the corre- lation coefficient between the above continuous and dichotomous variables across subjects in an occupation, it was possible to generate 600 correlations; one corre— lation for each of the six possible alternatives to each of the 100 items. These 600 correlations represent the relationship between the individual's interests and the interests of the group. Because of differences in the homogeneity of the criterion groups, the range of possible scores varied considerably from group to group. Therefore, it was difficult to make interpretations when comparing the potential classification of an individual based upon the sum of 100 point biserial coefficients. A lambda coef— ficient was introduced to eliminate the problem of vary- ing ranges of the correlation coefficient; also, the statistic did not make any assumptions concerning the shapes of the distributions of the two variables. In essence, the lambda coefficient was the ratio of the obtained point biserial to the maximum possible point biserial for the data under consideration. The upper limit of this statistic was 1.00 regardless of the 12 homogeneity of the group. Lambda coefficients were used as weights for the Kuder. Thus, there were 600 potential weights for each criterion category and a subset of 100 of these weights was used (according to the response pattern of the subject) to generate the subject's score. A subject's classification score was determined by summing the lambda coefficients for the 100 endorsed response patterns for a particular criterion category and this procedure was repeated for each criterion category under consideration. A subject was classified into the cri— terion category which yielded the largest sum of lambda coefficients. Preliminary research comparing the two Kuder scoring techniques (unit and lambda) indicated signifi- cantly less overlap in classification (greater accuracy) with the lambda technique (Kuder, 1970). CHAPTER II REVIEW OF THE LITERATURE General History A review of the literature revealed considerable evidence of efforts to develop weighting systems and scoring keys for interest surveys. Cowdery (1925) basing his work on an earlier study, determined a weighting system for each item when there were two criterion groups and the responses were dichotomous. The weight for each item was wt = ¢ 0, where ¢ was the Phi coefficient (1 - <12) and 0 was the standard deviation of the (a+b) and (b+d) cells of a two by two matrix. The matrix consisted of groups as one dimension and responses as the other dimen- sion. Response 1 Response 2 Group 1 a b Group 2 c d Kelly (1923) demonstrated that the Cowdery pro- cedure was a good estimate of a least squares regression weight (for each item). This procedure was used as the basis for scoring the Strong Vocational Interest Blank 13 14 (Stanley, 1964). However, the procedure was limited to dichotomous criteria and dichotomous responses. Tilton (1937), reported a method of weighting which compared the difference in the proportion of subjects selecting par- ticular response alternatives between two criterion groups. This empirical weighting system seemed to function quite satisfactorily, but it was limited to two criterion groups at any one time. Kuder (1963) cited both the advantages and disadvantages of using Tilton's proposed model and provided empirical evidence related to its effectiveness. The results indicated a high degree of accuracy, but marked limitatations in application (dichotomous items and groups). Berdie and Campbell (Whitla, 1968) asserted that there has been a prolonged disagreement with respect to the "best" method of weighting survey items; consider— ation had been given to unit weighting, fractional weight— ing, and pattern weighting. At this point in time it is not evident that any of these alternatives is superior to the others. Pattern Analytic Technique A large number of studies have employed pattern weighting techniques (rather than item weighting) in an attempt to improve prediction. The following studies represent the major endeavors to improve prediction in a potentially fruitful fashion. 15 Brigham (1932) considered item patterns or con- figural scoring as a possible method of response weight- ing. A response pattern was deve10ped for each individual and a subsequent comparison was made of the individual's responses with responses from specific criterion groups. Zubin (1937) generated a similar technique in conducting a study of likemindedness; and, concluded that although his results were not striking, there appeared to be great promise in the technique. Meehl (1950, 1954) probably made the most significant contributions to the advancement of configural scoring patterns. He strongly supported the contention that one item may not be predictive, but a pattern of two or more items, including the item that was non-predictive when used alone, may be highly pre- dictive. If patterns of items were not taken into account, he argued, important psychological information may be lost. Others who made contributions to configural scoring techniques were: Cattell (1949), Cronbach (1949), Cron- bach and Gleser (1952), and Dumas (1946). Although the results were not conclusive, each of the above mentioned studies indicated the potential value of pattern weight- ing in addition to item weighting. However, these re- searchers have not empirically and successfully shown the productivity of the method. More recently, other warriors have undertaken the crusade of configural scoring and have revealed either new techniques or empirical evidence regarding the validity of old techniques. 16 Ghiselli (1960) verified that items or "tests" as he termed them, can be employed to improve the accuracy of prediction of criterion classification when used in combinations even though no relationship was evident be- tween the individual "test" and the criterion measure. He demonstrated this with a linear and non-linear model. Lee (1961) approached the problem of configural scoring in a somewhat different manner, but clearly demonstrated that the additive model can be improved upon by taking into account certain "interactions" within the data. These "interactions" were in fact pattern configurations. She continued by defining non-additive models and closed with the statement: It is at least theoretically possible that these relations, or interaction effects, might in some cases be so extreme that a set of independent vari— ables, each of which independently correlates zero with a given dependent variable, could perfectly predict the dependent variable if considered jointly by appropriate methods (Lee, 1961, p. 804). Horst (1954) mathematically enhanced the claims of Meehl and presented evidence that the configural scor— ing methods represented a special case of non—linear item weighting. He concluded that configural scoring should be more fruitful with the introduction of non-linear mathe— matical models. Fricke (1956) found that the discrimi- natory power of configural-content intensity items was greater than that of the commonly employed linear methods. 17 In the late fifties and early sixties, MCQuitty (1957, 1958, 1959, 1961a, 1961b, 1963, 1966) produced a significant number of studies and introduced several new pattern analytic, or as he termed them, typal techniques. McQuitty's most prominent contribution was probably in identifying areas of concern and problems associated with patterning techniques. McQuitty developed elementary linkage analysis. Initially this technique appeared to be quite promising, but the related problems quickly negated utilitarian ideas. The foremost drawback was in the method of data reduction. A subject's by items matrix must be mathematically reduced in an attempt to force parsimony on the data. The results were mathematically consistent, but the resulting output presented a distorted picture of psychological reality. Other problems included limitations on the size of the data matrix and the number of criterion groups which can be simultaneously analyzed. The work on pattern analytic techniques has been continued with a criterion pattern analysis computer pro— gram generated by Clark (1968). Through the utilization of data processing equipment, it became feasible to con— sider a large number of items and a large number of subjects simultaneously. This program allowed for the analysis of nominal level data when both the criterion and predictor variables were dichotomous. A full description of Clark's method of scoring and the 18 functioning of the pattern analysis program will be pre- sented in the methodology section of this paper. Two sets of data were analyzed in Clark's study and they consisted of: (a) the responses of 99 women to an introversion-extraversion scale (independent variable, dichotomous items) and the hidden figure test of field dependence-independence (criterion variable, dichotomous grouping); and (b) the United Nations roll call voting record of 110 nations on 44 issues (one vote was con- sidered the criterion variable and the remaining 43 votes were considered the independent variables). Clark compared the results of the pattern analysis program to multiple regression and maximum likelihood estimates on original and cross-validation samples. The pattern analytic, regression and maximum likelihood esti- mates all predicted well using the original data, but upon cross-validation, the pattern analytic technique proved to be better than the maximum likelihood technique. The most successful technique in terms of cross-validation was the pattern analytic technique in combination with a linear prediction model. Distinct advantages of Clark's pattern analytic approach over previous configural scoring schemes were: (1) the program could handle large quantities of data; (2) it could consider all combinations of response alter- natives; (3) patterns were generated separately for each criterion group; (4) the program could identify configural 19 (several items in combination) as well as noneconfigural (one item) relationships; (5) the patterns could be used directly in prediction; and (6) the patterns were easily interpretable. His study demonstrated the feasibility and credibility of employing a pattern analytic technique in the analysis of multiple measure data. Certain inadequacies were also identified in Clark's prediction program; classification of an individual was based upon the highest discrimination index. The length of patterns, number of patterns, and the number of persons having the patterns were not considered in the prediction program. The present study proposed to explore the gener— alized use of the Clark technique to include items with more than dichotomous responses and more than two cri- terion groups. The potential of patterning techniques has consistently been identified but the accompanying methodological procedures have been lacking to some degree. This study planned to make contributions in this area. Discriminant Function Technique Another possible way of weighting item responses can be accomplished through the use of the linear dis- criminant function analysis. The early work in this area reverts back to Sir Ronald Fisher (1937) who initiated the analysis of multiple measures on groups of individuals. ables. among 9 their m mate st signifi Variabl. used, t; p' 117): dete Of\ Off Squa Sum Tatsmka hiStOry fifteen thmst o anal‘Jsis Fisher. ter10n 91 20 Rao (1948) worked on the classification of persons identi- fied as representatives of particular groups. This led him to the employment of the discriminant function analysis with a dichotomous criterion and several independent vari- ables. He concluded that it was possible to discriminate among groups on the basis of the likelihood estimates of their multiple variable response distributions. Approxi- mate statistical tests were presented to examine the significance of group discrimination. Welch (1939) demonstrated that when the criterion variable was dichotomous and the discriminant function was used, the results were equivalent to the multiple regres— sion analysis of these same data. According to Bock and Haggard (Whitla, 1968, p. 117): [The discriminant function] . . . procedures determine the coefficients of the linear combination of variables which best discriminate between groups of subjects, in the sense that between groups sum of squares is a maximum with respect to the within group sum of squares. The work of Rao was extended by Rulon, Tiedeman, Tatsuoka,and Langmuir (1967). Their book presented a history of personnel classification based on approximately fifteen years of research by the authors. The major thrust of the text concerned the discriminant function analysis and extension of the technique as proposed by Fisher. The method could accommodate more than two cri— terion groups and its purpose was to generate a G space 21 such that individuals could be maximally classified on a specific dimension of the G space. The G classification was accomplished through the assignment of a weight to each item which would maximally separate the criterion groups. These weights could be determined through a multivariate analysis. The procedure made it possible to generate the multivariate analysis of variance and then determine the eigenvector associated with the largest significant root of the determinental equation IW-lA-AII=O where I is an identity matrix and W is the pooled within groups deviation matrix; A represents the between groups deviation matrix. This vector yielded the apprOpriate weights for the reSpective items. Thus an item weighting, linear prediction model could be employed when using the technique. For purposes of individual classification, multivariate response contours were generated in con- junction with the significant latent roots of the function. The individual's resultant score was then obtained to determine in which contours his score fell (a subject received a score for each significant latent root). With less overlapping of the contours, there was greater dis— crimination and predictability of the function. If there was great overlapping of the contours, the classification of the subjects was extremely difficult and the predicta- bility was markedly reduced. According to Anderson (1958) and Morrison (1967), if there were G criterion groups, (1967) non-en functi the ba Prefil. engiHEI dents, in the CroSS~x and mul criter: engiDeE and COL Vic qu dis lar PGr Par 22 then it was possible to have G-l significant latent roots associated with the discriminant function if the number of subjects exceeded the number of variables (items), and the number of variables was greater than the number of groups. Three recent dissertation studies were completed at Purdue University utilizing the discriminant function analysis of the Strong Vocational Interest Blank. CoBabe (1967) using two criterion groups, student engineers and non-engineers, demonstrated that the multiple discriminant function could be reliably used to separate individuals on the basis of the Strong Vocational Interest Blank (SVIB) profile. The sample was limited to a group of freshmen engineering students and a group of non-engineering stu- dents, all at Purdue University. There were 476 students in the sample. The results were verified in subsequent cross—validation procedures. Chappell (1967) conducted a study using the SVIB and multiple discriminant function analysis. He used as criterion groups, 92 veterinary students, 32 graduate engineering students, and 76 graduate students in guidance and counseling. He asserted: The results and conclusions of this study pro— vided suggestive evidence that multiple discriminant function analysis is a powerful statistical tool for discriminating among groups, which, heretofore, have largely resisted sharp discrimination on self report personality measures. Additional research into com- parisons of other occupational groups is necessary as *1" 1‘ An author parame‘ Present a high the thr apparen Under tl analeis He attem engineer included the firSt 31 Years OlogiCal 1 BECauSe O 23 well as comparisons within occupational domains which have several sub-divisions. The use of multiple dis- criminant function analysis is hereby recommended (Chappell, 1967, p. 70). Chappell's study used a relatively small number of cases within each criterion group. With groups that are this small it is difficult to obtain stable estimates of population parameters. Since the three original samples were randomly divided for cross-validation procedures, the number of measures almost equalled the number of subjects used for determining the initial equations. Thus the author assured himself of a relatively good fit of the parameters on the original subjects. However, Chappell presented cross-validation data and the results indicated a high degree of success in correct classification among the three groups. Thus, the estimated parameters were apparently generalizable because the results held up under the cross-validation procedure. A third study of multiple discriminant function analysis using the SVIB was conducted by Clemens (1969). He attempted to discriminate between 229 practicing engineers and 210 second-year engineering students. He included two SVIB measures of the practicing engineers, the first when they were freshmen in 1935 and the second, 31 years later in 1966. This led to a serious method- ological problem due to the non-independence of samples. Because of this problem, the results are difficult to accurately interpret. For example, what would it mean if 24 he found a difference between the practicing engineers on the two measures? Was it due to a change in subject interests, a change in cultural activities, or some combi- nation thereof? A related problem deals with the corre- lation between the practicing engineers' responses from time one and time two. These correlations were not taken into account and there is no way to determine the effect of this imprecision on the analysis. Clemens was either unaware of the problems or chose to ignore them and re- ported that in fact the discriminant function technique did separate the three criterion groups. These results seemed to withstand cross-validation although the criteria for correct classification on this aspect of the study were not clearly specified. The author attempted to analyze the data further by breaking the engineers down into fifteen subgroups and then generated the discriminant function using the subgroups as the criterion groups. This discrimination was unsuccessful in accurately classifying individuals in criterion groups. Clemens concluded: In summary, based upon the empirical results of this study, meaningful conclusions were made possible because of the statistical discrimination yielded from multiple discriminant function analysis of SVIB data for three overall groups of engineers. These results supported the assertion that multiple discriminant function analysis is an effective statistical tech- nique for discriminating among occupational groups where descriptive-numerical data are available (Clemens, 1969, p. 76). Tiedeman. discrimir. to milita continued of the tw generaliz teria and more subj case. Th applicabi generate ; '1‘. function 1 Vard Univ: Vocational interest c- curricnlm 0f 32 in t ticl'Pélted function I 25 Much of the early work conducted by Rulon, Tiedeman, Tatsuoka, and Bryan (1951, 1954) using the discriminant function technique can be directly traced to military personnel classification. Creager (1957) continued in this vein and presented a lucid illustration of the two criterion-two variable case. Creager quickly generalized the arguments to allow for any number of cri- teria and any number of variables as long as there were more subjects than variables. This was the multivariate case. This discussion demonstrated the feasibility and applicability of the statistical technique being used to generate personnel classification. Tiedeman and Bryan (1954) applied the discriminant function technique to data obtained from students at Har— vard University responding to the Kuder Preference Record, Vocational. Each subject received a score in each of nine interest areas. There were 289 sophomore students in five curriculum areas (the number of students ranged from a low of 32 in biology to a high of 78 in government) who par- ticipated in the study. The resulting discriminant function based upon nine predictor variables and five criterion groups yielded two significant latent roots which accounted for approximately 91% of the total vari— ance. Contour scores were generated for each subject for classification purposes and these classification assign— ments were found to be quite accurate. Since the function 26 of the article was illustrative, cross-validation evidence was not included in the discussion. Chi-Square Weighting_Technique A multiple weighting scheme for the survey items on the Kuder for selected occupations was generated by Porter (1967) using a Chi-square technique. The purpose of his study was to generate a scoring system which would improve the predictiveness of the Kuder. A group of occupations classified as being similar in nature and a group of occupations classified as dissimilar in nature served as the two major target pOpulations for the study. The similar occupations were optometrists, pediatricians, veterinarians, physical therapists, and X-ray technicians; the dissimilar occupations were clinical psychologists, social case workers, optometrists, foresters, and auto mechanics. Porter provided evidence demonstrating his weighting system superior to the unit weighting system employed by Kuder (Form D) (note the distinction between Form D and Form DD) with occupations that were similar in nature. The results of the cross-validation indicated that only minor shrinkage occurred when the weights were applied to the new sample. Unspecified analytical problems associ— ated with the Kuder scoring system on the dissimilar occu- pations made comparisons in this area impossible. Results of Porter's study demonstrated the feasibility of using non-unit weighting systems to improve personnel- classi origin thus r than w 27 classification. The data analyzed in this study were the original data upon which Kuder based his scoring keys, thus rendering the Kuder weights slightly more predictive than would generally be the case. The method of obtaining the weights via the Porter approach was accomplished by listing the selected occu- pations along one margin and listing the six response patterns* available for each item along the second dimen- sion. Porter then obtained the observed and expected frequency counts for each cell and then determined the contribution of each cell toward the total Chi—square value. The resulting cell contribution served as the occupational weight for the response alternative for a given item. This procedure was repeated for each of the 100 items on the inventory. A positive or negative sign was assigned to the weight on the basis of the algebraic sign of the unsquared value of the numerator of the Chi— square definitional formula. Thus a subject's total *The six possible response patterns were derived by considering all possible response alternatives for a given item and then assigning a consistent value of 1 through 6 to each unique alternative. For example, for each item there are three statements; one statement must be chosen as "most liked" and one must be chosen as "least liked." Since the statement that is endorsed as most cannot be the same as the one that is chosen as least, only two alternatives exist for each least selection after the most selection is made. There are three possi— ble most selections for each item. Therefore, there is the possibility of six distinct response patterns for each item. This system exactly coincides with the re- sponse patterns presently employed in the Kuder Form DD lambda scoring procedure. cf SC< CECUII in da‘ the it consul for t} Kuder ‘ man anc researc in term zations CCEPOne sl'Sted Versity they Suk than One mined E PSYCholOS 0n 6 0f t} 28 score was the sum of all 100 weights of the response Patterns that he chose with respect to a specific occu- Pation. A subject was classified by determining the °°<=\Jpational scoring scheme which yielded the largest sum Of scores. It was immediately apparent that this pro- cefilure was somewhat complex and required several steps 511 data manipulation. However, as Porter asserted, with the introduction of the computer into educational and consulting situations this was a modest price to pay for the increased accuracy. Kuder Related Research Several studies have been completed in which the Kuder has been used as the basis for the research. Tiede- man and Bryan (1954) provided the initial multivariate research on the Kuder. Their study was rather limited in terms of number of subjects and possible generali- zations. Shutz and Baker (1962) conducted a principal components analysis of the Kuder. The respondents con- sisted of 450 freshmen males at a large southwestern uni— versity. After they derived the principal components, they submitted all components with eigenvalues greater than one to a varimax rotation. They subsequently ob— tained eight rotated factors which appeared to make psychological sense. It was interesting to note that one of the factors, health scientist, contained most of the occupations designated by Porter (1967) as being mEtho< evideI Concll more 1 inferE disCri method KudEr Practi tive t: 29 Shfl\lear. The dissimilar occupations identified by Porter aIDPeared to fall randomly throughout the eight factors. King and Norrell (1964) generated a factor analytic study on the Kuder using 464 freshmen males from a large midwestern university. It is not clear whether the authors employed a principal components analysis or a factor analysis, but an unnamed rotational procedure was applied to these data resulting in a six-factor solution. The results closely parallel the work of Shutz and Baker. Kuder (1957) did a comparative study on various methods of develOping scoring keys for his instrument. These included configural scoring, unit scoring, answer position scoring, and unit scoring on pattern analysis keys. The study was limited in both the characteristics of subjects (very homogeneous) and the relatively weak methodologies employed. The results did not provide evidence that one method was superior to another. He concluded this study with the assertion that considerably more research was necessary in this area before definitive inferences could be drawn; and to be certain of maximum discrimination, the exploration of the widest variety of methods should be attempted. In a later publication, Kuder (1961) argued that fractional weights were not practical due to the high cost of determining them rela- tive to the limited payoff. Cost was explained in terms of scoring complexity and payoff in terms of increased validity. imitat'u as compu‘ ramifica' fication stated, by hand < thereby I in scorir used, why differenc 0f mecha: aG‘Ogsted a aVailabil man and B ”Sighting SYStem is 30 val idity, reliability, or accuracy. The mechanical ltnlitations espoused by Kuder represented a weak argument as computer utilization became more accessible and the ramifications of lack of precision in occupational classi— fication were considered. In addition (Kuder, 1961, p. 4), stated, "the occupational inventory can be scored either by hand or by machine." The reference to machine scoring thereby weakens his claim for the necessity of simplicity in scoring. If a machine scoring procedure is going to be used, why not use the most accurate procedure if the only difference between procedures stems from the complexity of mechanical computations. Kuder (1966), subsequently adopted a non-unit weighting system. Considering the availability of computers, and the works of Porter, Tiede- man and Bryan, and Clark, it is apparent that fractional weighting systems are realistic and feasible alternatives to the unit weighting system. The question is, which system is most accurate? Summary Through the years, there has been much controversy concerning the most accurate method of weighting inventory items. The argument persists. Three distinct methods of weighting the items on an interest survey have been identi- fied in addition to the lambda technique currently used on the Kuder. They were: (a) item unit weighting; (b) pattern analytic; and (c) Chi-square multiple weighting. 31 A (igiscriminant function multiple weighting technique was aLsc: identified as a candidate to be used with at least ordinal level data. All the weighting schemes were amPirically based scoring systems. The generation of scores from any scoring system that is empirically based will fluctuate when the participant subjects or criterion groups are altered. The question most frequently posed was: can the utilization of a multiple weighting scoring scheme significantly improve the predictability of the Kuder, whether using a linear or non—linear model? Porter demonstrated that predictability could be improved using the Chi-square technique. Kuder and Strong have adopted scoring procedures that function "adequately" even though there may be more efficient methods than those currently practiced. Therefore, is it possible to identify which of the pattern analytic, Chi-square, discriminant function, or lambda predictive systems most accurately classifies individuals with respect to selected occupations? vestig scorin CHAPTER III DATA AND METHOD Statement of Problem The research reported here was an empirical in- vestigation of the relative accuracy of a pattern analytic scoring technique, a discriminant function sum of items weighting technique, the Chi-square item weighting tech- nique, and the Kuder lambda weighting technique. Accuracy is defined as the proportion of persons correctly classi- fied into their respective occupations using the cross- validation samples as the criterion groups. The following hypothesis specifies the nature of the research more precisely: H1: When considering more than two similar or more than two dissimilar occupational groups the dis— criminant function technique will most accurately classify the subjects into occupations according to the sum of item responses to the Kuder (Form DD), the pattern analytic approach will be next best, the Chi-square approach will be next best, and the Kuder lambda system will be least accur— ate. Overview of Procedure The procedure followed in testing the hypothesis is described under the seven main divisions of the 32 renainde pational discriru scoring validati among va score th it was n discrimi tem, and On the fl analysis Correct c IlaS used aSIDGCtS c 33 remainder of this chapter. These are: (l) Kuder occu- Pfitzional interest data; (2) Kuder scoring system; (3) discriminant function scoring system; (4) pattern analytic scoring system; (5) Chi—square scoring system; (6) cross- validation techniques; and (7) classification comparisons among various systems. To evaluate the hypothesis it was necessary to score the data using the Kuder lambda weights; likewise, it was necessary to score the data using the multiple discriminant function weights, the pattern analytic sys- tem, and the Chi-square weights. Scoring procedures based on the four systems were compared using a repeated measures analysis of variance on the cross-validation data. The correct classification for each cross-validation sample was used as an indicator of accuracy. Each of the seven asPects of the data and method will now be described. Occupational Data on the Kuder The data for the present study consisted of the responses on the Kuder of a total of 3906 male subjects from nine different occupations. These were the original data used by Kuder to construct the lambda scoring keys for the nine particular occupations. The present research could not have been conducted without the cooperation of G. Fredric Kuder (originator of the Kuder), who collected the data initially, and Andrew C. Porter, who had obtained the data from Kuder. PAtter; weightin! 34 Porter (1967) divided the nine groups into two sulagroups of similar and dissimilar occupations. Later factor analytic work gave evidence supporting the homo- geneity of the occupations in his similar group and the heterogeneity among the occupations in his dissimilar group. The homogeneous occupational set was composed of 406 optometrists, 436 pediatricians, 400 veterinarians, 386 physical therapists, and 274 X-ray technicians. The heterogeneous set included 500 clinical psychologists, 452 social case workers, 406 Optometrists, 348 foresters, and 298 auto mechanics. Note that the optometrist data appeared in both sets. Thus, there were two sets of five occupations. The similar occupations were labeled Set I and the dissimilar occupations were labeled Set II. For purposes of cross-validation, the data cards for each occupation were divided into random halves; these random halves were labeled A and B respectively for purposes of identification within each occupation. The information for each subject consisted of an identification number, an occupation or criterion group number, and a response pattern number for each of the 100 Kuder items; this information was contained on two Hollerith cards. Kuder Scoring System The scoring keys derived by Kuder via the lambda weighting were used to predict occupational classification on the through for use Kuder w halves accurac was la: syst ms Analysis that thi ations w “Ere dic. 0f handl; POIate RIC (18an Of exist. I “59 with - 35 on the sets of data. These lambda weights were obtained Ithrough the cooperation of Science Research Associates for: use in this study. Since there was only one set of Kuder weights, these weights were applied to both random halves of the data in a quasi cross-validation. The accuracy of classification with the Kuder scoring system was later compared with that of the three other scoring systems. Pattern Analytic Scoring_System Clark (1968) developed the Criterion Pattern Analysis. There was empirical evidence which suggested that this approach might be useful in predictive situ- ations when both the criterion and predictor variables were dichotomous. Clark's computer program was capable of handling criterion and predictor variables that incor- porate more than dichotomous alternatives. However, evi- dence of the feasibility of these generalizations did not exist. If Clark's procedures could be generalized for use with the occupational groups of the present study, his technique might be useful in scoring the Kuder. The computer program for this approach required the researcher to specify the following parameters: (1) the number of criterion groups; (2) the number of occu— pations within each group; (3) the total number of 36 ObEiervations; (4) the discrimination level* for a pattern to be stored; (5) the minimum number of observations Inettessary for a pattern to be stored; and (6) a signifi- cance level for determining additional pattern searches (typically set at .05). A flow diagram, Figure 1 (Clark, 1968, p. 32), is presented to facilitate the understanding of the workings of the program. The parameters for this study were set by the researcher to allow for as many patterns as possible to be generated and still remain within the capacity of the Control Data Corporation 3600 computer. Since there were five occupational groups within each data set, the cri- terion group parameter was set at five. The number of observations within each criterion group corresponded with the number of subjects within each occupation (this number varied from a low of 138 to a high of 250 subjects within any given group). The total number of observations was determined by summing the number of observations within each of the occupations for each set of data. The maximum number of observations the computer program could accommo— date was 999. For these data, these parameters were set at 965 and 999 for Set I and Set II respectively. The dis— crimination level, the number of observations necessary for the pattern to be stored, and the significance level *The discrimination level is defined as the ratio of subjects within a criterion group having a specific pattern to the total number of subjects in all criterion groups that have the pattern. 37 Input a Start with the first data ' criterion category grget number of items in a pattern equal to one. (r=l) 3” N; no F DOES P \ yes ~ 4’ \ emial one? 1 1 Generate a pat- -Generate a one tern with r item pattern items out of the patterns \ from List 2 ’ (each of r-l items) I Does the pattern Increase r ‘ of items exist by one in the data? ' J yes Is the pattern acceptable es with regard to the criterion category? / Print pattern w {Bhould pattern be saved? (Items annexed es to it could make it acceptable)4, ( no Store pattern 4> in List 1 V 7 Done generating I“) /’ patterns with \q__ r items° i yes Are any patterns stored in List 1? no Destroy patterns in List 2 and put patterns from List 1 into List 2. Destroy patterns in List 1. W Done with all criterion categories? es Go to the next criterion catefipry Figure l A Flow Chart for Performing Criterion Pattern Analysis that a the pa Cf cer contro and is 900d p; uP011 t1 After 1 rived I group, items ‘ Sdld tC of disc giVen C all the 38 for determining additional pattern searches were manipu- lated freely by the researcher in an effort to obtain the most accurate predictions; the final resolution was .70, .35, and .05 for the three parameters respectively. This manipulation is discussed in a later section of the thesis. The program, when executed, generates item patterns that are predictive of criterion groups with respect to the parameters specified by the researcher. The setting of certain of these parameters is completely under the control of the researcher (e.g., discrimination level), and is problem specific. Therefore the ascertainment of good predictive patterns was an interative procedure based upon the subjective manipulation of the program parameters. After these parameters had been specified, the program de- rived predictive patterns separately for each criterion group. This was accomplished by identifying patterns of items which occurred with some frequency. The pattern is said to be predictive if a certain high proportion (level of discrimination set by the researcher) of subjects in a given criterion group have the pattern with respect to all the subjects in the study. Typical lengths of the patterns ranged from two through five items. A simplified example will illustrate the procedure. Suppose we just have data from the Kuder on two criterion groups, and as already stated, there are six possible response alternatives for each item. Also suppose the crite the at Eatic: 39 researcher set the discrimination level at .60 and the minimum number of observations necessary to store the pattern was set at 25. This would indicate that before a pattern would be stored in computer memory, at least 60% of the persons in one of the two criterion groups must have the pattern and a minimum of 25 persons (group A plus group B) must have the pattern. Thus 15 persons (60%) in one criterion group must have the given pattern or the pattern and all other possible combinations of the items included in the pattern will be rejected from further consideration. A pattern is defined as predictive of the criterion group when the discrimination index generated on the actual data is equal to or greater than the discrimi- nation index specified by the program parameter card. The program continues to consider scoring patterns in the data until all possible response patterns for all subjects have been exhausted. At this point, the program prints all acceptable patterns found in the data (accord— ing to the original parameter specifications) from each criterion group. Criterion group A might have one pre- dictive pattern identified and group B might have two pre- dictive patterns identified. A possible pattern for group A might consist of the following: 1 (2), 3 (l) .84; this is indicative of item one and response number two paired with item three response number one. The .84 indicates that 84% of the persons who had the pattern were in gro: of per: patter: Interpi analogc are pla by Clar persons generat. terion g of the E that pat Patterns PAttern Criterio; subIECts 100 and J gram is i ject one “one A. Subject r dictiVe r Classific subjeCt rt 40 in group A. The program also indicates the total number of persons who had the pattern. Group B might have the patterns: 1 (l), 2 (l), 3 (2) .71; and l (2), 3 (2) .68. Interpretations of the patterns from group B would be analogous to those in group A. To predict or classify individuals, the patterns are placed directly into the prediction program generated by Clark (1968). This program systematically lists all persons involved in the study from one through N. It generates a symbolic representation of each of the cri- terion groups beside each subject. If a subject has any of the patterns predictive of any of the criterion groups, that pattern is momentarily stored until all predictive patterns of each criterion group are identified. The pattern with the highest discrimination index for each criterion group is printed out for each subject. Suppose subjects one and two were from group A while subjects 100 and 101 were from group B. The output from the pro— gram is illustrated in Figure 2. Note in Figure 2, sub6 ject one has only one pattern and it is predictive of group A. This would be a "hit" or correct classification. Subject number two also has one pattern, but it is pre- dictive of group B. This would be a miss or incorrect classification. Subject number 100 is a "hit" while subject number 101 is in a state of quandry. Clark advo- cates classification to be made in this situation with An reSpect t would be than .71 to the or patterns index for some thing ClaSSifi manna fr subjeCts might 1e: 41 . Criterion Number of Discrimination Subject Group Patterns Pattern Index 1 A 1 1 (2), 3 (1) .84 B 0 2 A 0 B 1 1 (2), 3 (2) .68 A 0 10° B 1 1 (2), 3 (2) .68 A l 1 (2), 3 (l) .84 101 B 2 l (1). 2 (1): 3 (2) .71 Figure 2 An Illustration of the Output from the Prediction Program Associated with Computer Pattern Analysis respect to the highest discrimination index. Subject 101 would be classified into group A because .84 is greater than .71 and therefore a miss would be recorded according to the original procedure. Note that subject 101 has two patterns predictive of group B but the discrimination index for the two patterns is .71 (listed pattern) and something between .60 and .71 (unlisted pattern). This classification procedure was modified in the following manner for the present study. A large number of patterns was generated for subjects in this study. As a result, any given individual might legitimately have had several patterns. It was possible that the "relaxix from whi cedures possible nation I patterns a subjec A positi t3ke bot Criminat weighted HE disc the numb Values 0 indices . deviated 1’2: an: subject 1 Printing along Wit criteriC: squared , Criteric. 42 pos sible to increase or decrease the number of patterns tha. t the original program generated by systematically "re laxing" or "tightening" the controlling parameters. Thu 3, there appeared to have been a number of continuums from which possible pattern analytic classification pro- cedures could have been selected. At one extreme, one Possible continuum could be the highest level of discrimi- halt ion (Clark technique) regardless of the number of Patterns and at the other extreme, the number of patterns a subject had regardless of the levels of discrimination. A position somewhere between the two extremes which would take both the number of patterns and the level of dis- crimination into account was adopted. This new procedure Weighted each pattern according to the squared value of tlie discrimination index thus taking into account both the number and the magnitude of the patterns. The squared vallues of the index was chosen to give more weight to iI‘Adices close to 1.0 and less value to indices as they deviated from 1.0. In the example of Figure 2, subjects 1 o 2, and 100 would not be affected by the change, but s‘~-‘-ll:3ject 101 would be affected. This change required the printing of all predictive patterns for a given subject along with the corresponding discrimination indices. The er iterion group which had the highest summation of the s(Inared discrimination indices was designated as the criterion for that subject. Using the new standard, 43 subject 101 [(.84)2 for group A compared with (.71)2 + (.60)2 for group B] would now be classified as a member of group B. As the discrimination index was relaxed, more patterns emerged from the program (other parameters held constant) and thus the distinction between the two methods became more crucial. A minor aspect of the present study compared the relative accuracy of the proposed pattern analytic classification procedure with Clark's original procedure. The item scores for this aspect of the study were the same six response patterns for each item as those used by Porter (1967) in the Chi—square study and those currently used in the Kuder lambda technique. The pat- terns generated by the computer for these subjects and these items were placed directly into the prediction program for purposes of cross-validation. Discriminant Function Scoring System It has been successfully demonstrated that the multiple linear discriminant function analysis is extremely proficient in classifying individuals with respect to criterion groups. This differentiation is achieved by a multivariate statistical procedure discussed by T. W. Anderson (1958). Conceptually the process is quite com— plex as it generates multivariate density functions in a number of dimensions (four for this study). The mathe- matical manipulations are somewhat less complex and can the alt Cr i. lam Prec 44 be mastered through an application of matrix algebra. Essentially, the problem reduced to the determination of one or more vectors of weights which when applied to the corresponding response data maximally separated the five criterion groups with respect to the ratio of among groups sum of squares to the within groups sum of squares. Vio- lation of the assumptions of the statistical model are not crucial in the development of a scoring system, pro- viding the initial results were meaningful and these re- sults hold up under cross-validation. There have been several studies which clearly did not satisfy the assumptions of the multivariate normal populations under- lying the sample data, yet the results have been encourag- ing, e.g., Chappell (1967), CoBabe (1967), Clemens (1969), Tiedeman and Bryan (1954). These findings lend support to the position that positive results may be obtained in a study of this nature even if the assumptions of the statistical model are not met. The level of measurement of the data has also not seemed to have seriously hampered the functioning of the technique in the studies of Chappell (1967), Clemens (1969), and Tiedman and Bryan (1954), although the data of the present study used in the dis- criminant function are ordinal. Essentially, the present research used the total lambda score of each subject for each criterion group as predictor scores. Thus there was a criterion variable 45 (group membership) and five predictor variables (one pre- dictor from each group summated lambda score). For example, a forester would have the classification score of the forester group plus a total lambda score for each of the five occupations in Set II. The total lambda score based upon each occupational key would be obtained by scor- ing the 100 response patterns of the subject with the lambda scoring key for the occupation under consideration. The lambda scores were analyzed and the resulting vectors of weights were used in the prediction model to differenti- ate among the five occupational groups. The prediction of group membership was determined through the use of Mahalanobis' 02 statistic. Essentially, this procedure determines the centroid of the multivariate response con- tour for each of the groups under consideration. Then each subject's score vector is compared with the centroid score of each group and the deviations of the vector scores (squared and summed) around the centroid are used as an indicator of the subject's similarity to the rest of the group. The centroid yielding the lowest D2 results in the subject being classified into that group. Because the derivations of the lambda scores were based upon the data used in this study, a true cross- validation procedure was not possible for this aspect of the study. Therefore a quasi cross-validation 46 procedure was employed.* This was accomplished by deriv- ing the weights for the discriminant function on half A of the data and then applying these weights to half B of the data. The effect of this non—independence of data samples probably resulted in a slight "over fitting" of the model, which ultimately translated into increased accuracy in prediction. Chi-Square Scoring System Porter (1967) developed an empirical weighting system for the Kuder based upon a Chi—square approach to group differentiation. The response pattern for each item was analyzed by creating a two-dimensional matrix of six response patterns by five occupational groups. The frequency of selection of each response alternative for subjects within each occupational group was determined and placed in the data matrix. Using the marginals and total frequencies, and applying the Chi-square statistic, the expected values for each of the 30 cells in the matrix were determined. The individual cell contributions to the total Chi-square value were subsequently obtained and used as weights in the Porter study. A signed value (+ or -) was given to each weight on the basis of the sign of the unsquared numerator value of the Chi-square *From this point, use of the term cross-validation will refer to cross-validation or quasi cross-validation, depending upon appropriateness. 4. L0 ins“ 47 2 def initional formula [XI—DEE under consideration. This procedure was followed for each ] for the particular cell item and resulted in a total of 600 weights for each occupation. The scoring for a given occupation for a given subject was based upon 100 of the possible 600 weights according to the response pattern of the subject. For example, subject one would necessarily have 100 response patterns (one for each item). The weights correSponding to these patterns for each of the respective 100 items would be summed (within an occu- pational key) and this sum would be compared to the sums resulting from application of the other scoring keys to the response patterns. The occupational key which re- sulted in the highest sum of weights based upon the subject's response patterns was used to designate the classification group for that subject. Cross-validation procedures were initiated. After Chi-square weights were determined by analyzing the data from half A, these weights were used to score the data from half B. The accuracy of prediction based upon the cross-validated Chi-square scores was compared with that of other scoring systems. Cross-Validation Technique Cross-validation is used in studies dealing with prediction or classification of individuals in an attempt to insure the general applicability of the derived scoring uic] 48 system. As applied in this study, the technique involved the random separation of a large sample of individuals under study into two independent groups. A predictive scoring procedure was then derived (independently) for one-half of the data. The derived scoring weights for half A were then applied to half B to determine the con— tinued accuracy of the scoring procedure. The purpose is to determine the amount of "shrinkage" in the prediction model. Shrinkage is defined as the reduction in accuracy or predictiveness due to chance relationships in the parent sample when comparing the results of the parent sample to the results of the cross-validation sample. Cross-validation procedures are designed to identify pre- diction equations which are the result of chance relation— ships or relationships which are idiosyncratic to the parent sample rather than "true" relationships. The present study employed cross-validation and/or quasi cross-validation procedures with each of the prOposed scoring systems and then used these results (hit or miss --dichotomous variable) as the criterion for determining the accuracy of the four predictive systems. The hit or miss criterion was used as the dependent variable in a repeated measures analysis of variance based upon the four methods of scoring and five occupational groups. Herzberg (1969) in a Psychometric Supplement discussed several properties of cross-validation studies and the a Carlo st‘u‘ size tenc measures tended t1 measures the squa interest cross-va estimate. cross-va Cross-Va, relation: ables. f the Pres. number 0: Emple 5. age of t] results I of predic CdSe the for an n cross‘Va variable. Variable 49 iflfil the associated statistics and parameters in a Monte Carlo study. Herzberg found that an increase in sample size tended to decrease bias associated with predictive measures, but that an increase in the number of measures tended to increase the bias associated with the predictive measures. This was true for the population parameter p2, the squared pOpulation multiple correlation. Of most interest was the finding that the sample and population cross-validities were approximately equal and both under- estimated pz. These findings indicated that the sample cross-validation was a good estimate of the pOpulation cross-validation, but probably underestimated the true relationship between the predictor and criterion vari— ables. This finding had significant implications for the present study, particularly because of the very large number of subjects in each of the criterion groups. With sample sizes this large, there should be minimal shrink- age of the cross-validation sample data results from the results based upon the parent sample. Also, the number of predictor variables in two cases was 100 and in a third case there were five predictor variables; this may account for an "overfitting" of the model. Greater shrinkage on cross-validation should occur in the methods using 100 variables compared to the method using five predictor variables. 50 Classification Comparisons Among Various Systems The comparison of scoring key accuracy was ascer- tained by placing the "hits" in the cross—validation samples for the four selected techniques (Kuder, Pattern Analysis, Discriminant Function, and Chi-square) in a repeated measures analysis of variance. Recent Monte Carlo studies on the analysis of variance model using nominal level data for the dependent variable have indi- cated that the parameter estimates and the expected values were remarkably close to the population parameters (Hsu and Feldt, 1969; Lunney, 1970; Draper, 1971). Draper (1971) also determined the expected values for a repeated measures design when the dependent variable was a Bernouli variable. These results indicated that a conservative test of the main effect (Greenhouse and Giesser, 1959) would yield valid results, i.e., the actual and theo- retical values for Type I errors are extremely close. The Draper study was also a Monte Carlo procedure. The independent variables in the design were: (a) occupation (five); (b) repeated measures (four scor- ing systems); and (c) subjects (nested within occupation). The dependent measure was a hit or miss classification score for each of the four measures. Since this was a repeated measures design, there were proportional sub- class frequencies, thus the unequal number of subjects among occupations was not a problem. Two separate 51 anailyses were conducted; one analysis was conducted on the similar occupations and one analysis on the dissimilar occupations. In addition, a comparison was made between Clark's original pattern analytic scoring technique and the modified procedure used in this study. This compari- son was based upon the proportion of correct classifi— cations in the cross—validations. Summary The major emphases of this research were: (a) to develop scoring systems for selected occupations on the Kuder (one derived by a pattern analytic and the other derived by a discriminant function technique); (b) to demonstrate the feasibility of these procedures; and (c) to compare the accuracy of classification of these new systems to each other and to the Kuder and Chi-square scoring systems. Cross-validation or quasi cross- validation procedures were employed in each of the four scoring systems. The data for the study were the original response patterns upon which Kuder built nine lambda occupational keys. CHAPTER IV RESULTS AND DISCUSSION General Overview The accuracy of the methods of scoring the Kuder are described in detail in this chapter. Data for five similar occupations (Set I) and data for five dissimilar occupations (Set II) were independently analyzed; thus, two distinct analyses were done. The first section deals with the proportions of subjects classified correctly in both analyses. The second section is concerned with the repeated measures analyses for both Set I and Set II. The third section reports the simple effects analyses and is followed by the Scheffé post hoc analyses. The fifth section is a report on the relationships among the scoring methods. The sixth section reports on the results of the Kuder lambda technique. The pattern analytic method and the comparison between Clark's original scoring procedure and the prOposed scoring procedure are illustrated in the seventh section. A report on the discriminant function scoring system results follows. Section nine is a brief discussion of the Chi-square scoring method. Section ten 52 :resent ased u I" ‘ 4 (V In {953' .. sample the sec and disl weights occupat; Kuder 15 Associat 53 presents a brief comparison among the scoring systems based upon the parent sample results (uncross-validated results). Proportions of Subjects Correctly Classified For Set I and Set II data, one—half of the total sample for each occupation (half A) was used to generate the scoring weights for the pattern analysis, Chi-square and discriminant function respectively. These scoring weights were then applied to the remaining data from each occupation (half B) for cross-validation purposes. The Kuder lambda weights were obtained from Science Research Associates and applied directly to half B data. It should be noted that the original data and the Chi-square scoring weights from half A for Set I and Set II were obtained from Porter (1967) and used in these analyses. A comparison among Porter's percentage of correctly classified males and the proportions obtained in this re- analysis revealed a rather close correspondence. However, there were minor discrepancies, and these data can be found in Appendix B. The reasons for the differences were most likely due to either: (a) a small random re- duction in the number of cases within a set of data so that all methods of analysis could be handled by avail- able computer capabilities; (b) mechanical errors in the manual classifying procedure; (c) errors in the recording 54 or key punching of the classification results; or (d) some combination of the above. These discrepancies did net; appear to be of sufficient magnitude to cause concern. Science Research Associates were very c00perative in the research effort. They provided the Kuder lambda weights for eight of the nine occupational groups . . . tWice. After the data had been scored and summarized with the original set of lambda weights, there appeared to be a lack of accuracy associated with the scoring system. A check with SRA confirmed the suspicion of the researcher. The initial set of weights were not the correct occupational keys. The situation was remedied and progress on the research continued. Another interesting facet of the study unfolded when the research firm could not locate the scoring key for the social case worker occupation. The original raw data were given to the researcher along with a formula for the lambda scoring system. From verbal reports at the testing firm, certain occupations just do not lend themselves readily to differentiation from other occu— pations, and apparently social case workers fall into that category as there appeared to be minimal usage of the scoring key. After applying the scoring formula to the raw data, a set of weights was obtained and used in the scoring procedure. Unfortunately, when 0% of the 226 social case workers were classified as social case workers, 55 the researcher again suspected the scoring key. After obtaining the derivation of the scoring formula from the research firm, two errors were found in the derivation. After the errors were corrected, the formula was applied to the original raw data for the social case workers and a new scoring key was obtained. Using half B of the data, the newly derived lambda weights correctly classified approximately 56% of the social case workers. Also, the discriminant function statistic based on half A data weights classified 76% of the social case workers in half B of the data in the quasi cross-validation. Table 1 presents proportions of males correctly classified by each of the four methods of scoring for each of the five occupations in the cross-validation using the weights obtained from half A, Set I data to score the data from half B, Set I. Table 2 presents the proportions of males correctly classified by each of the four methods of scoring for each of the five occupations in the cross-validation using the weights obtained from half A, Set II data to score the data from half B, Set II. Repeated Measures Analyses The results of the dichotomous classification cross—validation procedures were submitted to a mixed design analysis of variance with one between and one within subjects variable. In the repeated measures 56 com u mz loom u vz “mam u mz lama u Nz 1now u Hz ommm.usnm oemm.umnm mass.u~nm mama.ua.m new: avenues mane. u .mm comm. cons. ooom. omem. meanuacaumum> .m mamm. u .«m emom. ease. some. mane. mnmaamuasa Hmoamsaa .s ommm. u .mm mass. memo. mama. mama. mannonnumaema .m mane. u .wm some. anew. mwam. oooo. acanonanoma sauux .N seam. u .fim memo. meme. somm. mmma. mumnuumsouao .H new: cofiuocsm monfimq mumsom mflmhamcd msouu ucmcHEHuomaa Hocsx Iago cuouumm cowumoflam>nmmouo can so .mumc H now How onEmm cowmammmao wauoouuoo moans mo mcoauuomoum onann.a Manda com u mz imam u az «man u mz lama u Nz 10mm u Hz mmam.ueam mmme.um»m aheo.um.m «Ho~.uasm cams meoaumz omme. u .mm omms. omom. omos. omao. nmnuumsoumo .m Hmmm. u .«m Heme. memm. owns. swam. Hague: ammo Hanuom .e comm. u .mm have. «mom. Noam. ammo. “mummuom .m omme. u .mm seem. mmme. mmam. sema. oncmsomz cuss .N 0mm». u .am ommm. seam. osmm. ommm. umnmoaoaosmm Hmonanao .H can: coauocsm cussed mumsom mammamcfl msouu DGMCHEHHUmfia Hoosx Iago cuouumm .mumc HH now now oHQEMm cowumcwam>lmmouo may no ooamammmHo hauoouuoo moans mo mcowuuomonm o£BII.m mqm u m0 «mumflmmuowa Housmmsm w u so mmcmwowuumwoom H mm “newscacsooe mmqu H No «mumfiuuofioumo u mucouowmao ucmoamwcmwm n a museummwcmam mo Ho>oa Ho. n a mmH. emmmm.I «somo.I .oomm.I me I so mma. .oeeH.I .mom~.I .oaom.I me I no HmH. emmmo. aomom. smmoa. 4o I no sea. mmmH.I .msmm.I somom.I mo I No man. .oomo. ammo. momo.I 4o I «a mom. mono. oHHH.I «NeoH.I no I No oma. mmmH.I ammom.I .ommo.I mo I Ho mmH. .ooso. Hobo. moon. 4o I Ho mmm. mmoo. ooNH.I oomo.I no I Ho mam. oomo. ommo.I moms. me I do osam> «2 N: as HMOAWAWO coauocsh mumsom mammamcd moocmmwmman .mm a m uoaomsmuomma Iago cumuuam .oanmwum> moosuofi may mo Ho>oa some um mumo monoum H now no momMHmcm oon umom wmmosomtl.m Hands 71 mumanuoEoumo H mm “muoxuoz ommu ammoom u «0 “mnoumouom u no “moocmnooz cuss u no amumwmoHonohmm amowcflau u do oocoummuflo noncommcmfim u a mocmOHMHcmwm mo Ho>oa Ho. u a mma. Hmoo. .meo~.I .oomm. sommm. me I so mom. homo. Nooo. «man. Hooo. mo I mo omm. oomo. gamma. oooo.I .amm~.I so I no mom. ammo. osmo.I .moom. esmmm. mu I No mom. omoa. «ammo. .momm. ammo.I so I as sea. oamo. omao.I .mooo. .mmom. no I no mom. oooo. oomo. .ooao. somam. mu I Ho «on. mono. emoom. Iooom. mood. so I Ho 4mm. omno.I mono. aomom. soomm. mo I no «on. aoao.I homo. mooo.I mama. No I no osam> «2 m2 . NE a: mooconomuwn mewwmwm :0auocsm monamq mumsom mommamcm new: I ucmcwfiwnomfia Hoosx Ionu cumuumm . ”HQQHHM> moonuofi on» mo Ho>oa come we sumo museum HH now no momwaccm oon Doom wmmonomnl.n mqmfia 72 .mumfluuoEouoo u m0 m.ux3 ommo .oom u so “muoumonom mo x.cooz ousd H No m.no>mm .cmau u Ho ”HH new mom mmcmHumcHuouo> H mm m.uone .mhnm n «O mamaoauumwoom u no “smooacnooe woulx u no mummnuoeoumo n HO "H pom pom meowuocsm ucmnwfiauomao u v2 «monsmq nocsx u m2 “mumsomlfinu u N: “mammamcd cuouumm u as accommocmamucoz n mz Ho.uo um uneconocmwm n a mod. oma. moH. omm. «NH. mamm> Hmonumno wmmmaom mZoomo.I Immo~.I mZmooo.I mZVMHH.I mZomHo.I ozImz .oomm.I eaoom.I .momm.I mZoomo. mZoooo. ozImz .oooo.I mZmemo. .omo~.I «ooom. mZoomo. szmz sooos.I .mooo.I .mooo.I .Hoom.I .oooo.I ozIHz .oooe.I Ieoe~.I .mooe.I .ooom.I sommo.I szHz .ooom.I amomm.I .Heoo.I .ooos.I toooo.I szHz moan HH new man. mom. man. mos. mma. momm> Hmomumuu mommaum aoomm. .omoo. mZomoo. mZoeoo. mzmmoo. ozImz mZoooo. soaem. mZMHoo. mz~ooo.I mZOoNH.I «SIN: «ooom. mzHHmH.I mZAmmo.I mZoHoH.I .mmom.I szmz .omo~.I .HooH.I .omoo.I .oooo.I .mmmm.I ozIHz ozomoo.I emoom.I aooom.I .mmeo.I Immmo.I szHz .ommm.I .oooo.I .mooo.I .mmem.I aomam.I szHz when H new moocouommwa mo so no No Ho cams so no Ho>oa some um moonuoz .oHQMflum> mmsoum onu mo Hm>oa some um mumo HH now can H new no mommamcm ooz umom Wmmosomln.m Manda 73 procedure was used in this study because it was least sensitive to violations of assumptions; and, there were unequal cell sizes at each level of the groups variable. Table 6 illustrates the pair—wise contrasts of group means for each significant analysis for each level of the methods variable for Set I data. Table 7 contains similar information for Set II; both tables contain the critical values for the Scheffé test. An interval was created by the addition and subtraction of the critical value to the difference between means. If the interval crossed zero, the contrast was non-significant. If the interval did not cross zero, the contrast was significant. These analyses were generated using a = .01. The Scheffé critical values ranged from a high of .174 to a low of .152 for Set I data, and from .174 down to .144 for Set II data. Unless the differences between means ex— ceeded these values, the differences were non—significant. Optometrists (Set I) were significantly less accurately classified for the pattern analytic and Chi— square methods than the veterinarians, but more accurately classified than the physical therapists on the discrimi- nant function method. The x-ray technicians were more accurately classified than the physical therapists using the discriminant function method, but less accur- ately classified than the veterinarians using the pattern analytic and Chi-square techniques; and, they were less 74 accurately classified than the pediatricians using the pattern analytic method. The pediatricians were classi- fied significantly more accurately than the physical therapists, but significantly less accurately than the veterinarians on all techniques except the Kuder lambda. The veterinarians were classified significantly more accurately than the physical therapists on all but the Kuder lambda technique. All other pair—wise contrasts on Set I data at each level of methods variable were non- significant. Since the simple effects analysis for groups on Set I data at the Kuder lambda level was non—significant, this post hoc analysis was not generated. A11 pair—wise contrasts would have yielded non-significant results. An interesting phenomenon resulted when the simple effects analysis of Set II data yielded a significant F ratio for the groups at the discriminant function level and the Scheffé post hoc analysis failed to detect a significant pair-wise contrast. Two possible reasons for this were: (a) with the pooling of the variance estimates, any among groups heterogeneity of variance could have obscured the difference; and (b) the conservative nature of the Scheffé test and its lack of sensitivity to detect relatively small differences. Kirk (1968) argued that with sample sizes over 30 there is minimal bias associated with the pooling procedure. And, with a less conservative test, 75 or upon using a larger level of significance, significant contrasts could be obtained. In the Set II data, the clinical psychologists were classified significantly more accurately than the foresters and Optometrists when the pattern analytic method was used; and, they were classified significantly more accurately than the foresters, social case workers, and optometrists when the Chi-square method was used. The clinical psychologists were classified significantly more accurately than the social case workers when the Kuder lambda technique was used. The auto mechanics were classified significantly more accurately than the foresters and optometrists when the pattern analytic method was used; and, they were classified more accurately than foreSters, social case workers, and Optometrists when the Chi-square method was used. When the Kuder lambda method was used, the auto mechanics were classified more accurately than the social case workers. With the pattern analytic method, the social case workers were more accurately classified than the foresters and the Optometrists. The social case workers were also more accurately classified than optometrists when the Chi-square method was used. The results were reversed for the foresters and social case workers on the Kuder lambda method from that of the pattern analytic method, 76 i.e., foresters were more accurately classified than social case workers. All other pair-wise contrasts on the Set II data at each level of the methods variable were non-significant. Comparisons of Methods Means It was evident that method one, the pattern analytic, was significantly inferior to the other three (methods of scoring across all levels of groups in both sets of data, except the Kuder lambda procedure on the veterinarian group (see Table 8). The pattern analytic method simply failed to function. NO single method was always superior to the other methods. In Set I, the Chi-square method was signifi— cantly worse than the Kuder lambda method on the Optome— trist group. The discriminant function analysis was significantly better than the Kuder lambda on the veterinarian group, but significantly worse than the Kuder lambda on the physical therapist group. All other pair-wise mean differences within Set I at each level of the groups variable were non-significant. The Scheffé critical values ranged from a low of .129 to a high of .163. On Set II data, the Chi-square was significantly better than the Kuder lambda on two groups and signifi- cantly worse on two groups; there was no difference in the fifth group. The discriminant function was 77 significantly better than the Chi-square for the forester, social case worker, and Optometrist (Set II) groups. The discriminant function was also significantly better than the Kuder lambda for the social case worker group. All other pair-wise mean differences within Set II at each level of the groups variable were non-significant. The Scheffé critical values ranged from a low of .122 to a high of .150. Excluding the contrasts involving the pattern analytic method, the discriminant function had five sig— nificant contrasts, the Kuder lambda had four and the Chi-square had three significant contrasts. Overall, the discriminant function method classified subjects more accurately than the two remaining methods on the Set II data. When the occupations under consideration were dissimilar in nature, the discriminant function method classified individuals into their apprOpriate groups better than the Kuder lambda and Chi-square methods. For Set I data (occupations similar in nature), there was no one method which demonstrated consistent superiority over the other methods. Ignoring the pattern analytic technique, it appeared that the determination of the best method for classifying individuals appropriately for both Set I and Set II data was dependent upon both the method of scoring and the group under consideration. For example, the 78 Chi-square weights were most accurate in predicting the cross-validation classification of auto mechanics (97%), but could accurately classify only 40% of the Optometrists from Set II. Whereas the discriminant function weights were most accurate for classifying social case workers and foresters; and, the Kuder lambda weights were most accurate for classifying the Optometrists in both sets of data. Given the significant interaction, the major hypothesis Of this study was difficult to evaluate. Clearly it was apparent that there was no one best method to use to score the data from all groups to consistently Obtain the greatest accuracy. It was Obvious that one method was inappropriate. The pattern analytic method failed to function and was eliminated from consideration as a method Of weighting the Kuder occupational interest survey. This finding was in opposition to the research hypothesis which predicted that the pattern analytic method would be the second most accurate method Of scoring. The interaction of groups with scoring methods was not anticipated and also negated the research hypothesis. The Chi-square method proved to classify individuals better than anticipated as did the Kuder lambda method which was predicted to be the least accurate method but was the most accurate for five of ten groups. The Chi- square method was most accurate on three groups and the 79 discriminant function classified most accurately on the remaining two groups. Again, the resulting scores for comparisons were not all based upon true cross-validation thus putting the Chi-square method at a slight disad- vantage. Relationships Among Scoring Methods The variation and covariation of the dichotomous scores of correct or incorrect classifications defined by each of the four methods of scoring were considered. Phi coefficients for all pairs of variables are presented in Table 9. The pattern analytic method correlated minimally with the other three methods. The remaining three methods moderately correlated with each other (.45, .48, and .54 for Set I; and .31, .24, and .39 for Set II), based upon 965 and 999 observations respectively. With the ex— ception of the pattern analytic method, there was suffi- cient evidence to suggest that the methods of scoring had considerable overlap in the subjects who were accurately and inaccurately classified. Apparently (for the three functional classification methods) certain subjects lend themselves to classification (or non-classification) con— sistently across measures as noted by the moderate correlations among the measures. Appendix B contains the variance-covariance lnatrices of data Sets I and II based on the four methods .m mom N.Hv os m m mpmfimmumna Hmoamssm .v mam m.mv mmH m we mGMHoHuumHomm .m mma 0.5N OH 0 o mcmfioficnoma smuax .N mom o.os em ma mm mumfiuumeoumo .H mcoflummsooo umafiEHmnuH pom mocmno ma mcfluoom How Hmcflmwuo UmHMHooz z uomuuou comb mcumuumm coaummsooo vmwwflmmMHU omuommxm z owumumcoo m0 z wauomuuoo z .mumo HH now new H pom MOM mwusomooum m ehmuumu fimcmmmuo mam wmmomoum man mo somwummsounn.oa mqmda 2208 32:3 83 IUl enormous amount of computer time was expended. A rsaasonable and meaningful number of predictor patterns were derived for each set of data. Through a manual iterative process, it was determined that the discrimi- nation index must be set at .70 or higher, the level of significance set at .05 or lower, and the minimum number of observations set at 25 or higher in order to generate a solution to the problem. If any of the three basic limits was exceeded, the program would exceed the capacity of the CDC 3600 computer and subsequently abort the run, but only after consuming approximately one hour of central processing time. Even within the limitations of the pro- gram and the computer, "successful" runs consumed up to 90 minutes of central processing time. In all fairness to the pattern analytic technique, it should be noted that the research problems presented a severe test for the program. The program is capable of handling up to 999 observations with up to ten groups of subjects and a potential of over 100 variables with up to 10 responses per variable. The problems in this research involved up to 999 observations, 100 variables, five groups and six responses per variable for each set of data. There were problems with the determination of the predictor patterns. If the program controlling param- eters were "relaxed," e.g., minimum number of observations necessary for a pattern to be stored was dropped to 10, 84 time program typically exceeded the computer capacity. But, .tf'the parameters were "tightened," e.g., minimum number of (ibservations necessary for a pattern to be stored was in- creased to 75, the program would not exceed the computer capacity, but a very small number of predictor patterns ‘would be generated. Following many iterations, an acceptable number of patterns was obtained by setting the discrimination index at .75, the level of significance at .05 and the minimum number of observations necessary for the pattern to be stored at 35. This was true for both sets of data. Other combinations of these parameters failed to yield acceptable results. Typical occurrences for other combinations were: (a) zero patterns for less homogeneous groups, i.e., limited within group similarity; (b) the possible number of patterns for a particular group exceeded computer capabilities; (o) the job card time limit was exceeded; (d) minimal patterns were gener— ated for all groups; and (e) few patterns were generated for the less homogeneous groups while the more homogeneous groups yielded an overflow of patterns. The information given in Table 10 indicates that the number of patterns generated for specific groups varied from a low of 10 to a high of 345. Efforts to increase the minimum and decrease the maximum number of patterns within a data set were unsuccessful. The range of the number of item patterns was almost directly 85 proportional to the number of persons correctly classified :Ln.the occupation, i.e., veterinarians had 345 patterns and more than 100 persons correctly classified on the cross-validation procedure, whereas X-ray technicians had only 10 generated patterns and zero persons were correctly classified. Less than 5% of the total sample received no classification prediction on the cross- validation procedure. If, given no prior information, one considers chance to be the probability of being correctly classi- fied (one out of five for this study), correct classifi- cation exceeded the chance level in only five of the 10 groups. Correct classification exceeded the .50 level for but one of the five groups using the modified scoring procedure. Similar results were obtained when the origi- nal pattern analytic scoring procedure was employed. At this point, the researcher suspected a data error. After the Chi-square and lambda scoring pro- cedures were completed, a reanalysis of the data was con- ducted using the pattern analytic method. The prediction results were unchanged. The lack of precision can probably be attributed to limitations imposed on the program and the computer by the massive amount of data and the restrictions on the program parameters due to the "taxing" of the model. A third plausible explanation is that the model will provide accurate classification 86 for nominal level data only if the data are dichotomous (laoth for the predictor and the criterion). It is hereby recommended that the pattern analytic technique not be included as a research tool for purposes of occupational classification with respect to the Kuder. When the predictor and criterion variable are on a nominal scale of measurement and the data are not dichotomous, the pattern analytic technique should not be used. Addi- tional research may demonstrate the utility of the tech- nique under modified conditions, of course. Comparison of the Original and Modified Pattern Analytic Scoring Procedures The comparison of the pattern analytic scoring system prOposed by Clark with the modified version of the pattern analytic scoring procedure was a minor aspect of this research. Clark used only the highest discrimination index as the criteria for classification; the prOposed method used the sum of the squared discrimination indices. Any argument about the relative merits of the two methods was academic because over both sets of cross-validation data, neither method performed beyond the chance level (see Table 10). The only groups that were classified with any degree of accuracy within both sets of data were the veterinarians and possibly the clinical psychologists. This can probably be explained by the large number of 87 predictor patterns obtained for those specific groups iru the prediction program. The veterinarians and the clinical psychologists had 44% and 45% of the patterns associated with Set I and Set II data respectively. With a great number of patterns one would expect a large number {of hits. The number of predictor patterns a group had, compared with the total number of patterns within the set, almost dictated the number of hits that were likely to occur for a given occupation. Based upon the number of hits in the cross- validation data, the modified pattern analytic scoring method was equal to or better than the original method in eight of ten groups. When summed over groups in Set I, the original method performed slightly better than the modified method (but both were worse than chance). The modified procedure exceeded the original procedure for Set II data with both procedures hovering close to the chance level. It was concluded that neither method should be employed in attempts to classify persons if the problem considerations are similar to those of this research. Discriminant Function Scoring Results The weights for the discriminant function procedure were derived by scoring half A of the data with the Kuder lambda weights. These resulting scores were used (one for each occupation for each subject) in the discriminant 88 function analysis. Both Set I and Set II data yielded ‘ four significant latent roots (see Appendix A). The eigenvectors associated with the four latent roots were used to score a subject's response vector. The response vector (for each subject) was obtained from Kuder lambda scores in half B of the data. Mahalanobis' D2 statistic was employed to determine a subject's predicted occupa— tional classification. The sum of the squared distance of the observed scores from the group contour scores was the criterion for the classification. The classification total for each cross—validation sample is included in Appendix A. On the discriminant function, by definition, 100% of the trace was accounted for with the four latent roots of the determinental equation W—lA-AII= 0. For Set I, the four latent roots accounted for 41, 30, 20, and 8% of the trace respectively. For Set II, the four roots accounted for 48, 23, 19, and 11% of the trace respec- tively. The Chi-square test of significance for each of the latent roots was significant (p < .0001). This was interpreted to mean that the first latent root of Set I accounted for 41% of the explained variance, the second latent root of Set I accounted for 30% of the explained variance, etc. Similar results and explanations hold for Set II. 89 Even though the Chi-square approximations for ssignificance testing were significant well beyond the <:hance level, it was entirely possible that this resulted because the statistic had a great deal of statistical power due to the large sample sizes; i.e., high statistical significance does not necessarily insure a high degree of discriminatory power for a predictive instrument. One measure of the discriminatory power is the proportion of correct classifications on the cross-validation sample. Unfortunately, there is no single index of total dis- criminatory power or "percent of variance accounted for" with respect to the multiple discriminant function; no such multivariate statistic appears in the literature. However, Tatsuoka (1970) argued for the generalization of the univariate statistic wz. The argument reduced to the comparison of the variance attributed to group differ— ences compared with the variance attributed to the total sum of squares and cross-products. The multivariate analog to w2 becomes |_.}S:l N-k M 2 ITI " IW w mult ITI —— where T is the sum of the squares and cross-products matrix and W is the within groups sum of squares and cross-products matrix; k is the number of groups, and N is the total sample size. 90 When applied to the research data, Tatsuoka's :Lndex yielded wzmult values of .8019 and .9378 respec— ‘tively for Set I and Set II. This was interpreted to mean that approximately 80% and 94% of the total vari- ability of the four discriminant functions was attribu— table to group differences for Set I and Set II respec— tively. These results indicated a high degree of discrimination for these data. The data for dissimilar occupations (Set II) had slightly greater discriminatory power (based on the generated functions) than did the data for similar occupations (Set I). One would expect persons within similar occupations to respond similarly on the criterion measure and thus render differentiation more difficult. However, the group similarity did not restrict the "discriminability" of the Tatsuoka statistic. With the present research data, Wilk's lambda statistic provided a test of the equality of the five groups population centroids. The A values were computed as .197 and .062 respectively for data Set I and Set II. These results were both statistically significant (see Appendix A). The smaller the values of the A the higher the probability the group centroids are significantly different. Lambda is defined as the ratio of the pooled within groups deviation score cross-products matrix to the total deviation score cross products matrix. As the proportion of total variance attributable to among groups 91 differences increases (within groups variability remain- ing constant), the A score approaches zero. For this study, a very high proportion of the total variability was attributable to the among groups centroid differences. The use of discriminant function analysis on these data was successful in generating equations that were ultimately employed in a prediction model. Subse- quently, the equations were used to classify individuals into one of five occupational groups. As illustrated in Table l, for nine of the ten occupational groups the pro— portion of correct classification on the cross-validation procedure ranged from a high of .91 to a low of .64. The discriminant function technique worked very well for all but one group, physical therapists. Upon cross-validation, only 21% of the physical therapist group was accurately classified. This result puzzled the researcher in light of the total data. A data error was suspected, but none was found. The contour score for this group was in close proximity to at least one other contour when considered with the other similar occu- pations. Also, the physical therapist group did not seem to be as homogeneous as the other four groups. The pre- dictability for this group with respect to the other scoring methods was lower than all but one of the groups using the Kuder lambda and lower than all groups using the Chi-square method. 92 Set II prOportions of persons correctly classified on the cross-validation procedure were generally higher than Set I proportions for all scoring methods. This was expected because Set II data contained the dissimilar occur pations which typically lend themselves to more accurate classification than the similar groups. Chi-square Scoring Results The results for Chi—square scoring procedure were comparable in accuracy to the Kuder lambda and the dis- criminant function procedures. The Chi-square method over both sets of data was slightly lower in the proportion of correct classifications on the cross-validation than the Kuder lambda. The Chi-square technique was slightly better than the discriminant function on Set I data and slightly worse than discriminant function on Set II data. For certain groups, e.g., veterinarians and auto mechanics, the Chi-square procedure was the best method of scoring to obtain the most accurate results. The proportion of correctly classified persons on the cross-validation pro- cedure based on the Chi—square scoring system ranged from a low of .40 to a high of .97 across ten occupations. The Chi-square method was at a slight disadvantage com- pared with the Kuder lambda and discriminant function techniques because it used true cross-validation whereas the other two methods employed quasi cross-validation. 93 Comparison of Methods inguasi Cross-Validation In order to accurately compare the Chi-square procedure with the discriminant function and Kuder lambda procedures, the results of each method must be based on the same point of origin. Since the Chi-square method used true cross-validation and the other two methods used quasi cross-validation, the points of origin were different. Therefore an attempt was made to place the methods on equal footing. The results Porter (1967) presented for Set I, half A data weights used to score Set I, half A data via the Chi-square procedure (non cross—validation) were com- pared with the discriminant function and Kuder lambda procedures when applied to Set I data based upon the weights generated from Set I data in this study (non cross-validation) (see Appendix B). The results indicated over the five groups that the Chi-square procedure cor- rectly classified approximately 10% more of the persons than did the discriminant function and Kuder lambda pro- cedures. The Chi-square was more accurate than the discriminant function procedure in all but the Optometrist group for Set I data. The proportions of correct classifi— cations on the parent samples ranged from .61 to .91 for the Chi-square method, from .21 to .78 for the discrimi- nant function method, and from .53 to .71 for the Kuder lambda procedure. 94 A similar comparison was made for Set II data. The results were almost completely Opposite from those for Set I data, i.e., the proportion of persons correctly classified on the discriminant function and Kuder lambda procedures exceeded the Chi-square results by approximately 8% and 5% respectively over the five groups. The dis- criminant function results for three of the five groups for Set II data were more accurate than the Chi-square results. The proportions ranged from .73 to .89 and from .47 to .97 reSpectively. The discriminant function method was slightly more accurate than the Kuder lambda method, but there was no consistent pattern from group to group. These results supported the conclusion that no one method is best for all groups. With the discriminant function and Kuder lambda analyses based upon the rather large sample sizes, there was minimal "shrinkage" on the cross-validation sample. In several groups there was an increase in the proportion of correct classification on the cross-validation sample. When a similar comparison was made by Porter (1967) on the Chi-square scoring system, minimum to moderate shrink— age occurred on the cross-validation sample when compared with results Of scoring based on weights which were de- rived from and used with the parent sample. In all groups there was a decrease from the parent to the cross- validation proportions based on the Chi-square technique. 95 These results may be indicative of a highly generalizable set of weights based upon the Kuder lambda and discrimi- nant function procedures or possibly it is indicative of "overfitting" the prediction model through the use of quasi cross-validation. Also, it must be remembered that the Kuder lambda scoring keys were derived on samples twice as large as the samples used to derive the Chi- square scoring keys. Therefore, the Kuder lambda keys were probably more precise and yielded better estimates of the population parameters than did the Chi—square keys. This would account for the large shrinkage on the cross-validation sample when the Chi-square method was applied to these data. Herzberg (1969) presented evidence indicating that the population cross-validity is very accurately estimated by the sample cross-validity when the number of predictors is small and the sample size is large. The discriminant function results tended to support that finding. However, one must remember that these results were not derived through true cross-validation. His findings might explain the difference in the amount of shrinkage on the Chi-square and discriminant function scoring methods. With the sample size constant, the number of variables used in the predictor model for'the Chi-square was 100 whereas the discriminant function used only five variables. CHAPTER V SUMMARY AND CONCLUSIONS Four methods of scoring the Kuder Occupational Interest Survey (Form DD) were used in this research in an attempt to determine the most accurate method of classifying individuals into a select number Of occu— pations. A set of five similar and a set of five dis— similar occupations were used in conjunction with the pattern analytic, Chi-square, Kuder lambda, and dis- criminant function scoring procedures. The hit or miss classification of individuals within an occupation on a cross-validation sample was the basis for the comparison of methods. The data were analyzed by a repeated measures analysis of variance, using conservative tests for the sources of variation, and subsequent tests of all simple effects. The results indicated that the most accurate method of classification was dependent upon the homo— geneity of the group under consideration, the composition of the remaining groups in the data set being inspected, and, the method of scoring, i.e., no one method performed consistently better than the other methods. 96 97 The pattern analytic method was unacceptable for use as a method for scoring the Kuder. The method con- sistently failed to classify individuals correctly at more than the chance level for the majority Of the occupations. Additional research is indicated to determine the useful- ness Of the pattern analytic program when the predictor and criterion variables are nominal level data but not dichotomous and when the sample size is smaller than that which was considered in this research. Limited extensions of the technique as prOposed by Clark may be helpful in trying to determine the program's limitations. A minor aspect of this study considered the accur- acy of a modified version of the prediction program for the pattern analytic technique compared with the original prediction program. Neither method functioned very well; both resulted in chance level predictions. On Set I data, the Chi-square method was com- petitive (over all groups) with the Kuder lambda and the discriminant function methods. However, the accuracy of the Chi-square method in classifying persons among groups varied considerably from occupation to occupation as did the accuracy of the discriminant function technique. This variability was a contributing factor to the interaction between groups and methods. On Set II data, the accuracy of the Chi-square method again displayed considerable variability from group 98 to group, but across groups it did not perform as well as the Kuder lambda and discriminant function methods. Again, the variability contributed to the interaction between groups and methods. For certain occupational groups the Chi-square method performed extremely well, while in other groups it was not as accurate as the other methods. The Kuder lambda method performed the best of the four methods on five of the ten occupational groups; four of the groups appeared in Set I of the data. The pro- portion of correct classification on the quasi cross- validation ranged from .56 to .81 for all ten groups. Thus, the Kuder lambda was the most consistent method of scoring. The Kuder lambda scoring method had a definite advantage over the remaining three scoring methods. There was no true cross-validation for the Kuder scoring system, and the derivations of the Kuder weights were based upon sample sizes approximately twice the size of those used with the remaining three methods. With the larger sample sizes, parameter estimates are usually more accurate and this ultimately translates into higher proportions of correct classifications. It was possible that these two factors accounted for the consistent accuracy of the Kuder lambda scoring procedure. When this study was originally planned, there was a strong possibility that only the Kuder scoring keys and 99 not the formula for deriving the scoring keys would be made available to the researcher. Thus, a decision was made to incorporate quasi cross-validation into the design of the study. The scoring formula was subsequently released to the researcher but use of the formula was not integrated into this study. The discriminant function method performed very similarly to the Kuder lambda except on Set I, one group had only 21% accurately classified. On each of the other nine groups, the discriminant function method was either within chance error range or more accurate than the Kuder lambda results. On Set II data, the discriminant function method accurately classified 82% of the subjects across the five groups. As with all other studied scoring pro— cedures, the discriminant function procedure was more accurate in classifying the dissimilar groups than the similar groups. The four scoring methods were empirically based and changes in the composition of the five occupations comprising a set of data would probably alter the weights for all methods except the Kuder lambda. Replacing the current subjects with a new set of subjects within a par— ticular group may or may not have an effect upon the re- sults. There would probably be minimal changes associ- ated with the Chi-square results. The Kuder lambda weights had the advantage of being derived and tested with the same data. Thus, the Kuder lambda classification 100 proportions were probably slightly inflated due to the idiosyncrasies inherent in the data which contributed to the prediction model. These results probably would change if a new set of subjects were used in further research. The discriminant function procedure was based upon the Kuder lambda scoring procedure and therefore, the derived classification proportions are probably also slightly in- flated. These results would probably change if a new set of subjects were used in the research. The results of the study were clear. There was no one best method to score the Kuder Interest Survey to con- sistently obtain the most accurate occupational predictions. If one were forced to choose one particular method and ex- clude the others, the discriminant function method would probably be chosen for dissimilar occupations. NO method could be identified as best for similar occupations. However, if one could select the scoring method to fit the occupations under consideration for a specific situ— ation, a variety of procedures would probably be recom- mended. These results must be viewed as tentative as true cross-validation comparisons were not available for all scoring methods. An issue which has not been considered in this study is the expenditure of resources necessary to ascer- tain the discriminant function weights. Parameters which would be involved in the choice of a method would include: 101 (a) the desired or required accuracy of the scoring pro- cedure; (b) the purpose of the instrument; (c) the re- sources and facilities at hand; (d) the sophistication Of the researcher; and (e) the composition of the groups to be discriminated among. If the method of scoring can improve the accuracy with which classification decisions are made, as occurred in this study, then the ramifications for education are obvious. With the increased availability of computer hardware and software, gaining additional accuracy will benefit those for whom the instrument was intended. The cost of this increased accuracy seems minimal when com— pared with an error in classification. Suggestions for additional research include a weighted composite of the three (or more) scoring methods in an attempt to "tease out" the maximum possible pre- diction. The inclusion Of additional groups in the research design to determine the effects of increasing the number Of variables entering the multiple discrimi— nant function equation might prove fruitful. Using the Chi-square scores as a starting point for the discrimi— nant function analysis as was done with the Kuder lambda scores in this research, is a third possibility. Probably the most obvious suggestion for addi- tional research would be to redesign the present study allowing for true cross-validation of all methods of 102 scoring. This would provide more accurate comparisons among all scoring methods as well as stronger evidence regarding the potential of any single technique or combi- nation Of techniques for scoring the Kuder. BIBLIOGRAPHY BIBLIOGRAPHY Anderson, T. W. An introduction to multivariate statisti- cal analysis. New York: Wiley, 1958. Barnard, M. M. The secular variations of skull charac- teristics on four series Of Egyptian skulls. Annals of Eugenics, 1935, §_(Part IV), 352—371. Bartlett, M. S. Multivariate analysis. J. Roy. Statist. SOC. SBPplo ' 1947, _9_' 176-197. Bartlett, M. S. The statistical significance of cononical correlations. Biometricka, 1941, 32, 29-38. Beech, H. R., & Maxwell, A. E. Differentiation of clini— cal groups using cononical variates. J. Consult. Bock, R. D. Multivariate statistical methods in behavioral research. Statistical Laboratory, University of Chicago, 1969. (Mimeo, chapters 1, 2). Box, G. E. P. A general distribution theory for a class of likelihood criteria. Biometricka, 1949, 36, 317. Box, G. E. P. The exploration and exploitation of response surfaces. Biometrics, 1954, 10, 16-60. Brogden, H. E. An approach to the problem Of differential prediction. Psychometrika, 1946, 11, 139-154. Brown, G. W. Discriminant functions. Ann. Of Math. Stat., 1947, 18, 514-528. Bryan, J. G. The generalized discriminant function: Mathematical foundation and computational routine. Harvard Educ. Rev., 1951, 21, 90-95. Burt, C. On the discrimination between members of two groups. In A. Lubin, Linear and non-linear discrimi— nating functions. Brit. J. Psychol. (Stat. Section), 1950, 3, 104. 103 104 Cattell, R. B. r and other coefficients Of pattern similarity. Psychometrika, 1949, 11: 279—298. Cattell, R. B. (Ed.) Handbook of multivariate experi- mental psychology. Chicago: Rand McNally and COL, 1966. Chappell, J. S. Multivariate discrimination among selected occupational groups utilizing self report data. Unpublished doctoral dissertation, Purdue University, 1967. Claringbold, P. J. Multivariate quantal analysis. J. Roy. Statis. Soc., 1958, 20, 398-405. Clark, J. A. Criterion Pattern Analysis: A method for identifying predictive item configurations. Unpub— lished doctoral dissertation, Michigan State Uni- versity, 1968. Clark, J. A. Program Predict. Undocumented Fortran program. Michigan State University, 1969. Clemans, W. V. An analytical and empirical examination Of some prOperties of ipsative measures. Psycho- metric Monographs, 1968, 14. Clemans, W. V. An index of new criterion relationships. Educ. and Psychol. Measmt., 1958, 18, l. Clemens, B. T. Discriminant function analysis of inven- toried interests among selected engineering groups. Unpublished doctoral dissertation, Purdue University, 1969. CoBabe, T. A. The Strong Vocational Interest Blank as a predictor Of success in engineering. Unpublished doctoral dissertation, Purdue University, 1967. Cochran, W. G., & HOpkins, C. E. Some classification problems with multivariate qualitative data. Biometrics, 1961, 11, 10-32. Cooley, W. W., & Lohnes, P. R. Multivariate procedures for the behavioral sciences. New York: John Wiley and Sons, 1962. Creager, John A. Discriminant analysis and its role in the classification Of airmen. Research report: Air Research and Development Command, Lackland Air Force Base, Texas, 1957. 105 Crego, C. A. A pattern analytic approach to the measure of modes Of expression of psychological differenti- ation. Unpublished doctoral dissertation, Michigan State University, 1966. Cronbach, L. J., & Gleser, G. C. Assessing similarity between profiles. Psychol. Bull., 1953, 50, 456-473. Draper, J. Personal communication on properties Of repeated measure designs with dichotomous dependent variables. Unpublished doctoral dissertation, Michigan State University, 1971. DuMas, F. M. A quick method Of analyzing similarity of profiles. J. Clin. Psychol., 1946, 2, 80-83. Dwyer, P. 8. Solution of the personnel classification problem with the method Of Optimal regions. Psycho— metrika, 1954, 19, 11-26. Dwyer, P. S. The detailed method of Optimal regions. Psychometrika, 1957, 22, 43-52. Findley, Warren. A rationale for evaluation of item discrimination. Educ. and Psychol. Measmt., 1956, ‘16, 175-180. Fisher, R. A. The use Of multiple measurements in taxonomic problems. Annals of Eugenics, 1936, 1, 179—188. Fisher, R. A. The statistical utilization of multiple measurements. Annals of Eugenics, 1938, 13, 376-386. Fisher, R. A. The sampling distribution of some statistics Obtained from non—linear equations. Annals Of Eugenics, 1939, 9, 238. Fricke, B. G. A configural-content-intensity item for personality measurement. J. Educ. Psychol. Measmt., 1956’ $9., 54’62. Fricke, B. G. A coded profile method for predicting achievement. J. Educ. Psychol Measmt., 1957, 11, 98-104. Gaier, E. L., & Lee, Marilyn C. Pattern analysis: The configural approach to predictive measurement. Psychol. Bull., 1953, 59, 140-148. 106 Ghiselli, C. E. Differentiation of tests in tests of the accuracy with which they predict for a given indi— vidual. Educ. Psychol. Measmt., 1960, 20, 675-684. Graybill, F. A. An introduction to linear statistical —— models. Vol. I. New York: McGraw-Hill, 1963. Greenhouse & Geisser. On the analysis of profile data. Psychometrika, 1959, 24, 95—111. Herzburg, P. A. The parameters of cross-validation. Psychometric Monograph Supplement, 1969, 16. Horst, P. Pattern analysis and configural scoring. J. Clin. Psychol., 1954, 13, 107—114. Hsu, Tse—Chi, & Feldt, L. S. The effect Of limitations on the number of criterion score values on the significance level of the F-test. Educational Research Journal, 6(4), 1959, 515-527. Jones, L. V. Problems of devising and selecting appropri- ate measurement tools. Am. J. Ment. Def., 1959, 64, 384-394. Jones, L. V. Some illustrations of psychological experi— ments designed for multivariate statistical analysis. The Psychometric Laboratory, University of North Carolina, December, 1960. (Mimeograph) Jones, L. V., & Bock, R. D. Multiple discriminant analy— sis applied to "ways to live" ratings from six cul- tural groups. Sociometry, 1960, 23, 162—176. Kelley, T. L. Statistical method. New York: Macmillan, 1923. Kirk, Roger. Experimental design: Procedures for the behavioral sciences. Belmont, Calif.: Brooks/Cole, 1968. King, P., & Norrell, Gwen. A factorial study Of the Kuder Preference Record Occupational, Form D. Educ. Psychol. Measmt., 1964, 24, 57-63. Kuder, G. F. Kuder Preference Record ggcupational, Form D Test Booklet. Chicago: SRA, 1956. Kuder, G. F. A comparative study Of some methods Of developing occupational keys. Educ. Psychol. Measmt., 1957, 11, 105-114. 107 Kuder, G. F. gpder Preference Record Occupational, Form D Manual. Chicago: SRA, 1961. Kuder, G. F. A rationale for evaluating interests. Educ. Psychol. Measmt., 1963, 22, 3-10. Kuder, G. F. Kuder Occupational Intergst Survey, Form DD, General Manual. Chicago: SRA, 1970. Kuder, G. F. Kuder Occupational Interest Survey, Form DD, Test Booklet. Chicago: SRA, 1966. Lee, M. C. Configural vs. linear prediction of collegiate academic performance. Dissertation Abstracts, 1957, Lee, M. C. Interactions, configurations and non-additive models. Educ. Psychol.Measmt., 1961, 22, 797-805. Lohnes, Paul R. Test space and discriminant space classi— fication models and related significance tests. Educ. Psychol. Measmt., 1961, 22, 559-574. Lubin, A. Linear and non-linear discriminating functions. Brit. J. Psychol. (Stat. Sect.), 1950, 2, 90-104. Lubin, A. A methodological study of configural scoring. U.S. Army Personnel Research Bureau, 1954, Note NO. 42, 13pp. Lunney, Gerald H. Using analysis of variance with a dichotomous dependent variable: An empirical study. J. of Educ. Measmt., 1970, 2(4), 263—269. Marascuilo, L., & Levin, J. Appropriate post hoc com— parisons for interaction and nested hypotheses in analysis of variance designs: The elimination Of type IV errors. A.E.R.J., 1970, 22, 397-421. Maxwell, A. E. Cononical variate analysis when the vari- ables are dichotomous. Educ. Psychol. Measmt., 1961, 22, 259-271. McQuitty, L. L. Elementary linkage analysis for iso— lating orthogonal and oblique types and typal rele— vancies. Educ. Psychol. Measmt., 1957a, 22, 207—229. McQuitty, L. L. Isolating prediction patterns associated with major criterion patterns. Educ. Psychol. Measmt., 1957b, 22, 3-42. 108 McQuitty, L. L. Job knowledge scoring keys by item versus configural analysis for assessing levels of mechanical experience. Educ. Psychol. Measmt., 1958, 22, 661-680. McQuitty, L. L. A method for selecting patterns to differentiate categories of people. Educ. Psychol. Measmt., 19616, _2—1' 85—94. McQuitty, L. L. Item selection for configural scoring. Educ. Psychol. Measmt., 1961b, 22, 925-928. McQuitty, L. L. Rank order typal analysis. Educ. Psychol. Measmto ' 1963’ E, 55-61. McQuitty, L. L. Single and multiple hierarchial classifi- cation by reciprocal pairs and rank order types. Educ. Psychol. Measmt., 1966, 2Q, 253-265. “bed, A. M., & Graybill, F. A. Introduction to the theory Of statistics. New York: McGraw-Hill, 1963. MQzltrison, D. F. Multivariate statistical methods. New York: McGraw-Hill, 1967. b”‘tbiser, C. I. Problems and designs of cross-validation. Educ. Psychol. Measmt., 1951, 22, 5-11. bTunnally, J. Psychometric theory. New York: McGraw- Hill, 1967. IPillai, K. C. S. On the distribution Of the largest or smallest root of a matrix in multivariate analysis. Biometrika, 1956, 22, 122. Pillai, K. C. S. On the distribution of the largest seven roots of a matrix in multivariate analysis. Biometrika, 1964, 22, 270-275. Pillai, K. C. S. On the distribution of the largest characteristic root of a matrix in multivariate analysis. Biometrika, 1965, 22, 405-414. JPillai, K. C. S., & Samson, P., Jr. On Hotelling's generalization of T squared. Biometrika, 1959, fig, 160-168. 1?orter, A. C. A chi-sqpare approach to discrimination among occupations usingian interest inventory. Technical report NO. 24. University of Wisconsin, Center for Cognitive Learning, Madison, Wisconsin, 1967. 109 Ra:<::, C. R. Utilization of multiple measurements in problems Of biological classifications. J. Roy. Statist. Soc., 1948, 22, 159-193. Rao , C. R. Advanced statistical methods in biometric research. New York: Wiley, 1952. Roy' , S. N. P-statistics or some generalizations in analysis of variance appropriate to multivariate problems. Sankhya, 1939, 4, 381. R0)?’ .. S. N. On a heuristic method of test construction and its use in multivariate analysis. Ann. Math. Stat. ' 1953' a, 220-2380 Roy , S. N. Some aspects of multivariate analysis. New York: Wiley, 1957. Ik:r§(, J., & Borgmann, R. E. Tests Of multiple independence and the associated confidence bounds. Ann. Math. Stat., 1958, 22, 491-503. FLCDy, J. Step-down procedure in multivariate analysis. Ann. Math. Stat., 1958, 22, 1177-1187. Rozeboom, W. W. The fallacy of the null-hypothesis significance test. Psychol. Bull., 1960, 52, 416-428. Ikulon, P. J. Distinctions between discriminant and regression analysis and geometric interpretation of the discriminant function. Harvard Educ. Rev., 1951, 22, 80-90. Rulon, P., Tiedeman, D., Tatsuoka, M., & Longmuir, C. Multivariate statistics for personnel classification. New York: John Wiley and Sons, 1967. Rulon, P., Tiedeman, D., Langmuir, C., & Tatsuoka, M. The profile problem: A methodological study of the interpretatiopvof multiple test scores. Cambridge, Mass., Educ. Research Corporation, February 1954. Schotzoff, M. Exact distributions Of Wilks Likelihood Ratio Criterion. Biometrika, 1966, §;, 347-358. £3chutz, R. E., & Baker, R. L. A factor analysis of the Kuder Preference Record Occupational, Form D. Educ. Psychol. Measmt., 1962, 22, 97-104. EStellwagon, W. R. An examination of the use Of linear and non-linear discriminant functions in the classifi- cation Of college students into academic groups. Dissertation Abstr., 1960, 22, 4435. 110 Stephenson, Richard. A new pattern analysis technique for the SVIB. J. Of Counsel. Psych., 1961, 2, 355-362. Stormes, Lowell, H. Discrepancies between factor analysis and multivariate discrimination among groups as applied to personality theory. J. Ment. Sci., 1958, 104, 713-721. Tat: suoka, M. M. The relationship between cononical corre- lation and discriminant ana2ysis; and ayproposal for ptilizing qpalitative data in discriminaptanalysEg. Cambridge, Mass.: Educational Research Corp., 1955. Tat; suoka, M. M., & Tiedeman, D. V. Discriminant analysis. Rev. Educ. Res., 1954, 22, 402-420. Tri<=xrndike, R. L. The problem of classification of personnel. Psychometrika, 1950, 22, 215-235. TIL(edeman, D. V. The utility of the discriminant function in psychological and guidance investigations. Harvard Educ. Rev., 1951, 22, 71-80. Tiedeman, D. V., & Bryan, J. G. Prediction of college field of concentration. Harvard Educ. Rev., 1954, 22, 122-139. Tiedeman, D. V., Bryan, J. G., & Rulon, P. J. The utility of the Airman Classification Battery for assignment of airmen to eight Air Force specialties. Cambridge, Mass.: Educ. Research Corp., June, 1951. 'Tiedeman, D. V., Rulon, P. J., & Bryan, J. G. The multiple discriminant function: A symposium. Harvard Educ. Rev. ' 1951' El, 71,950 Tilton, J. W. The measurement of overlapping. J. Educ. Psychol., 1937, 22, 656-662. Tintnes, G. A note on rank, multi-colinearity, and multiple regression. Ann. Math. Stat., 1945, 2Q, 304-308. Yrravers, R. M. W. The use of a discriminant function in the treatment of psychological group differences. Psychometrika, 1939, 2, 25-32. ‘lotaw, D. F., Jr. Methods of solving some personnel- classification problems. Psychometrika, 1952, 22, 255-266. 111 Weiss, D. J. A technique for curvilinear multivariate prediction. Dissertation Abstracts, 1964, 22(11), 4788. Welch, B. L. Note on discriminant functions. Biometrika, 1939, 22, 218-220. Wilks, S. S. Certain generalizations in the analysis Of variance. Biometrika, 1932, 22, 471-474. Zubin, J. The determination of response patterns in personality adjustment inventories. J. Educ. Psychol., 1937, 22, 401-413. Zubin, J. Nomographs for determining the significance of the differences between the frequencies of events in two controlled series or groups. J. Amer. Statis. Assoc., 1939, 22, 539-548. APPENDICES APPENDIX A DISCRIMINANT FUNCTION ANALYSES OF SET I AND SET II DATA 112 omma. memo.- «nae. mmmm. methumehumuo> amok. ammo.u Hemm. esm~.- mumflomuoee Houseman Hmos.n mooo. knee. mmmo.- methoehuthomo oemm.n ~mea.- mome.- Hmme.- methoherooe smhux mmmo.- memo. ance.- meeo.- mumhuuoeouoo e m m H mumo d mamm H uwm so commm munmwmz cofiuocsm ucmcwefluomwn Hooo. u o N u on omH.o¢H u «x moth» mo we~.o u e room Hooo. u o e u no Hmv.omm u «x moon» mo mom.o~ u m uoom Hooo. u m o u no mmo.mme u «x oomuu mo moo.om u N Doom Hooo. u.m m u no mma.amm u Nx menus Go wae.ae u H Doom mm.hm u whom ems has. n menses m.xahz memo.~ u moths mma m NH b ca mcmwumcfiumum> moomuz ms me me mm Hm homeomhmee Hmoemmgo moomuz Hm mm oea ma NH methofluumeoom mmamuz m ma NH mm ea methoflerome smhux mmmauz ma ma ea NH oea mumhuuosouoo mmomuz muo>a sea momma mousse moooa masouo m moanmfium> coaumOAHm>lmmouu so mcowumowuflmmmao Hmuoa mo Honesz .mumo m mama H pom muoom 0» com: a mama H uom sou cohuocsm ucmcwseuomfloun.H pom 113 moHo. Hoem.u omen. Hmme.u mumHuuoEouoo Hmmm.u eOHN. moom.- mmem. whoxuoz ommo HmHoom HeeH.u meme. emmw. memm.u mumomohom eeeH. mmem.u mmHm. mHmo. moHemromz ooze omHm. Heme. mmem.u eMHe. mumHooHoeosmo .eHHO e m N H upon 4 mamm HH now so Ommmm munmflmz cofiuocsm ucmcHEHHomHo Hooo. u m m u no oom.mem u «x women no wHe.OH u 4 room Hooo. u o e u no moo.emm u «x more» mo www.mH u m uoom Hooo. u o o u we mom.meo u «x moon» mo weo.- u N uoom Hooo. u o m u no mon.oHHH u «x moon» mo woe.ke u H poor em.mHm u emmm owe «me. u mohemH m.xHHz mm~m.e u moths emH RH m HH m mumHnuonuoo moomuz mH HRH o H mm whorhoz ommo HmHoom mommuz e o eeH e o automouom mmeHuz m m o omH H moHeorooz ooze momHuz eH mm m 0 men momHooHorosmo .eHHO mommuz muooa some .umhome ouoaa who .eHHOa mmsouw m mmHQmaum> m cowumowam>lmmono so mOOADOOAMHmmmHU Hmuoa mo umnEsz .mumo m OHmm HH pom ouoom 0» owns 4 MHmm HH now how eoHoocom nemcHeHuomHoun.HH pom APPENDIX B A PRESENTATION OF THE NON CROSS-VALIDATED CLASSIFICATION PREDICTION FOR THE CHI- SQUARE, KUDER LAMBDA AND DISCRIMINANT FUNCTION TECHNIQUES .mcsum ucmmmum on» scum OOUH>OHmt¢ .mcsum hmmH m.umuuom EOHH OOOH>Oumr 114 mmHm. moom. mums. ommh. mmhm. ammo. omHh. Hmuoa ommb. omHm. omom. oomm. omoe. mmmm. omme. mumfiuumfioumo HHmn. Hom5. mnmm. mnmm. omHm. mnmm. momm. mumxuoz mmmo HMHoom hmvm. momm. mmom. mmom. momm. mmev. nHmm. mumummuom boom. nmmm. mmmh. mmmm. mmhm. nmmm. «mum. mOHcmnomz ou5¢ ommm. ommb. comm. ovqm. ovmm. oemm. omom. .Honommm HMOHGHHU HH uwm ommm. Hmom. onmm. mmvm. mmvm. ovmm. ovoh. Hmuoa oomm. omnn. ooam. ommm. ooom. ommm. omom. mGMHHmcHHmum> hmom. HHHN. sHHm. mnmm. mome. mumm. Nmom. mumammumse HOOHmmzm mmvm. mhmm. Noam. «mow. mmmm. «New. much. mQMHOHHpMHoom bmmm. mmnm. mmnm. hHmm. mmnm. vmmv. mmem. mcMHOchome woulx some. mmum. mvmm. mmmm. nmmm. mva. mOHm. mumHquEoumo H now «am «ad «tm «rd tam «m «d MHmm HHmm HHmm HHmm HHmm HHmm HHmm cofluocsm ucmcHEHuOmHO MOQEOH HOOOM mumsomnwcu .mmsvfisnomu cOHuOcsm ucmcHEHuomHo can .OOQEMH umosx .mumsvmlHno on» How mumo m HHOA can 4 HHms muoom 0» Oman can 4 HHmn EOHH OO>HHOO mucmHms map com: comma mumc HH now can H pom cH mOHmE OOHHHmmOHO mHuomuuoo mo mcoauuomonm may HO OOHumucmmmum c APPENDIX C VARIANCE COVARIANCE MATRICES OF THE DICHOTOMOUS CROSS-VALIDATION SCORES FOR SET I AND SET II DATA APPENDIX C VARIANCE COVARIANCE MATRICES OF THE DICHOTOMOUS CROSS-VALIDATION SCORES FOR SET I AND SET II DATA Set I M1 M2 M3 M4 M1 .1517 .0180 .0007 .0292 M2 .2297 .1022 .1136 M3 .2253 .1257 M4 .2402 Set II M1 M2 M3 M4 M1 .1607 .0258 -.0103 .0040 M2 .2185 .0635 .0427 M3 .1861 .0646 M4 .1502 M1 = Pattern Analysis M2 = Chi-square M3 = Kuder lambda M4 = Discriminant Function 115 B wifiifififlfihfiflifllflmflfiflflfifiljilflfiflifllfii‘lfll’fl“