MULTIPLE AGREEMENT ANALYSIS Thesis {or ”an Degree of D“. D. MICHIGAN STATE UNIVERSITY Peter Wing Hemingway .1961 .4. .9 ..b~ \ ‘ . P..|i-\‘ ' ' ~54. ‘ ‘- "‘C 'I‘zr"' _ 14-12.‘f-Li mm mm WW 3 1293 10714 7567 This is to certify that the thesis entitled MULTIPLE AGREEMENT ANALYSIS presented by Peter W. Hemingway has been accepted towards fulfillment of the requirements for £2.13; degree in my LIBRAR Y Michigan Star- University MSU LIBRARIES “ RETURNING MATERIALS: Place in book drop to remove this checkout from your record. FINES will be charged if book is returned after the date stamped below. MULTIPLE AGREEMENT ANALYSIS by Peter Wing Hemingway AN ABSTRACT OF A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Psychology 1961 ABSTRACT MULTIPLE AGREEMENT ANALYSIS by Peter Wing Hemingway This study reports the development and application of a method for the objective classification of objects on the basis of their common characteristics or patterns. A review of the literature reveals that one of the major problems of pattern analytic methods is the large number of potential patterns. The goal of such methods is to isolate the patterns which are most meaningful or useful. In those methods where no external criteria are available for determining the utility of obtained classes, analysis tends to yield a proliferation of overlapping classes, with no basis for selecting the more relevant or meaningful ones. The method presented here offers a strict criterion for the termi- nation of classes based upon the maximization of information contained in each class. Information is defined as the product of the class size (number of members) times the pattern size (number of common character- istics). This criterion achieves the purpose of both maximizing the amount of information accounted for by each obtained class and mini- mizing the number of classes obtained. This method, multiple agreement analysis, is largely derived from McQuitty's 1956 paper on agreement analysis, and the principles of taxonomic classification. A theoretical framework is presented, and the computational procedure outlined. This procedure, deve10ped for computer use, is basically an iterative procedure of reductive matrix partitioning. Beginning with a matrix of n persons recorded as either possessing or not possessing each of r characteristics, successive sub- Peter Wing Hemingway matrices are extracted. These submatrices are of maximum product size, each having identical rows (characteristics) for all class subjects. In order to investigate the ability of the method to yield useful results, a set of 20 senators with a predetermined class structure was analyzed, using their votes on 32 issues as the characteristic set. Results indicated the reliability, meaningfulness and utility of the obtained classes satisfied the theoretical claims for the method. Application of the method to the full body of senators, using the voting records of 88 senators on 95 issues, resulted in a hierarchical classification structure. This consisted of 15 major classes, of which seven contained only two members each. The eight larger major classes ‘were further broken down into subclasses, the larger of these were further divided into subsubclasses. Of all 44 obtained classes, which utilized 72% of the available information, not one contained both a Republican and a Democrat. Further, none of the subclasses contained members of more than one major class. Prediction of the passage or failure of 96 additional issues on the basis of the votes given by a senator from each of the eight larger major classes gave 88% correct prediction. While the method in its present form is useful as a classification technique, restrictions of the computational procedure not required by the theoretical assumptions imply that results obtained are conserva- tive approximations of the "true" class structure existing in the pap- ulations studied. Further investigations as to the relative value of this method compared to other methods is suggested, as well as potential modifications of the computational procedure for particular classification problems. MULTIPLE AGREEMENT ANALYSIS by Peter Wing Hemingway A THESIS Submitted to Michigan State University . in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Psychology 1961 ‘L in appendices. received. Filmed as UNIVERSITY MICROFILMS, INC. . PLEASE NOTE: Light and dark, 'b’l‘t’trréa type ' 1 ACKNOWLEDGMENTS The author wishes to express his thanks to the members of his Guidance Committee, Dr. C. Frost, Dr. L. Katz, and Dr. C. Wrigley, and especially to his Chairman, Dr. J. S. Karslake, for their help and guidance during the course of this study. A special word of appreciation is extended to Dr. Wrigley for his critical readings of the many drafts of this manuscript and his many helpful suggestions both as to style and content. Finally, the author would like to acknowledge his appreciation of the many years of friendship and inSpiration given to him by Dr. J. S. Karslake, to whom this thesis is affectionately dedicated. ii TABLE OF CONTENTS ACKNOWLEDGEMENTS . . . . . . . . . . . . . . . . . . LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . Chapter I. II. III. Iv. V. REFE RENCES O O O O O O O O O O O O O O O O O O O O O O O APPEmIXES O O O O O O O O I O O I O 0 O O O O O O O O O MULTIPLE AGREEMENT ANALYSIS . . . . . . . . . . . . . . Introduction and Background . . . . . . . . . . . . . . Patterns in Psychology . . . . . . . . . . . . . . . . “man 0 I O O O O O O O O I O I O O O O O O O O O O 0 Agreement Analysis . . . . . . . . . . . . . . . . . . Theoretical Considerations . . . . . . ... . . . . . . Multiple Agreement Analysis . . . . . . . . . . . . . Computational Procedure . . . . . . . . . . . . . . EMPIRIML IWESTIQTION O O I I O O O O O I O O O O The Assumed Class Structure . . . . . . . . . . . . . The Obtained Class Structure . . . . . . . . . . . . . The Reliability of Multiple Agreement Analysis . . . . The Effects of Alternate Solutions . . . . . . . . . . The Validity of Obtained Class Patterns . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . AN EMPIRICAL.APPLICATION OF MULTIPLE AGREEMENT ANALYSIS Data and MethOd O O O O O O O O O O O O O O O O O O 0 Results 0 O O O O C O I O O O O O O O O O O O O O O smary O O O O O O O O O O O O O I C O O O O O O O 0 CONCLUSIONS AND DISCUSSION . . . . . . . . . . . . . . iii page ii iv l6 l6 18 26 28 36 37 39 44 52 61 66 68 68 7o 92 94 102 105 Table 1. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. LIST OF TABLES The Response Matrix . . . . . . . . . . . . . . . The Original Agreement Score Matrix . . . . . . . Multiple Agreement Analysis: Class Structure Obtained The Assumed Class Structure of 20 Senators . . . Obtained Classes, Group A Issues . . . . . . . . Obtained Classes, Group B Issues . . . . . . . Comparison of the First Six Obtained Classes with the Assumed Classes: Group A and Group B Issues . . Obtained Classes: Double-Entry Method, Using 32 Random Response Columns to Classify 20 Subjects . . Obtained Classes: Single-Entry, Group A Issues . . Obtained Classes, Arbitrary Pairs . . . . . . . Cumulative Amount of Information (Product) Utilized under Various Conditions . . . . . . . . . . . . . . Obtained Classes, 20 Senator Group . . . . . . . . . Response Patterns for Each of the First Seven Classes Pattern Scores for the 20 Senators on Each Key . . Pattern Scores for the Remaining 68 Senators . . . . Obtained Structure of the U. 5. Senate . . . . . . . Obtained Class membership, U. S. Senators . . . . Response Patterns, First Eight Classes . . . . . . . Number of Omossions for Each Senator . . . . . . . Disagreement Score for Each Senator on Each Pattern . Senators Voting For or Against Each Issue . . . . . The 15 Issues Differentiating Class 1 and Class 2 . Responses and Issues Uniquely Defining Major Classes Issues Upon Which Subclasses Disagree Absolutely . . Predicted and Observed Voting, 96 Group B Issues . iv Page 31 31 38 41 46 48 51 54 56 58 6O 62 63 65 71 72 74 76 77 81 83 84 91 ”By the classification of any series of objects is meant the actual, or ideal, arrange ment together of those which are like, and the separation of those which are unlike; the pur- pose of this arrangement being to facilitate the operations of the mind in clearly conceiv- ing and retaining in the memory, the characters of the objects in question." T. H. Huxley. An Introduction to the Classifi- cation 9; Animals. London, John Churchill & Sons. 1869. CHAPTER I MULTIPLE AGREEMENT ANALYSIS Introduction and Background This paper reports the deve10pment and application of a multi- variate classification technique designed to isolate significant patterns in unordered data, such as individual item responses. The technique is based upon McQuitty's original method of agreement anal- ysis (1956), with several modifications designed to provide objective criteria for termination of classes and sequential reduction of the response matrix. It is a method best suited for electronic computers, due to the lengthy computations required, and has been programmed for MISTIC, the computer at Michigan State University. The development of objective pattern analytic methods is a comparatively recent phenomenon in psychological research, although the concept of patterning has been utilized by many fields for a much longer period. Even in the ancient histories, we find that Aristotle spoke of patterns in his classification of animal life, and it is in this area, animal classification, that we find the most formal classi- fication system based upon patterns of characteristics. As Cain (1954) illustrates in his summary of the chapter defining the concepts of taxonomic classification, the "definition" of a particular Species may be based either upon one or more unique (to that specie) characteris- tics or to a configuration of non-unique characteristics. He goes 2 on to explain that there are recognized species which have no charac- teristics which, in themselves, are definitive, yet which provide, in combination with other such characteristics, precise definition of the specie. The concept of patterns as significant indicators of relation- ships has been a part of many human activities, both scientific and non-scientific. Philosophers and scientists in many fields have dis- cussed and defined patterns of all sorts, from the prehistoric observers who perceived mythological figures in the patterns of the stars to the present day sociologists who write of patterns of delin- quency. The usual methods for isolating patterns have tended to be subjective, arbitrary, and selective observational techniques, where the observers usually began with a particular characteristic, searched for another which would give increased precision to their model, and, over a period of years, they would realize a fairly elaborate structure defining their criterion, whether it was resemblance to a mythological beast or delinquent behavior on the part of a subject. The problem with patterns derived by such methods was their difficulty in remain- ing invariant; different observers, by selecting different subsets of the characteristics, obtained different results. Also, the development of statistical methods which examined only linear relationships pro- duced such powerful advances in relational and correlational analysis that the characteristics which had been considered as elements in patterns were now studied either individually, or in dimensional groups, with the result that those characteristics dependent wholly on configural (nonlinear) effects were ignored. It is only recently that linear methods have been sufficiently analyzed and refined so that some investigators have felt it worthwhile to go back and attempt to bring patterns into consideration of configural effects as objective, meas- urable phenomena, and the study of patterns has now become reasonable ‘within the framework of measurement, with statistical and.mathematica1 methods for their analysis being feasible. Patterns in Psychology The recent increase of interest of psychological investigators in patterns, or interactions, among sets of characteristics or varia- bles appears to arise from two sources. One is the increasing feeling that the refinements of the standard linear techniques for studying multivariate relationships have become increasingly complex and mathematically sophisticated, and so are attempting to analyze data in much more detail than the data themselves meaningfully contain. Thus, the focus has switched from analysis of the "real" data to analysis of the mathematical models which are hypothesized as isomorphic to the phenOmena which give rise to the "real” data. The other and closely related reason for interest in patterns is the feeling that linear models have reached an asymtote in their ability to account for multi- variate (and univariate) relationships. Essentially, the present linear techniques are sufficient to determine the linear relations within a set of data. Further advances must therefore necessarily be accompanied by either more precise measurement or by new methods which explore more than the linear effects, or, most desirably, both. This interest in "patterns" in psychology has stemmed primarily from the focus of the clinical psychologist on configurations suppos- edly representing the complex interrelations of differing aspects of "the whole person" in making his subjective evaluations and predictions. The desire of the clinician to utilize objective (i.e. "scientific") measures and the failure of available linear models to perform successfully in clinical situations has done much to create the current interest in objective configural methods of data analysis. The general area of configural analysis has been rather widely studied in terms of profile analysis, but these techniques differ from pattern analysis in their dependence upon linear (dimensional) variates for their starting point. That is, all subjects are measured on a number of tests (variables) and the similarity of their profiles are examined, using one or more combinations of their profile measures, such as shape, level, or scatter, to compare individuals and groups. Such methods, while of considerable interest, involve many assumptions not required by the method presented here, and more apprOpriately may.’ be considered as complex non-linear multivariate techniques. ' Thus, pattern analysis, in its most general form, is an attempt both to remain more nearly at the data level and to allow non-linear relationships to be eXpressed if they exist. As McQuitty (1957a) has pointed out, there are two basic modes of pattern-analytic methods; the cumulative and the reductive. The cumulative approach, as typified by the studies by Lubin and Osburn (1957), is the more traditional in form; the object is to determine those patterns of response which are optimally related to an external criterion for the group under investigation. Patterns are built up serially, beginning with that item which best predicts the criterion, pairing this with every other item to find the Optimal triad, etc. The procedure terminates when addition of a further item does not further increase the predictive power of the pattern. The 6 features of this method which tend to reduce its effectiveness are the Vdependence upon an external criterion for determining the optimal solutions, the unitary addition of items which ignores any conjunctive effects of item pairs or larger groups, and concentration upon a single optimal pattern or set of items, neglecting the possibility that different persons may be Optimally related to the criterion in terms of different sets of items. The reductive method is the opposite of the cumulative; it begins with an individual response pattern covering all items of the test, and reduces this to one or more patterns of less than all the items by eliminating those items which do not have identical responses for a person or persons grouped with the initial individual. The advantages of this method are that combinatorial effects are retained, different patterns may be realized for different individuals or groups, and the procedure may be used either with or without the inclusion of external criteria in the analysis. One of the major difficulties inherent in any reductive method which does not utilize external criteria for selecting patterns is the extremely large number of possible patterns which may exist. If the items are binary, such as true-false, agree-disagree, etc., there are 2n possible patterns for n items. If the items are multiple-choice with k alternative responses, there are kn possible patterns. In both cases it is assumed that the available responses are mutually exclusive; if not, the possible patterns are further increased. For example, if a true-false item can be reaponded to by checking either, both, or neither alternative, it is equivalent to a multiple-choice item with four alternatives, thus capable of yielding 4n possible patterns. Thus 7 a test of ten such items could contain 410 or 1,048,576 patterns. It is figures like this which have made this type of analysis rather a forbidding task. The alternative approach of the cumulative method without use of a criterion leads to an even larger class of possible patterns. Under this approach, it would be possible to classify the subjects into groups giving identical responses to the first item, then further classify each of these groups on the basis of their responses to the next item, and so forth until either the items or the subjects are exhausted. Presuming that all items were utilized, there would be again kn possible patterns. But, as the order of the items affects the composition of any particular group, and there are n! possible order- ings, there would be k“.n! possible patterns in all. Thus, the cumula- tive method becomes even less attractive than the reductive method when no criterion is available for determining the order of the items. The purpose of any classificatory system which is independent of external criteria is to place together individuals or groups which are most similar, and to separate individuals or groups which are most dis- similar. Using patterns to define groups, it is evident that, for a set of subjects and a set of responses of approximately the same (finite) magnitude, there will exist many more possible patterns than subjects. Similarly, it is unlikely that more than a few subjects will possess identical patterns of response on the complete set of items. Thus the use of the complete response set usually gives little classi- fication beyond the pair level. However, the elimination of responses and the corresponding reduction of possible patterns allows an increase in the size of the groups. This procedure may be followed, sequentially, building up larger and larger classes which are differen- tially defined by fewer and fewer responses. This is.the approach now used in taxonomy--the science of biological classification. As Cain (1954) points out in considerable detail, species are differentially defined on the basis of a compara- tively large number of morphological characteristics; some subset of these characteristics is used to define the genus, and still smaller subsets define the higher levels, such as family, order, etc. It should be noted that this system allows different sets of character- istics to define different groups on the same level, but does not allow for cross-classification of individuals or groups. The taxonomic method represents a culmination of centuries of study which, while often fragmentary and subjective in its approach, has finally yielded an objective and comprehensive classificatory system. The one principal advantage of this system has been the selection of certain "marker" characteristics for the definition of classes (i.e., the inability of different species to reproduce when crossbred). Using such markers, it then becomes a comparatively simple task to list other defining characteristics of already delimited classes. The current problem in taxonomy (aside from frequent dis- agreements as to appropriate "markers") is in develOping the system below the Specie level. Here, where markers have not been determined, taxonomy is beginning to concern itself with analytic methods of classi- fication, eSpecially objective methods of isolating predominant patterns of characteristics (Cain, 1954). One other field which is becoming intensely concerned with objective classification methods is in the area of information classification, such as library and museum cataloging. This area differs from the taxonomic in that cross-classification is not only allowed but highly desired. The classification of material possessing many characteristics, where the inclusion of all relevant material under any specific characteristic is essential and yet simplicity of the system is required, becomes a highly challanging task. The system presented by Perry and Kent (1957) is one of the first attempts to present a comprehensive theory for such a system, and yet the method proposed is surprisingly similar to the method developed by TOOps (1948) for studying patterns of characteristics in the psychological area. In the reductive methods deve10ped in the psychological area, provision is sometimes made for cross-classifications, so that subjects classified in'a particular group on the basis of one set of responses may be further classified with another group of subjects on some other set, even though the reaponses defining the two patterns may be either distinct or overlapping. These two types of classification involve rather different assumptions. The hierarchical type, which does not allow cross-overs, assumes that the placement of a subject into the first (lowest) level of classification is the terminal point for subject classification. The higher levels are realized by the combi- nation of the lower levels, each first-level class being considered as a unit, and these classes then being the "subjects" whichare combined by the method at the higher level. In the methods which allow cross— classification, a rather different basis is utilized for classifi- cations beyond the first level. The individual subjects are in effect released from their initial classes, and allowed to form new classes on 10 the basis of other patterns. The hierarchical systems thus require that higher-level classes be characterized by patterns made up of sub- sets of the characteristics forming the patterns of the first level classes; while the cross-class systems are more general, later classes being characterized by patterns consisting of 222 different subset of the available characteristics. The primary problem with such cross- class methods is the development of systematic methods for searching for appropriate subsets without returning to previously utilized sub- sets. Another problem is the reporting of such a complex classificatory scheme. We might summarize existing pattern—analytic methods at this point before we turn to a consideration of the experimental evidence as to their value. The most widely used methods have been the cumulative, primarily due to the comparative ease with which the patterns most highly related to the criterion can be isolated. The reductive methods have been more extensively developed in terms of the number of tech- niques (see for example, McQuitty, 1954a, 1956, 1957c). There are two main reasons for this. First, the freedom from dependence upon a criterion offers more alternative approaches to the selection of appro- priate patterns, allowing the techniques to be treated purely as classificatory systems, with no requirement that the classes realized be related to any specific criterion, the assumption being that the classes are related only to some unknown one. Secondly, the methods are usually evaluated on a logical rather than an empirical basis. Hence different methods can be easily constructed to handle specific logical constructs, without actually putting them to empirical tests as to their comparative efficiency in predicting any further relationships. ‘ 11 McQuitty (1954b) has developed a number of schemes for the empirical classification of persons (and/or stimuli) in such a way that configural similarities and differences are the basis for defining classes. These procedures are held to be useful due to the fact that a given response may have different meanings in different contexts. These methods characteristically provide a hierarchical classification structure, so that any attempt to use them in prediction provides the opportunity of finding the level of the hierarchy which minimizes errors of prediction. Jenkins and Lorr (1954) have used methods similar to those of McQuitty's with the exception that a priori configurations serve as the basis for classifying the members of a sample. Meehl (1954) has devised an example in which two dichotomous items each correlated zero with the criterion (and hence the multiple correlation with the criterion was also zero), but such that when all four response configurations are considered, perfect prediction of the‘ criterion is possible. This has been referred to as the "Meehl paradox." However, the paradoxical aspects of this situation were removed when Horst (1954) showed that appropriate coefficients a1, a2, and ‘12 could be found for the polynomial T a: a0 + 31x1 + a2x2-+ allexz such that the criterion, T, was predicted perfectly from the two items, x1 and x2 (where the criterion, T, and the items each have possible values one and zero). A similar form of ”configural scoring" has been used by Stouffer et a1 (1952) in an attempt to increase the reproduci- bility of Guttman scales. Here, items were grouped into clusters of 12 two or more, and each cluster was scored as a single item according to the "pattern" of response to the cluster as a whole. A solution and generalization of the Meehl paradox is also possible in terms of elementary probability theory. If a set of items, X1, . . '_Xj’ . . . Xt, each assumed to have affinite number of possi- ble response alternatives) are each unrelated to the criterion, T, so that P(T‘xj) a P(T) for all j, then we have the situation in which all of the item-criterion correlations are zero. In other words, the criterion is pairwise independent of each and every item. However, pairwise independence does Egg imply mutual independence. That is, although pairwise independence may hold, it is not necessarily true that P(Tlle, . . . xlr) a P(T) for all the subsets (jl, . . . jr) which may be taken from the set of item subscripts (1, 2, . . . t) with r taking on values 2, 3, . . .'t. For the two-item case, suppose we have x1 and #2 and that we wish to predict the criterion, T. Then, if P(T‘xl) a P(T) and P(Tlxz) a P(T), the correlation of each of the items with the criterion will be zero. But it does not follow from this that P(Tlxl and x2) - P(T). This two-item situation is one of the cases with which Meehl (1950) dealt in his first discussion of the "paradox.” The above discussion can be summarized in the following way: Pairwise stochastic independence of each item with the criterion implies zero correlations of each item with the criterion and hence a zero multiple correlation. However, pairwise independence does not imply that the items and criterion will be independent when we consider pairs of items in relation to the criterion, triplets of items in relation to the criterion, etc. 13 It is of interest to note that Feller believes ”practical examples of pairwise independent events which are not mutually indepen- dent apparently do not exist," (Feller, 1957, p. 117). In other words, Feller doubts the existence of actual data such as those repre- sented by the extreme case of the Heehl paradox. However, whether the relation of predictors to criterion can be enhanced by considering ”higher order" dependence for a given set of data must be determined empirically. Perhaps the clinical psychologist's insistence on con- sidering the "whole person" or the "configuration of traits" displayed by the individual is a reflection of such higher order dependence. Using Horst's solution for the Meehl paradox, Lubin and Osburn (1955) developed their methods for predicting a quantitative variable from response patterns. Briefly, the procedure is as follows: for each of the 2t configurations obtainable from a t-item test (in which the items are dichotomous), a corresponding mean on the criterion is obtained, i.e., the mean criterion value is calculated for each group of persons giving exactly the same response configuration. The result is a set of 2C criterion means which is designated the configural scale. One value on the configural scale is then associated.with each of the 2C reaponse patterns. The predicted value for an individual giving a particular response pattern is the value on the configural scale corresponding to that pattern. Rao (1948) has given a general proof of the ability of the maximum likelihood solution to produce the minimum number of misclassi- fications, whether the predictors are quantitative or qualitative. Lubin and Osburn (1955) have shown that the least squares solution is , 14 equivalent to the maximum likelihood when the distribution of criterion scores within each re5ponse pattern is normal. The empirical studies which have compared configural methods with linear methods have produced conflicting results. Better predic- tion has been claimed using the pattern approach by Meehl (1954), Saunders (1955) and Lubin and Osburn (1955), while the linear (multiple regression) methods have been equally as good or better predictors in the studies done by Bell (1957), Lee (1954), and Ward (1954). An additional point of confusion in evaluating configural methods is due to differences inherent in the reductive and cumulative approaches. The cumulative methods, such as Lubin and Osburn's, focus upon the maximization of predictive power (hence the necessity of an external criterion), whereas the reductive methods, such as many of McQuitty's, are primarily concerned with classification of the subject based upon the total set of available information (item reSponses). Such classification methods may or may not yield predictions as effi- cient as either cumulative or linear methods, depending upon the criterion chosen and the level of classification being utilized. Configural methods, which search for non-linear variable rela- tions, are generally at a disadvantage in empirical comparisons with linear methodsf~because of the much greater number of free parameters. Thus, unless the number of subjects is very large, the greater suscep- tibility of the non-linear methods to shrinkage on cross-validation tends to weaken the comparative effectiveness of these methods. The method tb be presented in this paper is of the classifi- catory, reductive type. It provides for, but does not require, 15 cross-classification and is based upon a theoretical view of organisms as possessors of traits which are not necessarily linearly related, but which are so related that type concepts (in terms of the organisms) can be meaningfully examined regardless of the linearity or lack thereof in the trait relationships. CHAPTER II METHOD The computational procedure to be presented in this section is based upon McQuitty's original paper (1956) on Agreement Analysis. MCQuitty's method will be discussed in some detail in relation to the method and theory employed in the present study. Agreement Analysis While McQuitty and several others have presented numerous special techniques for classifying subjects on the basis of patterns of item responses, McQuitty's (1956) paper on Agreement Analysis proposes a procedure which is both general and comprehensive. The basic postu- late of the method is "that there are various kinds of underlying psychological structures or predispositions (not just dimensional ones), which result in patterns of responses." (p. 7) These patterns are then the expressions of the particular classes or categories of subjects in the population. This implies that types, as defined by the classes, exist and are determinants of differential behavior (i.e., responses). The general method of agreement analysis was itself based on Zubin's (1938) definition of the agreement score as a measure of the similarity between subjects. McQuitty uses the agreement score as the tool for combining subjects into classes, adding a correction factor to correct for the amount of agreement by chance on irrelevant responses. This correction factor, while necessary in agreement analysis, will be 16 17 shown to be unnecessary in multiple agreement analysis by modifying the order in which classes are formed. The method prOposed by McQuitty in this 1956 paper can be briefly described as a hierarchical sequence of combining smaller classes into larger and larger classes based on the magnitude of the corrected agreement scores. The result is a complete system of classes, from individuals to (potentially) one final class consisting of all subjects. Its basic procedure is that of combining that pair of individuals who have the highest corrected agreement score, recomputing the agreement scores between this two-class and all remaining individu- als, and repeating this procedure, treating each two—(or larger) class as a new individual. Thus, at any particular point in the process, the next class may be formed either by combining two classes of the same size or a larger class with a smaller one. In the ideal, or at least the simplest, situation, the method would, beginning with N subjects, yield in sequence N/Z two-classes, N/4 four-classes, and so on until there would be one N-class. This approach to agreement analysis has two obvious short- comings, both of which have provided the basis for multiple agreement analysis. The first, which is hardly a fault of the method, but rather of the inability of humans, is that the results are too complete and comprehensive. If an investigator is concerned with the relation between classes and some external criteria, he may be forced to compare more classes than he originally had subjects in order to determine which level (n-classes) of classification is best differentiated by each criterion, and then face the possible problem of having different levels or classes most meaningfully related to different criteria. 18 Obviously, a method which yielded a more limited number of classes would be helpful, but only if the limitation could yield the poten- tially meaningful classes, while suppressing those which were of less value. The second shortcoming of agreement analysis is its hierarchi- cal nature. While the method as McQuitty presents it in detail and even illustrates with an example is strictly hierarchical, one sentence points out the possibility of a non-hierarchical system of analysis and indeed, in combination with another statement, provided the basis of multiple agreement analysis. McQuitty states that ”responses which do not fit these patterns can be used later to reclassify the subjects in terms of less predominant patterns if it seems worth while." (p. 9). Immediately before this sentence he has defined predominant patterns as those which include the greatest possible number of responses. These two concepts, the maximization of the number of responses in a class, and the use of previously unused responses for reclassification, will be shown to provide both a theoretical and computational basis for multiple agreement analysis. Theoretical Considerations The basic assumption of any method which classifies subjects into distinct classes is that such groupings allow simplification of the subject set by reducing it from an n-size group of subjects to an r-size (r <14) (29) (n) (12) (u) (12) 66 Differences between Democrats and Republicans are not as pro— nounced on the other keys, although in each case a Median Test showed highly significant differences between the two groups. While it might be possible to derive efficient predictors of the four assumed sub- groups on the basis of differential weighting of all seven of the keys for each subgroup, such a procedure would imply that these seven groups were the major groups in the entire Senate. Rather than accept this structure, based on the analysis of a selected set of Senators and only 32 items, it would appear more reasonable to examine all Senators on the largest possible set of items and see what sort of structure this analysis would yield. The next chapter presents the results of such an analysis. Summary Empirical investigations conducted to ascertain the reliability and validity of Multiple Agreement Analysis have been reported. The results of these investigations have shown that this method yields sub- ject classes which are both reliable and meaningful. Also, the response patterns defining classes can be utilized both in differentiating the obtained classes and in predicting class membership of new subjects. Thus far the method has been given as a logical system based on the stated theoretical assumptions of Chapter II, with some empirical evidence of its capabilities and shortcomings presented in this chapter. A major omission to date has been any examination of the content of the issues which have differentially defined the obtained classes. We have found that different items seem to be effective in making different dis- criminations, but thus far no mention has been made of what these items 67 mean in terms of content. The interested reader will find the issues summarized in Appendix B, but rather than looking at the issues used in this chapter, this topic will be discussed in the following chapter in a more complete analysis of the structure of the entire Senate. The results of the investigations made up to this time have illustrated several properties of the method. One of these is that the use of the maximum amount of information (the double-entry method) gives the more stable and meaningful class structure. For this reason the double-entry method will be utilized in the study presented next. Also, as the analysis by subjects gives more discrete subject classes than analysis by items, the next response matrix to be used will not be transposed. Finally, as the use of alternative pairs as the starting point of an analysis has little effect on the class structure, the use of the pair with the highest agreement score will continue to be used as the starting point, with increased confidence in the stability of the resultant structure. The next chapter will report on the results of the application of Multiple Agreement Analysis to a more extensive set of issues for the full U. S. Senate. Based upon the results obtained from the anal- yses reported in this chapter, the double-entry method will be used and the analysis will be by subjects, for the purpose of obtaining the clearest classification of the subjects used in the study. CHAPTER IV AN EMPIRICAL.APPLICATION 0F MULTIPLE AGREEMENT ANALYSIS bIn this chapter results of a full-scale study of the structure of the United States Senate of the 83rd Congress will be reported. The purpose of this investigation is to determine, within the capabilities of the method of Multiple Agreement Analysis, the group structure of this Senate. Another purpose is to establish further the abilities and limitations of the previously presented analytic method in yielding meaningful and reliable classifications. The procedure will be shown to simplify complex behavior, such as legislative voting, and to give fuller understanding of the factors which influence it. Data and Method The subjects were the 88 Senators of the 83rd Congress who were in office during both sessions (1953-1954) of this Congress. The adequacy of this group to be considered a representative sample of U. S. Senators over a longer period of time is debatable. In view of the rapid changes which have occurred during this century, it is doubt- ful that any one Senate may be considered a representative sample for any larger set of Senators. On this basis, this particular Senate will be treated as a discrete pOpulation, and no attempt will be made to generalize to larger sets of Senators on the basis of these results. 68 69 Voting records (responses) were analyzed for 95 issues. These were Fitch's Sample A issues numbered from 33 to 127. The first 32 A issues had already been used in the methodological studies reported in the previous chapter; issue 128, the only other A issue, was omitted because of lack of computer capacity. Voting records of the Senators will be found in Appendix A; the content of each issue is summarized in Appendix B. In contrast with Fitch's procedure, omissions (no vote) were left as omissions, rather than being assigned a yes or no value based upon the response of the Senator most similar to the non-voting Senator on other issues. This change from Fitch's procedure may have made it more difficult to obtain clear results, but was considered desirable for two reasons. First, substituting for missing data on any basis other than random assignment has the appearance of ”stacking the deck“ in favor of the investigator. Second, the ”no-vote” may be meaningful, being used in some instances by Senators who, on that particular prob- lem, do not want to be on record either for or against the issue. The analysis was run utilizing the double-entry method, because of the findings already reported. Approximately 28 hours of computer time were required, with formation of 44 classes. Because of the lack of reliability of the classes with the smaller products reported previously, the criterion for the formation of the initial pair of a class was set at a minimum agreement score of 19. This meant that only classes with products of at least 40 (or .005 of the available informa- tion) would be obtained. 70 Results A. The Senate Structure Results are summarized in Table 16, which places the classes into their apparent hierarchical structure.- Table 17 lists the names of the Senators forming the major classes, with the subclasses within major class blocked and indicated at the side. The issues (response patterns) defining the first eight major classes are listed in Table 18. These results, while complete in a descriptive sense, do not offer a great deal of information about the meaning of the obtained structure in this summary form. The only obvious statements which can be made on the basis of these results are that the structure is com~ posed of many more major classes than is true of the earlier reported studies, and there appears to be a very simple hierarchical structure, with no cross-classes consisting of both Democrats and Republicans, and no cross-classification of members of different major classes. B. The Effects of Omissions One problem of the method is how far down the class structure one gets classes of general interest and importance. As mentioned earlier, the fact that the program forces classes of pair size to be formed until the criterion is reached implies that such classes might as reasonably be considered in terms of the individuals. In the results obtained in this analysis, we find major classes of pair size occurring with the ninth class, and three of these major classes, 27, 28, and 30 arise after most of the larger subclasses. It would appear reasonable to exclude such classes from consideration as meaningful major classes, especially if their separate formation could be 71 .9nnnnnnannan .mmmao some mswaamov «uncommon soaaoo mo Hogans any st .Amoawuuno wovuo enu cav woman: «undo sea s momnoaonsansm \e #H A a. Aomv .u . . m ._.u QDGGGG .. . . mammoao Home: ms‘-..o.¢“.’- N * op. Cid. 135 Sample I Votes ngggfg_ Peg; Vote Dar; QISIIDIIE :0. Indian health operations to yes > as: 57-27 June 29, 1954 7 51. Increase personal income tax 568 66-69 June 30, 1956 9 exemptions 52. Grant each taxpayer 820 yearly 568 33-50 July 1, 1956 9 tax credit 53. Delete certain estate tax 568 23-60 July 1, 1956 9 exemptions 56. Recommit tax bill 568 15-62 July 2, 1956 9 55. Internal Revenue bill 569 63-9 July 2, 1956 9 56. Authorisation of model rehabilita- 251 66-61 July 7, 1956 3 tion center 57. Create civilian post to coordinate 330 13-54 July 9. 1954 ' 6 military findings 58. Limit ABC authority 563 36-55 July 21, 1956 9 59. Preference given to public bodies 563 65-61 July 22, 195i 9 and c00peratives in use of excess . ABC power 60. Table amendment to establish new 563 67-9 July 22, 1956 9 division in ABC 61. Table amendment on licensing 563 61-37 July 23, 1956 9 ostents in atomic energy field 62. Limit A80 debate to amendments 566 66-62 July 1956 9 already submitted 63. Create power advisory group 566 30-56 July 26, 1956 9 66. Use all ABC revenues to pay off 566 37-60 July 26, 1956 9 principal on national debt 65. Licenses put under Roderal Power 566 23-56 July 26, 1956 9 Act * 0p. cit. 66. 67. 68. 69. 70. 71. 72. 73. 76. 75. 76. 77. 78. 79. 80. 136 Sample 8 Votes Issue Atomic energy bill Housing redevelopment bill Increase portion of mutual security funds available as loans Authorize mutual security funds for aircraft construction Amendment to encourage purchase of surphuses .Bliminate provision in employment security bill Reduce hoover Co-ission appro- priation Reduce mutual security funds by $500,000,000 Raise basic commodities supports to flexible 90 to 100 per cent of parity Raise basic cousodities supports to floxable 82.5 per cent Continue dairy support at 75 to 90 per cent level Delete certain mandatory grain supports Encourage grazing land improve- ments Insert House language on certain dairy provisions Prohibit Secretary from limiting terns of county conservation committgea A4 v—v WW— * op. cit. 250 295 295 295 250 186 295 162 162 162 162 163 163 163 57-28 59-21 73-57 7-81 32-58 31:68 19-53 65-61 12-81 49-44 ao-oa 52-29 45-41 20-56 July 27, July July July July July Aug. Aug. Aug. Aug. Aug. Aug. Aug. Aug. Aug. 28, 29, 30, 30. 13, 10, 10, 10, 10, Peg; 225; 2.5. 9159.2!!! 565 ' 1956 1956 I956 1956 1956 1956 1956 1956 1956 1956 1956 1956 1956 195‘ 1956 137 Sale 8 Votes Issue Eggs, Vote 2!!! 1 988882!!! 81. Omnibus farm bill 163 62-28 Aug. 10, 1956 1 82. Amenbant to subversive activities 653 31-57 Aug. 12, 1956 7 bill to establish Committee on Security 83. Retain existing language in sub- 653 85-1 Aug. 12, 1956 7 versive activities bill 86. Atomic energy conference bill 565 61-68 Aug. 13, 1956 9 85. Reduce military aid funds 186 61-36 Aug..l6, 1956 2 86. Clarify certain definitions in 653 62-19 Aug. 17, 1956 7 . subversive activities bill s7. Table motion to reconsider vote as: 43-39 m. 17. 1954. ' 7 on membership in Coo-mist party 88. Half of cost of Delaware River 566 21-56 Aug. 17, 1956 9 project to be borne locally 89. Table amendment to attach federal 566 67-30 Aug. 18, 1956 9 pay bill as rider to Santa haria River bill 90. Tie postal rates increase in uith‘ 651 16-55 Aug. 20, 1956 7 federal pay raise 91. Change reconvening data to Nov. 22 672 2-76 Dov. 18, 1956 8 92. Dismiss first count on McCarthy 672 21-66 Dec. 1, 1956 8 censure 93. McCarthy not to be condemned for 672 20-68 Dec. 1, 1956 8 failure to appear before committee 96. Table second count on McCarthy 673 33-55 Dec. 2, 1956 8 censure 95. Amenchent to McCarthy censure 673 66-26 Dec. 2, 1956 8 96. Reduce Army Civil Puctions 183 5-82 Hay 25, 1956 2 ' gppropriation * op. cit APPENDIX C Computer Program 138 COMPUTER LABORATORY LIBRARY ROUTINE RIO-M TITLE Multiple Agreement Analysis TYPE Complete DESCRIPTION This routine reduces a binary response matrix into a number of submatrices, each of which has identical columns (responses) for each subject (row) in the submatrix. These submatrices are formed iteratively, beginning with that pair of subjects with the maximum number of common (identical! responses. Each additional subject is then scored against these common responses, with that subject having thd highest agreement score being the next member added to the group defining the scoring key. This procedure is repeated, with the new scoring key consisting of these responses common to all current members of the scoring key group, until noremaining subject agrees with the key on more than one response. At this point the group and its common responses .which contains the maximum subject-response product (information) is printed out. The common responses of the subjects in this submatrix are then elimi- .nated from the response matrix and the procedure is repeated, continuing until there remains no pair of subjects having as many common responses as required by a preset parameter. The fundamental assumption of this technique is that subjects who are members of the same class will tend to possess identical characteristics; ‘members of different classes will tend to possess dissimilar ones. In the more traditional statistical format, this is simply the equivalent of the statement that "within class” variance is less than the "between class" variance. The difference between this method and a standard analysis of variance is that in this method the data determines the classes, rathdr than assuming predetermined classes. Also, this method is designed primarily for unordered, or categorical, data, such as items, characteristics, etc, For a fuller discussion of this method and its applications, see the unpublished doctordl thesis, "Multiple Agreement Analysis", by Peter w. Hemingway, Dept. of Psychology, MSU, 1961. CAPACITY The capacity of this program is given by the equation M! + 2N + 2) + 5 - 703, where N is the number of subjects to be classified and D is the number of locations required for the responses of each subject. That is,I) - p/39 rounded up to the next integer, where p equals the number of subject responses. 139 TIMI IIO Depends on number of subjects, nuiber of items, and criterion. No hours will usually give sufficient results. needed. A spillout will allow more to be run if Each class obtained takes an equal interval of time. IIPUT 1) Program Tape - make a capy of the appropriate form (tape or card input) of the program master tape in the computer laboratory library. 2) Parameters: a) Card Input - punch in binary form in the rows and columns indicated the following six parameters. Row Y; Columns 29-40: P1 (the number of responses in the first word) as» Y; " 69-80: Pr (number of responses in the last word) Row X; " 29-40: 0 (number of words, p/39, per subnect) Row x; " 69-80; I (number of subjects Row 0; " 29-40: C (criterion for tenminating; see Note 1.) Row 0: " 69-80: I (number of rows punched per subject) See program K9-H for a.more detailed description of the card format of parameter and data cards for the card input routine. b) Tape input - Punch the following four parameters on tape. 00 4x ' 00 P 00 (Pt)! 00? 00(1))! 00 P 00 (N) P 00 F 00 (C) F 24 999R 3) Data: a) Card Input - follow the format given for programil94l, substituting "subject" for "item". See Note 2 for explanation of response form. b) Tape Input - punch all responses (see note 2) for each subject followed by an S at the end of each subject's responses. No other punches, other than fifth hole characters, may be permitted in the data tape. Note 1. The determination of the criterion for stapping is at the discretion of the operator. The function of the criterion is to terminate the analysis when there remains no pair of subjects who agree on as many as C responses. Thus, if the criterion is large, relative to the nubber of responses, the analysis is terminated after fewer classifications than if the criterion is relatively small. It is recommended that the criterion never be set at less than 5% of the total responses. Note 2. The responses utilized in the analysis are again entirely at the discretion of the investigator. If the data is in the form of items, such as true-false, oramultiple-choice, the investigator may wish toclassify only on the basis of "right" responses (one per item), or on the basis of all responses (true and false, or right and wrongs). 141 Each response used must have a location;thus in the first case each item is punched (card or tape) as a "l" or a "0". In the second ease, each item has more than one response, thus a ”true" is punched as a 10 and a false as an 01. For a multiple-choice item‘with four alternatives, the data would be punched as 1000 for the first alternative, 0100 for the second, and so forth. The analysis is based on the number of responses, not the number of items. If the investigater desires, the analysis may be carried out deparstely on each response - one an the true responses, another on the false responses, for example. Such repeated analyses allow for the analysis of larger sets of items and/or subjects; but also increases the task of comprhhending the combined results. However, if the two separate sets of responses are consid- ered to be equivalent forms, shch an analysismay be considered a form of split-half reliability. OPBIATTON 1. Card Input: Place cards in happer of II! 528 (using Lingoes' plug board), Parameter card first, followed by data cards in order by subjects, termination cards (Y-punch - col. 1) and two blank cards. Put HAA program in reader; start with bootstrap, place Slack Switch on Ignore (see note 3). 2. Tape Input: Put HAA in reader, bootstrap start. Steps on 24 999. Place data tape in reader, Slack Switch. Steps on 24 999. Place Parameter tape in reader, Black Switch. Stops on 24 999. leplace HAA in reader, Slack Switch, then place Black Switch on Ignore (see note 3.) 0 Note 3. A special spillout program is placed at the end of the regular HAA program. On analyses which require more than 1 hour of computer time, it is recommended that a spillout be obtained at the end of each 1 hour period, approximately. This provides a safety feature in case of machine failure, as the spillout can be reinserted as a data tape, without repeating the entire analysis. To use the spillout, the Black Switch is placed on Run after two hours. The machine stops on 34 02416 at the end of the next printout. The spillout routine is bootstrapped in without clearing memory. After the data is punched out, the machine steps on 34 024 . A Black Switch start, with Black Switch then placed on Ignore, returns trol to the HAA routine for further computation. If machine failure occurs, the use of the last spillout before the failure as a tape input allows the completion of the analysis without repeating the earlier computations. OUTPUT The output is of the following form: N“. number of subjects in class (Product) Subject Number 1 (Initial pair) 99 N 2 H N 1 KEY (the common class responses are l's) (~ 12 l 0 l...01039 O l 0 0 0 . . . 1 1 039 (n- rows) STOPS W l. 00! 00 l023P in loc. 31. Data overflow of mdory. 2. OPP L5 ( )P in Ice. 175 8nd, by criterion test 3. 34 L 22 216 L in loc. 261 Ind of current class, Black Switch on Ignore circments this stop. Use when spillout routine is to be used, by leaving 3.8. on m. stones 0.0.1. “mass can: um (“Pl INPUT) census-rs mrnuzauos was so mum sums s - 2 meme (001) mm mm sssronss perm sums-r masses W‘m scours m scams at W m: Approximately 0-2 and 998-1023 2-7 (by Card Input) 5-7 (by Tape Input) 22-60 (Twenty) 20-63 and 50-53 (T-porery) 8-13 22-35 (Tamerary) 36-266 265-285 286-289 290-319 0-2 and 999-1023 (Reinput for Spillout) 950-969 (Temporary) 320 to 320 + pl 320eMl to 320mm!“ 320+DN+M2 to 320-00 lull-2 320+DII>28+3 to 320m I'D-2.0» +3 320+DN+2I+ +4 to ”Odom-21w: +6. 123- seconds per class; total time is a function of the amber of classe obtained. EB 8 888888 838888888888883 8: 36! 583835 F5 88583 42 42 L4 L5 42 42 L5 L4 46 42 15? 16F 17? 18P 19F 9L 5F 20? -SF 999? 22R 29L l4! 2! SL 27F 15F 79L 133L 9! l6! 175L 200L 17F 17F 90L 170L 00 00 1F 00 2F 00 1F 00 39? 00 32°F 75 6? L4 13? L4 9? P4 6? F4 6? F4 SP L4 5? L0 9L 26 10L 001023? 10 9? L5 13? 40 21F 00 F 41(22)r 40 L 32 3L 41 1t 42 SL 41 ( )3 40 SL L5 19? 32 SL L5 10? 40 34s 42 74L 42 106L 42 157L 42 75L 42 158L 42 l98L 51 as 00 20s 42 83L 4s 150L 45 179L 143 ORDERS Set Constants (0) (1) (2) (1-11 (39) (FHA) Initialization l (D). L (FHA) L (NM-DH), L (l) L (SKA). L (N) Legntwm L (SKA), L (D) L (£55). L (D) L (DIM-l), L (1')) Test for Overflow T1 L (D). L (1) L (D-l), L (FHA) L (D), L (FWA+D) Transfer to DOI L (N) Main Routine by IL L (T2) L (FHA+DN) by 4L, 6L L (uws+1) L (2) L (c6). L (n) L (SNA) L (l) L (IIA) L (0) L (IIA) L (SKA) L5 42 51 00 46 L5 26 41 00 51 83858885883828385855883383 835555835588333555833 18! 169L 8! 20? 31L 21! 31L F F ( )F 1? 34L 22F 23F 12F 36L 6! 22? 26! 23F 24F 45L 11F 31L 29F 53L 29F 32? 32F 20F 33F 33F 20? P 1? F 31L 2F 14? 2? 28F 25F 5F 20? 8F 20? 31L 72L 61L 32F 33? 2F 25? 7F 42 151L 42 214L L5 L4 42 42 00 41 00 J0 L0 26 4O 40 L0 26 26 L4 41 F5 10 L5 40 L5 32 L5 L5 42 10 40 L0 51 00 L5 L5 46 40 L5 36 41 41 26 40 75 IA 46 L5 14 L0 41 41 41 40 40 41 15 32 13F 13F 90L 31L F 36F 22F ( )F 9F 35L 22F 23F 23F 39L 32L 28F 22F 24? 5F 31L 31L 28F 47L 28F 31L 33F 20F 32F 20? 8F 20F 31L 1? 31L 31L 2? 64L 1F 24? 31L 25F 25F 13? 3 11. 8P 5F 14? 8F 8F 1? ( )F ( )3 24F 29F 79L 144 L (SKA) L (0). L (FHA) L (res) 1.(FHA+D) T2 Reset L (81). L (SJ) L (1) L (C1) L (62) L (39). L (C2) L (0) L (C ). L (48} L (Aé)s L (cl) L (C2), L (03) L (03). L (n) L (1-1) L (A3) L (ASH) L (A8) L (ASH) L (81A). L (81A) L (SiA) L (D-l), L (314) L (314). L (D-l) L (81A). L (0) L (0-1) L (PHA+DN) by 72L L (Cg) L (AS) 1. (v4). 1. («24) L (D). L (C4) L (EVA) L (0) . L (0)' L (D) L (rws+ns) L (0) 1.(0) l.(91A), L (8N1) by 11L L (SJA), L (8N2) by 14L I.(Cg) l (131,), L (ASH) I.(c) r 20! 33! ()1 82L 83L 22s 89L 22s ()l' l? 93L 22s 23s 12s 22! 30! 24! 103L 22! 90L 5! 108L 9! ll3L 106L 2:! ll4L 30s 24s 17s 90L 90L 31s 90L 128L 30! 1F 15! 22? 29F 27F 15! 9? L3 ( )r 46 82L 42 82L J0 ( )2 4o ( )2 11s 83L 22p 5r s2L 23s ( )r 99 94L 221 23s 97L 91L 30: 241 5r 90L 88338 8885 23! 90L ( )F 109L 109L 106L 25! 6! 106L ll4L 25F 20! 90L 30? 122L 1! 14! 125L 31F 33F 119L 106L 23F 90L 34! 35? 27F 27F 133L 85383288333388533858383888 'I 5:88:3533’88855 145 113.1 stop, L (8:1) by 12L L (334) by 88L, by l4SL L (SK) by 19L L (1-1) L(01)' L (01). L (D) L c1.L(c2) L (8!).by 20L. L (SiDats) by 26L L (1) L (Cl) L (C2) L (39) L (Cl), L (8A8) L (8A8), L (C3) L (C3), L (D) L (1-1) L (C1), L (C2) L (D) by 112L, 1. (8'1) by 12L L (1) L (04) L (0‘). L (N) L(SAS), (waste) L (C3), L (C4) L (8A8) L (SASID L (lflAtDN) L (343). L (363!) L (81A1)' L (8IA) L (ASH), L (n) L (P) L (66) L (sue). L (06) 55883338333888SESSGGBGSSSSGE8GGS$SG 338833383§23 55 33F 31F 35F 33F 17F 20F 13F 15F 31! 31! 31! 27s 28! 33! ( )P 150L 150L 151L 241 157L 241 158L 24! 6F 157L 158L 157L 20! 18! 15! 16! 23! 137L 26? SP 169L 170L 169L 9? 226L 179L ( )P 25! 5! 179L 179L IDOL 17SL 25F 20! 175L 770? 40 75 4o sasssasssssss:ssssssssssssssssssss ( )F 27F 28F l46L 82L 83L 82L 90L 106L 22p 9r l69L 82L 29? 34v 352 150L ( )F ( )r 11! 151L 24! St lSOL ( )r ( )F 24! l63L 157L l58L 17p lSOL 151L 157L 158L 241 ( )r ( )r 26: l75L l69L 170L ( )r 177L 9! 180L ( )r ( )r 251 186L 11s 180L l79L l75L 17r 179L 143! 5792 146 L (334)', L (SNi) by 132L L (865“). L (C6) L (43)' L (P) L (SJA)' L (SKA) L (pus) L (sns) L (c1) L (848). L (l) L (sass) L (sass) L (3491), L (ASH) L (cs), L (n) L (AS)'. L (P) L (830' L (St), L (SjData) L (3!) L (1-1) L (C3) L (C3), L (D) L (C3). L ($51 (waste), L (SN) L (OJ) L (N) L (SKA) L (TIA) L (sue) L CHIA) L (SK) L (sx) L (05). L (c5) L (D) L (ax)', L (Eiinscs) L (lliDeta) L (Cb). L (04) L (04). L (BIA) (“CLLL') (I) s (.) E L5 34F 192L 975? 35! 195L 387? 24? ( )F 200L ( )F 24? 1F 1? 205L 131? 198L 200L 196L 139? 386! 135! ( )P 23s 66? 2! 24s 12s 23! 135! 214L 22! 53 214L 34F 189L 147 (2 spaces), L (n) tn }’-3 (4 spaces) ((). L (P) to P-3 ()) (Km-LP), L (C3) L (c1), L(§fi) by 16L L (l) L (8N1) by 17L L (FHA), L (CJ)' L (C3)’ L (D) L (1) to P-3 (Ci+LP) (Delay) L (c3). (3¢I+Ll) (L.S.). x r (he's) s (20»33) L (c1). L (Eli) by 23L L (c2)' (1) (0) (specs). L (C3)' L (c3)’, L (39) L (c2)' (203+LP) L (c3) L (c1) L (c1), L (n) sror L (T3) L (2). L (n) Em Mtch (Is-2) ngt Duties (nu-2) Spillggt gggtins JUN 7 196 L [85 3‘14543