The area under the ROC curve (AUC) and the error rate are two important criteria designed to measure the performance of classifiers. The maximum AUC and the minimum error rate indicates the best classification. However, one cannot get the minimum error rate and the maximum AUC simultaneously under the same classifier. It is thus of interest to investigate the relationship between the AUC and the error rate. Studying the relationship between the AUC and error rate, Cortes and Mehryar (2004) have provided an expression of the expected value of the AUC for a given error rate. In this thesis, I first study the validity of the expression given by Cortes and Mehryar (2004), after that, I investigate the error rate distribution under a fixed range of AUC.My results show that Cortes and Mehryar’s expression is not valid under some specific situations, and the expected average value of AUC is always smaller than the estimate of AUC from Mote-Carlo samples. When the proportion of positive samples is not close to 0.5, the expected average value of AUC calculated by Cortes and Mehryar’s expression deviates largely from the Mote-Carlo samples of AUC. This indicates that the expression of the expected average value of AUC for given error rate may not be accurate and should be caution used. I also provide useful information for the quantiles of the error rate for given fixed range of AUC, with the proportion of positive samples varying in [0.1, 0.5].
Read
