Page 110 - Demo

%u062c%u0645%u064a%u0639 %u0627%u0644%u062d%u0642%u0648%u0642 %u0645%u062d%u0641%u0648%u0638%u0629 %u0640 %u0627%u0625%u0644%u0639%u062a%u062f%u0627%u0621 %u0639%u0649%u0644 %u062d%u0642 %u0627%u0645%u0644%u0624%u0644%u0641 %u0628%u0627%u0644%u0646%u0633%u062e %u0623%u0648 %u0627%u0644%u0637%u0628%u0627%u0639%u0629 %u064a%u0639%u0631%u0636 %u0641%u0627%u0639%u0644%u0647 %u0644%u0644%u0645%u0633%u0627%u0626%u0644%u0629 %u0627%u0644%u0642%u0627%u0646%u0648%u0646%u064a%u0629110Suppose we have 2 classifiers, M1 and M2 to decide which classifier is better : Use 10-fold cross-validation to obtain and ,These mean error rates are just estimates of error on the true population of future data cases. What if the difference between the 2 error rates is just attributed to chance? Use a test of statistical significance and then obtain confidence limits for our error estimates. Estimating Confidence Intervals: Null Hypothesis : Perform 10-fold cross-validation, assume samples follow a t distribution with k%u20131 degrees of freedom (here, k=10). Null Hypothesis: M1 & M2 are the same . If we can reject null hypothesis, then we conclude that the difference between M1 & M2 is statistically significant and choose model with lower error rate. Estimating Confidence Intervals: t-test: If only 1 test set available (pairwise comparison): for ith round of 10-fold crossvalidation, the same cross partitioning is used to obtain err(M1)i and err(M2)I average over 10 rounds to get , t-test computes t-statistic with k-1 degrees of freedom: where If two test sets available: use non-paired t-test where and