Exact Tests

Exact Tests provides two additional methods for calculating significance levels for the statistics available through the Crosstabs and Nonparametric Tests procedures. These methods, the exact and Monte Carlo methods, provide a means for obtaining accurate results when your data fail to meet any of the underlying assumptions necessary for reliable results using the standard asymptotic method. Available only if you have purchased the Exact Tests Options.

Example. Asymptotic results obtained from small datasets or sparse or unbalanced tables can be misleading. Exact tests enable you to obtain an accurate significance level without relying on assumptions that might not be met by your data. For example, results of an entrance exam for 20 fire fighters in a small township show that all five white applicants received a pass result, whereas the results for Black, Asian and Hispanic applicants are mixed. A Pearson chi-square testing the null hypothesis that results are independent of race produces an asymptotic significance level of 0.07. This result leads to the conclusion that exam results are independent of the race of the examinee. However, because the data contain only 20 cases and the cells have expected frequencies of less than 5, this result is not trustworthy. The exact significance of the Pearson chi-square is 0.04, which leads to the opposite conclusion. Based on the exact significance, you would conclude that exam results and race of the examinee are related. This demonstrates the importance of obtaining exact results when the assumptions of the asymptotic method cannot be met. The exact significance is always reliable, regardless of the size, distribution, sparseness, or balance of the data.

Statistics. Asymptotic significance. Monte Carlo approximation with confidence level, or exact significance.

• Asymptotic. The significance level based on the asymptotic distribution of a test statistic. Typically, a value of less than 0.05 is considered significant. The asymptotic significance is based on the assumption that the data set is large. If the data set is small or poorly distributed, this may not be a good indication of significance.
• Monte Carlo Estimate. An unbiased estimate of the exact significance level, calculated by repeatedly sampling from a reference set of tables with the same dimensions and row and column margins as the observed table. The Monte Carlo method allows you to estimate exact significance without relying on the assumptions required for the asymptotic method. This method is most useful when the data set is too large to compute exact significance, but the data do not meet the assumptions of the asymptotic method.
• Exact. The probability of the observed outcome or an outcome more extreme is calculated exactly. Typically, a significance level less than 0.05 is considered significant, indicating that there is some relationship between the row and column variables.

Exact Tests Data Considerations

Data. Calculating exact results can be computationally intensive, time-consuming, and can sometimes exceed the memory limits of your machine. In general, exact tests can be performed quickly with sample sizes of less than 30.

Assumptions. The asymptotic method assumes that the dataset is reasonably large, and that tables are densely populated and well balanced. If the dataset is small, or tables are sparse or unbalanced, the assumptions necessary for the asymptotic method have not been met, and you should use either the exact or the Monte Carlo method.

Related procedures. To set the random number seed so that you can duplicate results using the Monte Carlo approximation, use Random Number Seed on the Transform menu.

1. From the menus choose:

   Analyze > Descriptive Statistics > Crosstabs...

   or

   Analyze > Nonparametric Tests > any Nonparametric Test...

2. In the dialog box, select Exact.
3. Select a method for calculating significance.

• If you have selected the Monte Carlo method, enter a value for the confidence level, and specify the number of samples used in approximating the Monte Carlo approximation. If you want to duplicate your results, you should set the random number seed every time you use the Monte Carlo method. Monte Carlo results can be obtained more quickly than exact results.
• If you have selected the Exact method, enter the maximum time limit for calculating each test. If a test exceeds a set time limit of 30 minutes, it is recommended that you use the Monte Carlo method. If you find that you have insufficient memory to obtain exact results, you should first close any other applications that are currently running in order to make more memory available. You can also enlarge the size of your swap file. If you still cannot obtain exact results, use the Monte Carlo method.