The discipline of statistics gives us guidelines for how to design a test statistic and
how to define the critical region and acceptance region. The guidance is in terms of error.
There are two kinds of errors.
Type 1 error: We may wrongly reject the Null hypothesis when it is true.
Type 2 error: We may wrongly reject the Alternative hypothesis when it is true.
In hypothesis testing, the probability of a type 1 error is fixed by the significance level
of the test. If the Null hypothesis is true, it is the probability
that the test statistic takes a value in the critical region. Typical values for the fixing of the
probability of a type 1 error are 5%, 1%, or .1%. The fixed value is called the significance level
of the test.
Of all the possible definitions for a critical region in which the probability that the test
statistic will fall into the critical region is fixed ahead of time, which one should be chosen?
That answer depends on the Alternative hypothesis. Of all the possible critical regions, each of
fixed probability under the Null hypothesis, the one we desire is the one having the highest
probability for the test statistic under the Alternative hypothesis. By doing this for a fixed
probability of a type 1 error, we minimize the probability for a type 2 error.