Monte Carlo Assessment of Type I and II Error

Can we assess the result of performing an inference test when the assumptions of standard methods are violated?. For example, we assumed that the population distribution is normal when it is not.  We can use Monte Carlo simulation to estimate that result.

Type I error occurs when we reject the null hypothesis H0 when it is true. By sampling from (pseudo) population that represents the H0 and determining which sample make Type I error or not, the recorded result can then be used as the probability of making a Type I error.

The procedure is similar for estimating Type II error but we must sampling from the alternative hypothesis.

The following code is for the MC assessment of type I error of a test.

H0:  $\mu=45$
H1:  $\mu > 45$

Notice that this is an upper-tail test.

n=100; %sampling size
m=1000; % mc iteration
sigma=15;
alpha=0.05;
mu=45;
sigxbar=sigma/sqrt(n);
cv = norminv(1-alpha,0,1);
Im = 0;
for i=1:m
    xs = sigma*randn(1,n)+mu;
    Tm=(mean(xs)-mu)/sigxbar;
    if Tm >= cv % reject H0
        Im = Im+1;
    end
end
alphahat = Im/m;

In the end, the value of alphahat is
>> alphahat
alphahat =
    0.0490

The 0.0490 is close to 0.05 theoretical result.

The following code is for estimating Type II error. The code continues from the above code.
Im = 0;
mu1 = 47.2;
for i=1:m
    xs = sigma*randn(1,n)+mu1;
    Tm=(mean(xs)-mu)/sigxbar;
    if Tm < cv % no reject H0
        Im = Im+1;
    end
end
betahat = Im/m;

>> betahat
betahat =
    0.5740

The theoretical value for beta is:

>> normcdf(cv,(mu1-mu)/sigxbar,1)
ans =
    0.5707

or alternatively

>> normcdf(cv*sigxbar+mu,mu1,sigxbar)
ans =
    0.5707