Comparison of Type I Error of some Non-Parametric Tests on Multiple Regression Models Coefficients

Various non-parametric methods have been used to perform hypothesis test on multiple regression coefficients. In this article, at first the most important methods which has been introduced from other statisticians as proper methods, such as Kennedy, Freedman and Lane, and modified Kennedy, are explained and then, Freedman and Lane (Huh-John) method will be modified in the form of Kennedy method; finally, all aforementioned methods will be compared as simulating. At last, we look for a method that done best. So, Huh-John (2001) modify the method of Kennedy which was proposed in 1995 and showed by simulation that is called modified Huh-John method; and it has less type I error. On the other hand, Anderson as simulation (1991) and Schadrekh as theory (2011) had shown that Freedman& Lane method has lower type I error in comparison with Kennedy method. We did some modification on Freedman and Lane method that Huh-John had done on Kennedy method and compared this modified method with Freedman and Lane and Huh-John method. We conclude that Freedman and Lane modified method often has lower type I error estimation and higher power than Freedman& Lane and Huh-John method.


INTRODUCTION
When fundamental hypotheses to perform hypothesis test are not true on linear models regression coefficients, doing classic tests on regression coefficients will not give a reliable answer, so we should use non-parametric methods. One group of these tests is re-sampling tests including Jacknife, Botstape, and permutation tests.
Since most of the time, fundamental hypotheses are not true on real data, so permutation tests attain significant place. There is an agreement among statisticians about linear simple regression for hypothesis test, but different methods have been proposed for multiple regression. Important methods that are based on Permutability principle include: Freedman and Lane (1983), Kennedy (1995), Manly (1991), Ter Bruak (1992). In 1999, Anderson and Legendr investigated four above methods by doing extensive simulation and showed that Freedman and Lane method and Kennedy method perform better than other. In 2010, Schaderch theoretically showed that Freedman and Lane method can perform better than Kennedy's. Huh and John modified Kennedy method in 2001 and showed that Kennedy modified method has better performance than Kennedy. Schaderch in 2010 also showed that Freedman and Lane method is better than Huh and John. Now, our goal is to modify the method of Freedman& Lane as the way that Huh and John has modified Kennedy method, as well as compare it by Freedman and Lane and Kennedy method as simulation.

Permutation test methods
In this section, we neglect the presence of missing data and we investigate only the modes have one dependant variable, and expand its results to multi-variable modes. The method that is identified appropriate for this mode will be proper to permutation methods with models having several dependant models. Here, for the ease of problem, we survey linear multiple regression model with two independent variables. We consider the following model for studying with two variables, We use the following statistic for testing the hypothese, 02 12 :0

Permutation of remaining under reduced model
Freedman and Lane utilized a model in which the value of remaining are used as permutable units.
Assume a model in which 02 :0 H   is true, so we can write model (1) under the title of "reduced model" as below: It is evident that 1  and   used in reduced model are different from 1  and  in complete model. There are two different methods for the use of remaining permutation in regression coefficient estimation. First method was proposed in 1983 by Freedman & Lane and the second one was in 1995 by Kennedy; we explain both here:

I) Proposed model by Freedman and Lane:
Make Y regression on 1 X and 2 X , and estimate the value of 2 b for 2  coefficient using minimum squares method.
And by use of it, we obtain statistic t value for primary data. We call this statistic Ref (3), and obtain the estimation of 0 b for 0  and 1 b for 1  ; then we calculate Ŷ : 2) Obtain the remaining as follow: 1) Add Permutated remaining to Ŷ in order to obtain new values of Y*. A considerable note here is, Y* are not permutated values of Y, but indicate the Y that are obtained from permutation of remaining.
2) Make the Y* obtained from step 4 regression as the model below on both variables of 1 X and 2 X .
Finally obtain an estimation 2) Calculate absolute value of Ref as the method that was explained at the first of this chapter, and decide to accept or omit zero hypotheses.

II) Proposed method by Kennedy:
The steps of Kennedy method are as follow: 1) First 3 steps are done like Freedman and Lane method.
2) Calculate regression equation of 2 X variable on 1 X as the model below and obtain remaining of 5) P-value is obtained like methods described.

Freedman and Lane method in multiple mode:
Consider the following regression model: Where errors are came from an independent random sample uni-distributed by zero mean and variance. Explain individual test of regression coefficients. A regression coefficient which is considered for test is , and the goal of performing this test is Freedman and Lane method. Rewrite equation (8) as matrix form: Where error terms have unspecified distribution F as above: N o v e m b e r 1 8 , 2 0 1 5 B= Y= = Indeed, separate the coefficient is considered for test from others, and assumes other coefficients as a vector. Now, we perform Freedman and Lane method as follow: Step 1) We first obtain value and also Ref t statistic value from main data (non-permuted) of equation (11).
Step 2) Multiply both sides of the equation (11) by the following matrix: The following relation is achieved from above operation: We know that =I-matrix is a symmetric and idempotent matrix. Furthermore, the matrix has the following feature:

=0
(15) By the use of minimum squares method, we estimate coefficient as below: By simplifying this term and regarding the symmetry and idem potency of , we have: Using estimation, estimation is calculated and is called .
Step 3) Calculate the remaining of subtracting from and call it . Then, we permute these remaining and sowed them by .
Step 4) Add permuted remaining to in order to obtain new values of . Then, we make them regress on and , regression equation is in the form of below: We obtain estimator by permuted data : and by using it we obtain = /s( ).
Step 5) we repeat steps 3 and 4 and by using values of s we obtain the distribution of permutation of t s and then pvalue. In Friedman and Lane's approach from the reduced regression equation based on residuals about the null hypothesis, we decided that we need to know the distribution of . This is a point that in model (11) has an equal distribution with but the distribution of in model (14) with the distribution of is different.

Distribution of Errors and Compare them:
In multiple Friedman and Lane's approach mean and variance of the error is as follows: Also in equation (14) we express the mean and variance of the error, now if we consider P as a permutation matrix, we have: Therefore and are identically distribution. For comparing the distribution of and by using the permutation matrix we have: Considering the above term, we find that the variance of is affiliated so that by any permutation of , would be multiply the new number and the variance would be change. Mean and variance with are not the same; as a result, their distribution is not equal. In view of the hypothesis =0 the distribution of will be different from the distribution of . This is due to Friedman and Lane's approach that is under doubt.

Modified Freedman and Lane method:
Error expressed in Friedman and Lane method was first in the Kennedy method that Huh and John gave a way to resolve this problem; similarly we used this method to improve Friedman and Lane method. Considering that the matrix ( -) is an idempotent matrix with rank =n-k, (where k is the number of independent variables in the model.) For Correcting Friedman and Lane method the following steps will apply: Step 1) Calculate eigenvectors of matrix ( -) and its rank.
Step 2) Consider eigenvectors corresponding to the eigenvalues of one that their number with rank ( -) is equivalent, as a matrix.
Step 3) Using Gram-E.schmidt method the above matrix is converted to a orthogonal matrix.
Step 4) Dividing each column by norm of that column it goes to a unit vector and resulting matrix will be represented by that is an (n ) orthogonal matrix so that has the following features: Step 5) Since the eigenvalue of matrix is one and by using theory of Spectral Decomposition presented in index, To consider the distribution of permutated vector namely , we use permutater matrix p , therefore So, and are identically distribution, in fact with this method distribution of the error terms after the permutation also are fixed, so by using new regression equation in the Freedman and Lane algorithm, this problem will disappear. By using Sum Least Squares Error we obtain : Now we explain how we obtained the above estimation

Simulation way:
For this purpose, we compute the estimation of type I error for all three permutation methods (Huh and John, Freedman and Lane, and modified Freedman and Lane) in terms of following 4 factors: 1-Sample size (n) 2-Correlation between two variables of 1 X and 2 X which is demonstrated by 3-regression coefficient value which is not tested.

4-Error variable distribution
To perform stimulation, we consider the values of random variable of 1 X and 2 X from three uniform distributions (-3, 3), exponential distribution with parameter 2, and Gama distribution with parameters1 and 2 so that one of them is symmetric and two others are skew to the right. Y vector is also generated through Y=X + where X is a matrix that its columns are

=X× (33)
Where is matrix that is achieved by Choleski decomposition of R, is also a matrix whose columns are consisted of and vectors, which are correlated to each other by R and consistent with correlation coefficient of . finally, Y vector is regenerated by using this matrix based on Y= + model.
The estimation of type1 errors at significance level of 0.05 is calculated by 1000 replication for 3 methods namely, Freedman and Lane, Huh and John, and modified Freedman and Lane. Calculation method of type I error for every type and every combination that is called experimental or permutation error is such that, we obtain the proportion of times that p-values are less than significance level of =0.05, and consider it as type I error estimation.
At the end, we compare the power of all three methods considering values of {0.5, 1, and 1.5} for parameter . We prepare and simulate these programs in S-PLUS. Some of figures from simulation are in the below. It is sufficient to compare type I error and power of three methods in the following examples.

INDEX (Simulation Programs)
These three algorithms for a bivariate regression model with the hypothesis test =0 have been regulated and they have the capacity that for different values of sample size and correlation coefficient (r) can change. To determine these values we marked them by (*).