Basics of correlation \(r\), multicollinearity, and variance inflation factor (VIF). Concept only — no calculator, just understanding.
1) The correlation coefficient \(r\) measures the strength and direction of a linear relationship between two variables, and it always lies between −1 and 1.
2) If a predictor has almost zero correlation with \(Y\), its VIF will automatically be large.
3) Multicollinearity refers to strong linear relationships among predictors (the X’s), not between a predictor and the response \(Y\).
4) Severe multicollinearity always makes the model’s \(R^2\) very small.
5) If a predictor has VIF = 1, that means it is strongly correlated with at least one other predictor in the model.
6) A VIF of 1 for a predictor means that predictor has no linear relationship with the other predictors in the model.
7) If two predictors have a correlation of about 0.95, then their VIF values will usually be less than 2.
8) High VIF mainly hurts us by inflating the standard errors of the slopes, which makes the t–statistics smaller and the individual p–values less significant.
9) High multicollinearity usually means the regression line is completely wrong and predictions will be terrible even inside the range of the data.
10) One reasonable fix for severe multicollinearity is to remove or combine predictors that measure almost the same thing (for example, two very similar survey questions).
11) Simply centering or standardizing the predictors (subtracting the mean or dividing by the standard deviation) always removes multicollinearity.
12) The VIF for a predictor checks how strongly that predictor is correlated with the response variable \(Y\).
13) If all VIF values are close to 1, we say the model still has serious multicollinearity.
14) If a predictor has a high VIF, the only acceptable solution is to delete that predictor from the model.
15) In our simple 3-variable examples (Y plus two X’s), VIF values between about 1 and 3 are usually not a problem.