Most theories and methods depend on or derive from mathematics and probability, so it is necessary to review probability before getting started on studying machine learning.

2 Review of Probability(just an outline)

2.1 Probability

Some Probability Formulas:

(1)Sum rule: $Pr[A\cup B]=Pr[A]+Pr[B]-Pr[A\cap B]$

(2)Union bound: $Pr[\cup A_i]=\sum\limits_{i=1}^nPr[A_i]$

(3)Conditional probability:

Fortunately, several days ago I received the decision letter from the Graduate School of Virginia Tech, and I was accepted by the department of Industrial and Systems Engineering(PhD program). Exciting!

Also, I definitely appreciate all the people who have ever provided me support, assistance and encouragement. Many thanks. I will cherish this precious opportunity and try my best to become one of the best students. Keep on fighting!

Thanks to my university policy, this semester I begin to take a graduate course an introduction to machine learning(ML), although I am still an undergraduate student. Before this, though I have ever applied machine learning methods like SVM to solve some real world problems  in some contest and research, this course  provides me a good opportunity to study statistical learning systematically. Thus, I believe it is worthy to make notes for this class.

1 Overview

1.1 Concepts

This afternoon I found a extremely interesting and popular game called 2048 in GitHub. The link of game is http://gabrielecirulli.github.io/2048/. To be honest, this game is not very difficult, but initially, it is very easy to make a mistake which probably will result in "game over". Anyway,  it is worthy to have a try. Additionally, here is a possible solution of this game and it is also funny. Good night.  :-D

Continue to discuss this topic about multicollinearity in regression. Firstly, it is necessary introduce how to calculate the VIF and condition number via software such as R. Of course it is really easy for us. The vif() in car and kappa() can be applied to calculate the VIF and condition number, respectively. Consider the data from the last article of this series for example

> #vif
> vif(lm(GNP~.,data=longley));
GNP.deflator   Unemployed Armed.Forces   Population         Year
81.946226    35.924858     9.406108   171.158675  1017.609561
Employed
196.247880
> #condition number
> kappa(longley[,-1]);
[1] 8521.126

From the output, it is clear that both of VIF and condition number are extremely large which means the data exist extremely multicollinearity.

2 Lasso and Least Angle Regression