A complex model is used, such as deep decision trees, or neural networks.The more features there are, the greater the chance of discovering a spurious relationship between the features and the response. There are a large number of features available, relative to the number of samples (observations).Overfitting is more likely in the following circumstances: This why testing performance of an overfit model is lower than training performance. The model cannot anticipate the fluctuations in the new, unseen data of the testing set. However, the specific values of the random fluctuations will be different than those of the training set, because after all, the noise is random. If there is random noise in the training set, then there is probably also random noise in the testing set. Noise can obscure the true relationship between features and the response variable. Noise consists of the random fluctuations, or offsets from true values, in the features (independent variables) and response (dependent variable) of the data. A model with high variance is likely to have learned the noise in the training set.

One way to think about this is that whatever variability exists in the training data, the model has “learned” this very well. When a model overfits the training data, it is said to have high variance. This way, the model’s capability for predictions with new, unseen data can be assessed. Comparing model performance metrics between these two data sets is one of the main reasons that data are split for training and testing. For example, the prediction error of the training data may be noticeably smaller than that of the testing data. A model is overfit if performance on the training data, used to fit the model, is substantially better than performance on a test set, held out from the model training process. Overfitting, underfitting, and the bias-variance tradeoff are foundational concepts in machine learning.

What does high variance in data mean