Overfitting Data: A Beginner’s Guide

Written by Coursera Staff • Updated on

As you enter the exciting world of machine learning, exploring common obstacles like overfitting can help you optimize your models and prevent errors. Learn what overfitting is, why it occurs, and how you can help prevent it in your statistical models.

[Featured Image] A male and female coworker stand in an office next to a presentation screen discussing overfitting in their business.

Key Takeaways

Overfitting is a type of machine learning behavior where the machine learning model is accurate for training data but cannot generalize new information. Understanding overfitting and how to prevent it is crucial for data professionals to build accurate machine learning models that yield valuable insights and reliable predictions. Some important things to know are:

  • Overfitting happens when a statistical model cannot accurately generalize from the training data.

  • The model may be very accurate with inputs close to your training data, but have a high error rate for new data.

  • You can’t prevent overfitting 100 percent of the time, but by identifying several triggers of overfitting, you can significantly reduce the likelihood of it occurring.

 If you’re ready to start learning data analytics, consider earning the Google Data Analytics Professional Certificate. By the end, upon completion, when you’ve finished, you’ll have earned a career certificate and gained the skills you need to succeed in this area. Read on to explore what a statistical model is, why overfitting occurs, and how you can take steps to prevent overfitting in machine learning models.

What is a statistical model?

This is a mathematical model developed based on statistical analysis. Statistical models represent the relationship between different variables to explain complex research questions, make predictions, identify patterns, and test hypotheses. Statistical models generally involve two types of variables for the statistical analysis:

  • Dependent variables (response or outcome variables): Outcomes or results we're interested in predicting or explaining

  • Independent variables (predictor variables or explanatory variables): Variables we believe influence or cause the dependent variable

When using a statistical model, you generally try to determine how your independent variables affect your dependent variables. You can choose many types of statistical models depending on your industry and area of interest. Statistical models generally fall into two categories: Supervised and unsupervised learning techniques. 

Supervised learning includes regression and classification, while unsupervised learning includes algorithms such as clustering or association. Choosing an appropriate statistical model is vital to helping professionals understand data, identify patterns and relationships among variables, and make data-driven decisions.

What is overfitting?

Overfitting occurs when a statistical model fails to generalize from the training data accurately. This means your model may be very accurate with inputs close to your training data but have a high error rate for new data. For example, imagine you're showing a child pictures of flowers from your garden. These flowers are all perfectly lit and set against a clean background. The child becomes proficient at identifying flowers under these specific conditions, based on your pictures. 

However, if they encounter a flower in the wild with different lighting or a cluttered background, they might not recognize it as a flower because their learning was too focused on the specific details of your pictures, not the general characteristics of flowers.

This scenario is similar to what happens with a statistical model during overfitting. When building a model, you start with a training data set. The model learns from this data, just like the child learns from pictures of flowers. If the model fits the training data too closely, its results may appear similar to those of a child who can only recognize flowers depicted in pictures. The model might perform very well on that specific data but struggle to perform on data outside the training set.

What is underfitting?

Underfitting occurs when a data model exhibits a high error rate for both training and new data. This generally occurs when the model is too simple, either needing more training time, fewer restrictions, or more guidelines for what to identify. 

Underfitting can happen as a result of trying to prevent overfitting. Because overfitting can occur from a model adhering too closely to training data, you may prevent this by giving fewer inputs during the training phase. However, if you restrict your inputs too much, your model may need more restrictions to distinguish between information accurately.

Read more: Overfitting vs. Underfitting: What’s the Difference?

Why does overfitting happen?

You can’t prevent overfitting 100 percent of the time, but by identifying several triggers of overfitting, you can greatly reduce the likelihood of it occurring. Overfitting can occur for several reasons, including the following.

1. Lack of training data 

If your data set is small, the training data may not represent all the types of input data your model is intended to recognize. 

2. Noisy data

Overfitting can also occur if your training data contains excessive extraneous information. When you have too much extra information, your model might begin to recognize this noise as features of the data. Training a model for too long on sample data can also lead to it recognizing noise as part of the input parameters, rather than the general patterns.

3. Lack of regularization

Another cause of overfitting is a lack of regularization. Regularization is a technique that can be used to prevent overfitting by adding a penalty to the loss function. It helps prevent the model from learning overly complex patterns in the data, keeping the model simpler. 

A loss function measures how far off the predictions of the model are from the actual values in the training data. You want to minimize this loss. However, without any constraints, a complex model might become too tailored to the training data, even capturing the noise or outliers. 

By adding a penalty term to the loss function, we ensure that the model aims to minimize its prediction error on the training data. If you do not apply enough regularization, the model may overfit. 

How to prevent overfitting

You can employ one or several of the following methods to reduce the likelihood of overfitting. By taking these steps from the beginning, you can avoid redoing your models later.

  • Data augmentation: You can create new synthetic training samples by modifying the existing data. For instance, in image data, you can rotate, flip, or crop images to create new samples. This creates “new” training data from existing information and helps improve the model’s ability to generalize.

  • Ensembling: You can combine the predictions of multiple models to give a final prediction. Bagging and boosting are two ensemble methods that train different models sequentially or in parallel, respectively.

  • Regularization: Regularization aims to counteract overfitting my reducing accuracy on training data while improving accuracy on new data. Regularization identifies the most significant variables influencing your results and assigns them greater weights compared to less critical features. 

  • Pruning: You can remove unnecessary structures from a model to simplify it. It can simplify the model and eliminate unnecessary noise. 

  • Early stopping: During the training of a machine learning model, you can assess model performance on a validation set. At a certain performance threshold, you can stop the model’s training process. Doing so helps to prevent the model from learning noise as part of the training data.

Learn more with our free resources.

Exploring a career in machine learning or data analysis? Stay updated on the latest career trends with our LinkedIn newsletter, Career Chat! Or, browse our other free resources:

Accelerate your career growth with a Coursera Plus subscription. When you enroll in either the monthly or annual option, you’ll get access to over 10,000 courses.

Updated on
Written by:

Editorial Team

Coursera’s editorial team is comprised of highly experienced professional editors, writers, and fact...

This content has been made available for informational purposes only. Learners are advised to conduct additional research to ensure that courses and other credentials pursued meet their personal, professional, and financial goals.