Optimization Using Gradient Descent: Linear Regression¶

In this assignment, you will build a simple linear regression model to predict sales based on TV marketing expenses. You will investigate three different approaches to this problem. You will use NumPy and Scikit-Learn linear regression models, as well as construct and optimize the sum of squares cost function with gradient descent from scratch.

Table of Contents¶

• 1 - Open the Dataset and State the Problem
  • Exercise 1
• 2 - Linear Regression in Python with NumPy and Scikit-Learn
  • 2.1 - Linear Regression with NumPy
    • Exercise 2
  • 2.2 - Linear Regression with Scikit-Learn
    • Exercise 3
    • Exercise 4
• 3 - Linear Regression using Gradient Descent
  • Exercise 5
  • Exercise 6

Packages¶

Load the required packages:

Import the unit tests defined for this notebook.

1 - Open the Dataset and State the Problem¶

In this lab, you will build a linear regression model for a simple Kaggle dataset, saved in a file data/tvmarketing.csv. The dataset has only two fields: TV marketing expenses (TV) and sales amount (Sales).

Exercise 1¶

Use pandas function pd.read_csv to open the .csv file the from the path.

Expected Output¶

TV	Sales
0	230.1	22.1
1	44.5	10.4
2	17.2	9.3
3	151.5	18.5
4	180.8	12.9

pandas has a function to make plots from the DataFrame fields. By default, matplotlib is used at the backend. Let's use it here:

You can use this dataset to solve a simple problem with linear regression: given a TV marketing budget, predict sales.

2 - Linear Regression in Python with NumPy and Scikit-Learn¶

Save the required field of the DataFrame into variables X and Y:

2.1 - Linear Regression with NumPy¶

You can use the function np.polyfit(x, y, deg) to fit a polynomial of degree deg to points $(x, y)$, minimising the sum of squared errors. You can read more in the documentation. Taking deg = 1 you can obtain the slope m and the intercept b of the linear regression line:

Exercise 2¶

Make predictions substituting the obtained slope and intercept coefficients into the equation $Y = mX + b$, given an array of $X$ values.

Expected Output¶

TV marketing expenses:
[ 50 120 280]
Predictions of sales using NumPy linear regression:
[ 9.40942557 12.7369904  20.34285287]

2.2 - Linear Regression with Scikit-Learn¶

Scikit-Learn is an open-source machine learning library that supports supervised and unsupervised learning. It also provides various tools for model fitting, data preprocessing, model selection, model evaluation, and many other utilities. Scikit-learn provides dozens of built-in machine learning algorithms and models, called estimators. Each estimator can be fitted to some data using its fit method. Full documentation can be found here.

Create an estimator object for a linear regression model:

The estimator can learn from data calling the fit function. However, trying to run the following code you will get an error, as the data needs to be reshaped into 2D array:

You can increase the dimension of the array by one with reshape function, or there is another another way to do it:

Exercise 3¶

Fit the linear regression model passing X_sklearn and Y_sklearn arrays into the function lr_sklearn.fit.

Expected Output¶

Linear regression using Scikit-Learn. Slope: [[0.04753664]]. Intercept: [7.03259355]

Note that you have got the same result as with the NumPy function polyfit. Now, to make predictions it is convenient to use Scikit-Learn function predict.

Exercise 4¶

Increase the dimension of the $X$ array using the function np.newaxis (see an example above) and pass the result to the lr_sklearn.predict function to make predictions.

Expected Output¶

TV marketing expenses:
[ 50 120 280]
Predictions of sales using Scikit_Learn linear regression:
[[ 9.40942557 12.7369904  20.34285287]]

You can plot the linear regression line and the predictions by running the following code. The regression line is red and the predicted points are blue.

3 - Linear Regression using Gradient Descent¶

Functions to fit the models automatically are convenient to use, but for an in-depth understanding of the model and the maths behind it is good to implement an algorithm by yourself. Let's try to find linear regression coefficients $m$ and $b$, by minimising the difference between original values $y^{(i)}$ and predicted values $\hat{y}^{(i)}$ with the loss function $L\left(w, b\right) = \frac{1}{2}\left(\hat{y}^{(i)} - y^{(i)}\right)^2$ for each of the training examples. Division by $2$ is taken just for scaling purposes, you will see the reason below, calculating partial derivatives.

To compare the resulting vector of the predictions $\hat{Y}$ with the vector $Y$ of original values $y^{(i)}$, you can take an average of the loss function values for each of the training examples:

$$E\left(m, b\right) = \frac{1}{2n}\sum_{i=1}^{n} \left(\hat{y}^{(i)} - y^{(i)}\right)^2 = \frac{1}{2n}\sum_{i=1}^{n} \left(mx^{(i)}+b - y^{(i)}\right)^2,\tag{1}$$

where $n$ is a number of data points. This function is called the sum of squares cost function. To use gradient descent algorithm, calculate partial derivatives as:

\begin{align} \frac{\partial E }{ \partial m } &= \frac{1}{n}\sum_{i=1}^{n} \left(mx^{(i)}+b - y^{(i)}\right)x^{(i)},\\ \frac{\partial E }{ \partial b } &= \frac{1}{n}\sum_{i=1}^{n} \left(mx^{(i)}+b - y^{(i)}\right), \tag{2}\end{align}

and update the parameters iteratively using the expressions

\begin{align} m &= m - \alpha \frac{\partial E }{ \partial m },\\ b &= b - \alpha \frac{\partial E }{ \partial b }, \tag{3}\end{align}

where $\alpha$ is the learning rate.

Original arrays X and Y have different units. To make gradient descent algorithm efficient, you need to bring them to the same units. A common approach to it is called normalization: substract the mean value of the array from each of the elements in the array and divide them by standard deviation (a statistical measure of the amount of dispersion of a set of values). If you are not familiar with mean and standard deviation, do not worry about this for now - this is covered in the next Course of Specialization.

Normalization is not compulsory - gradient descent would work without it. But due to different units of X and Y, the cost function will be much steeper. Then you would need to take a significantly smaller learning rate $\alpha$, and the algorithm will require thousands of iterations to converge instead of a few dozens. Normalization helps to increase the efficiency of the gradient descent algorithm.

Normalization is implemented in the following code:

Define cost function according to the equation $(1)$:

Exercise 5¶

Define functions dEdm and dEdb to calculate partial derivatives according to the equations $(2)$. This can be done using vector form of the input data X and Y.

Expected Output¶

-0.7822244248616067
5.098005351200641e-16
0.21777557513839355
5.000000000000002

Exercise 6¶

Implement gradient descent using expressions $(3)$: \begin{align} m &= m - \alpha \frac{\partial E }{ \partial m },\\ b &= b - \alpha \frac{\partial E }{ \partial b }, \end{align}

where $\alpha$ is the learning_rate.

Expected Output¶

(0.49460408269589495, -3.489285249624889e-16)
(0.9791767513915026, 4.521910375044022)

Now run the gradient descent method starting from the initial point $\left(m_0, b_0\right)=\left(0, 0\right)$.

Remember, that the initial datasets were normalized. To make the predictions, you need to normalize X_pred array, calculate Y_pred with the linear regression coefficients m_gd, b_gd and then denormalize the result (perform the reverse process of normalization):

You should have gotten similar results as in the previous sections.

Well done! Now you know how gradient descent algorithm can be applied to train a real model. Re-producing results manually for a simple case should give you extra confidence that you understand what happends under the hood of commonly used functions.