PyTorch is the open-source Deep Learning library developed and maintained by Facebook.

Here we are going to implement Linear Regression from scratch and by built-in function using PyTorch library.

Download full code :

### Linear Regression Using PyTorch - Download

We create a datasets of production of Rice and Wheat in different states across India.

In a linear regression model, each target variable is estimated to be a weighted sum of the input variables, offset by some constant, known as a bias :

`production_rice = w11 * temp + w12 * rainfall + w13 * humidity + b1`

`production_wheat = w21 * temp + w22 * rainfall + w23 * humidity + b2`

w11, w21, w12 … are the weighted sum and b1, b2 are the biases.

Uncomment the command below if Numpy or PyTorch is not installed

**Importing library**

## Training data

The training data can be represented using 2 matrices: inputs and targets, each with one row per observation, and one column per variable.

We have separated the input and target variables because we’ll operate on them separately. Also, we’ve created NumPy arrays, because this is typically how you would work with training data: read some CSV files as NumPy arrays, do some processing, and then convert them to PyTorch tensors as follows:

```
tensor([[ 73., 67., 43.],
[ 91., 88., 59.],
[ 87., 134., 68.],
[102., 43., 37.],
[ 70., 96., 79.]])
tensor([[ 62., 71.],
[ 91., 102.],
[109., 123.],
[ 34., 43.],
[ 99., 111.]])
```

**Linear regression model from scratch**

The weights and biases (w11, w12,… w23, b1 & b2) can also be represented as matrices, initialized as random values. The first row of w and the first element of b is used to predict the first target variable i.e. yield of rice, and similarly the second for wheat.

```
tensor([[-1.0224, 0.5534, -0.7410],
[ 0.5915, 0.7653, -1.5188]], requires_grad=True)
tensor([ 0.1860, -0.1722], requires_grad=True)
```

`torch.randn`

creates a tensor with the given shape, with elements picked randomly from a normal distribution with mean 0 and standard deviation 1.

`@`

represents matrix multiplication in PyTorch, and the `.t`

method returns the transpose of a tensor.

The matrix obtained by passing the input data into the model is a set of predictions for the target variables.

```
tensor([[ -69.2355, 28.9736],
[ -87.8736, 31.3908],
[ -64.9986, 50.5581],
[-107.7195, 36.8740],
[ -76.7958, -5.2854]], grad_fn=<AddBackward0>)
```

```
tensor([[ 62., 71.],
[ 91., 102.],
[109., 123.],
[ 34., 43.],
[ 99., 111.]])
```

You can see that there’s a huge difference between the predictions of our model, and the actual values of the target variables. Obviously, this is because we’ve initialized our model with random weights and biases, and we can’t expect it to just work.

**Loss function**

Before we improve our model, we need a way to evaluate how well our model is performing. We can compare the model’s predictions with the actual targets, using the following method:

- Calculate the difference between the two matrices (preds and targets).
- Square all elements of the difference matrix to remove negative values.
- Calculate the average of the elements in the resulting matrix.

The result is a single number, known as the **mean squared error** (MSE).

`torch.sum`

returns the sum of all the elements in a tensor, and the `.numel`

method returns the number of elements in a tensor. Let’s compute the mean squared error for the current predictions of our model.

tensor(15604.2129, grad_fn=<DivBackward0>)

**Compute gradient**

With PyTorch, we can automatically compute the gradient or derivative of the loss w.r.t. to the weights and biases, because they have `requires_grad`

set to `True`

.

The gradients are stored in the `.grad`

property of the respective tensors. Note that the derivative of the loss w.r.t. the weights matrix is itself a matrix, with the same dimensions.

```
tensor([[-1.0224, 0.5534, -0.7410],
[ 0.5915, 0.7653, -1.5188]], requires_grad=True)
tensor([[-13551.3320, -14163.9590, -9432.0117],
[ -4912.1289, -6032.6831, -4062.4680]])
```

Here’s how we can interpret the result: On average, each element in the prediction differs from the actual target by about 124.91 (square root of the loss 15604.2129). And that’s pretty bad, considering the numbers we are trying to predict are themselves in the range 50–200. Also, the result is called the loss, because it indicates how bad the model is at predicting the target variables. Lower the loss, better the model.

The loss is a quadratic function of our weights and biases, and our objective is to find the set of weights where the loss is the lowest. If we plot a graph of the loss w.r.t any individual weight or bias element, it will look like the figure shown below. A key insight from calculus is that the gradient indicates the rate of change of the loss or the slope of the loss function w.r.t. the weights and biases.

If a gradient element is **positive**:

**Increasing**the element’s value slightly will**increase**the loss.**Decreasing**the element’s value slightly will**decrease**the loss

If a gradient element is **negative**:

**increasing**the element’s value slightly will**decrease**the loss.**decreasing**the element’s value slightly will**increase**the loss.

```
tensor([[0., 0., 0.],
[0., 0., 0.]])
tensor([0., 0.])
```

**Adjust weights and biases using gradient descent**

We’ll reduce the loss and improve our model using the gradient descent optimization algorithm, which has the following steps:

- Generate predictions
- Calculate the loss
- Compute gradients w.r.t the weights and biases
- Adjust the weights by subtracting a small quantity proportional to the gradient (learning rate)
- Reset the gradients to zero

Let’s implement the above step by step.

```
tensor([[ -69.2355, 28.9736],
[ -87.8736, 31.3908],
[ -64.9986, 50.5581],
[-107.7195, 36.8740],
[ -76.7958, -5.2854]], grad_fn=<AddBackward0>)
```

Note that the predictions are the same as before since we haven’t made any changes to our model. The same holds true for the loss and gradients.

`tensor(15604.2129, grad_fn=<DivBackward0>)`

```
tensor([[-13551.3320, -14163.9590, -9432.0117],
[ -4912.1289, -6032.6831, -4062.4680]])
tensor([-160.3246, -61.4978])
```

Finally, we update the weights and biases using the gradients computed above.

#### A few things to note above:

- We use
`torch.no_grad`

to indicate to PyTorch that we shouldn’t track, calculate or modify gradients while updating the weights and biases. - We multiply the gradients with a really small number (10^-5 in this case), to ensure that we don’t modify the weights by a really large amount, since we only want to take a small step in the downhill direction of the gradient. This number is called the learning rate of the algorithm.
- After we have updated the weights, we reset the gradients back to zero, to avoid affecting any future computations.

Let’s take a look at the new weights and biases.

```
tensor([[-0.8869, 0.6950, -0.6466],
[ 0.6406, 0.8256, -1.4782]], requires_grad=True)
tensor([ 0.1876, -0.1715], requires_grad=True)
```

`tensor(10605.1387, grad_fn=<DivBackward0>)`

We have already achieved a significant reduction in the loss, simply by adjusting the weights and biases slightly using gradient descent.

#### Train for multiple epochs

To reduce the loss further, we can repeat the process of adjusting the weights and biases using the gradients multiple times. Each iteration is called an epoch. Let’s train the model for 120 epochs.

`tensor(308.1867, grad_fn=<DivBackward0>)`

As you can see, the loss is now much lower than what we started out with. Let’s look at the model’s predictions and compare them with the targets

```
tensor([[ 60.3416, 75.9504],
[ 79.3842, 94.8268],
[131.0640, 141.5825],
[ 30.1753, 66.1006],
[ 87.8137, 74.0852]], grad_fn=<AddBackward0>)
```

```
tensor([[ 62., 71.],
[ 91., 102.],
[109., 123.],
[ 34., 43.],
[ 99., 111.]])
```

**Linear regression using PyTorch built-ins**

The model and training process above was implemented using basic matrix operations. But since this such a common pattern, PyTorch has several built-in functions and classes to make it easy to create and train models.

Let’s begin by importing the torch.nn package from PyTorch, which contains utility classes for building neural networks.

As before, we represent the inputs and targets and matrices.

```
tensor([[ 73., 67., 43.],
[ 91., 88., 64.],
[ 87., 134., 58.],
[102., 43., 37.],
[ 69., 96., 70.],
[ 73., 67., 43.],
[ 91., 88., 64.],
[ 87., 134., 58.],
[102., 43., 37.],
[ 69., 96., 70.],
[ 73., 67., 43.],
[ 91., 88., 64.],
[ 87., 134., 58.],
[102., 43., 37.],
[ 69., 96., 70.]])
```

We are using 15 training examples this time, to illustrate how to work with large datasets in small batches.

**Dataset and DataLoader**

We’ll create a `TensorDataset`

, which allows access to rows from `inputs`

and `targets`

as tuples, and provides standard APIs for working with many different types of datasets in PyTorch.

```
(tensor([[ 73., 67., 43.],
[ 91., 88., 64.],
[ 87., 134., 58.]]), tensor([[ 56., 70.],
[ 81., 101.],
[119., 133.]]))
```

The `TensorDataset`

allows us to access a small section of the training data using the array indexing notation (`[0:3]`

in the above code). It returns a tuple (or pair), in which the first element contains the input variables for the selected rows, and the second contains the targets.

We’ll also create a `DataLoader`

, which can split the data into batches of a predefined size while training. It also provides other utilities like shuffling and random sampling of the data.

The data loader is typically used in a `for-in`

loop. Let’s look at an example.

```
tensor([[ 69., 96., 70.],
[ 69., 96., 70.],
[ 87., 134., 58.],
[ 73., 67., 43.],
[ 87., 134., 58.]])
tensor([[103., 119.],
[103., 119.],
[119., 133.],
[ 56., 70.],
[119., 133.]])
tensor([[102., 43., 37.],
[ 91., 88., 64.],
[102., 43., 37.],
[102., 43., 37.],
[ 91., 88., 64.]])
tensor([[ 22., 37.],
[ 81., 101.],
[ 22., 37.],
[ 22., 37.],
[ 81., 101.]])
tensor([[ 73., 67., 43.],
[ 87., 134., 58.],
[ 91., 88., 64.],
[ 73., 67., 43.],
[ 69., 96., 70.]])
tensor([[ 56., 70.],
[119., 133.],
[ 81., 101.],
[ 56., 70.],
[103., 119.]])
```

We break the large dataset in smaller with batch size of 5.

In each iteration, the data loader returns one batch of data, with the given batch size. If shuffle is set to True, it shuffles the training data before creating batches. Shuffling helps randomize the input to the optimization algorithm, which can lead to faster reduction in the loss.

**nn.Linear**

Instead of initializing the weights & biases manually, we can define the model using the `nn.Linear`

class from PyTorch, which does it automatically.

```
Parameter containing:
tensor([[-0.1157, -0.1654, -0.0899],
[-0.0018, 0.1914, -0.2231]], requires_grad=True)
Parameter containing:
tensor([-0.1529, -0.1061], requires_grad=True)
```

PyTorch models also have a helpful `.parameters`

method, which returns a list containing all the weights and bias matrices present in the model. For our linear regression model, we have one weight matrix and one bias matrix.

```
[Parameter containing:
tensor([[-0.1157, -0.1654, -0.0899],
[-0.0018, 0.1914, -0.2231]], requires_grad=True),
Parameter containing:
tensor([-0.1529, -0.1061], requires_grad=True)]
```

We can use the model to generate predictions in the exact same way as before:

```
tensor([[-23.5530, 2.9927],
[-30.9990, 2.2947],
[-37.6058, 12.4452],
[-22.3998, -0.3151],
[-30.3157, 2.5273],
[-23.5530, 2.9927],
[-30.9990, 2.2947],
[-37.6058, 12.4452],
[-22.3998, -0.3151],
[-30.3157, 2.5273],
[-23.5530, 2.9927],
[-30.9990, 2.2947],
[-37.6058, 12.4452],
[-22.3998, -0.3151],
[-30.3157, 2.5273]], grad_fn=<AddmmBackward>)
```

**Loss Function**

Instead of defining a loss function manually, we can use the built-in loss function `mse_loss`

.

The `nn.functional`

package contains many useful loss functions and several other utilities.

Let’s compute the loss for the current predictions of our model.

`tensor(10686.6699, grad_fn=<MseLossBackward>)`

**Optimizer**

Instead of manually manipulating the model’s weights & biases using gradients, we can use the optimizer `optim.SGD`

. SGD stands for `stochastic gradient descent`

. It is called `stochastic`

because samples are selected in batches (often with random shuffling) instead of as a single group.

Note that `model.parameters()`

is passed as an argument to `optim.SGD`

, so that the optimizer knows which matrices should be modified during the update step. Also, we can specify a learning rate which controls the amount by which the parameters are modified.

**Train the model**

We are now ready to train the model. We’ll follow the exact same process to implement gradient descent:

- Generate predictions
- Calculate the loss
- Compute gradients w.r.t the weights and biases
- Adjust the weights by subtracting a small quantity proportional to the gradient
- Reset the gradients to zero

The only change is that we’ll work batches of data, instead of processing the entire training data in every iteration. Let’s define a utility function fit that trains the model for a given number of epochs.

Some things to note above:

- We use the data loader defined earlier to get batches of data for every iteration.
- Instead of updating parameters (weights and biases) manually, we use opt.step to perform the update, and opt.zero_grad to reset the gradients to zero.
- We’ve also added a log statement which prints the loss from the last batch of data for every 10th epoch, to track the progress of training. loss.item returns the actual value stored in the loss tensor.

Let’s train the model for 100 epochs.

```
Epoch [10/100], Loss: 253.2979
Epoch [20/100], Loss: 174.6172
Epoch [30/100], Loss: 129.3957
Epoch [40/100], Loss: 129.5960
Epoch [50/100], Loss: 131.8819
Epoch [60/100], Loss: 111.7488
Epoch [70/100], Loss: 39.7454
Epoch [80/100], Loss: 51.2959
Epoch [90/100], Loss: 61.9765
Epoch [100/100], Loss: 53.8264
```

Let’s generate predictions using our model and verify that they’re close to our targets.

```
tensor([[ 58.5166, 71.6928],
[ 81.0406, 96.8584],
[119.5196, 139.6976],
[ 28.5983, 44.4463],
[ 95.4499, 108.1284],
[ 58.5166, 71.6928],
[ 81.0406, 96.8584],
[119.5196, 139.6976],
[ 28.5983, 44.4463],
[ 95.4499, 108.1284],
[ 58.5166, 71.6928],
[ 81.0406, 96.8584],
[119.5196, 139.6976],
[ 28.5983, 44.4463],
[ 95.4499, 108.1284]], grad_fn=<AddmmBackward>)
```

```
tensor([[ 56., 70.],
[ 81., 101.],
[119., 133.],
[ 22., 37.],
[103., 119.],
[ 56., 70.],
[ 81., 101.],
[119., 133.],
[ 22., 37.],
[103., 119.],
[ 56., 70.],
[ 81., 101.],
[119., 133.],
[ 22., 37.],
[103., 119.]])
```

Indeed, the predictions are quite close to our targets, and now we have a fairly good model to predict crop yields for rice and wheat by looking at the average temperature, rainfall and humidity in a region.

## Subscribe To Our Newsletter

Join our mailing list to receive the latest news and updates from our team.

## You have Successfully Subscribed!