Softmax Regression Implementation from Scratch

[mli]

https://d2l.ai/chapter_linear-classification/softmax-regression-scratch.html

[StevenJokes]

In 3.6.4, y_hat[range(len(y_hat)), y]
What did y_hat[y] mean?

print(y_hat[y])

---

IndexError Traceback (most recent call last)
in
----> 1 print(y_hat[y])

IndexError: index 2 is out of bounds for dimension 0 with size 2

---

And I found other styles:

def cross_entropy(y_hat,y):
	return -torch.log(y_hat.gather(1,y.view(-1,1)))

What differences?

[StevenJokes]

In class Accumulator::
self.data = [a+float(b) for a, b in zip(self.data, args)]
What is the meaning of a+float(b)?
It couldn’t be better,
if you can combine these to explain what happened behind
metric.add(float(l)*len(y), float(accuracy(y_hat, y)), len(y))
and
metric.add(l_sum, accuracy(y_hat, y), y.numpy().size)

[StevenJokes]

3.6.9

1. Nothing happened!? And, max number of 64float is 2^1024 - 2^(1023-52).
   So e^1024 will overflow？

X = torch.tensor([[50., 51., 52.], [54., 55., 56.]])
X_prob = softmax(X)
X_prob

tensor([[0.0900, 0.2447, 0.6652],
[0.0900, 0.2447, 0.6652]])

2. log(0) will error!
3. Use RELU to replace softmax?
4. In medical diagnosis, we may more need to find all possible result to avoid condition worsening.
5. A large vocabulary will make every word’s probabilty near to 0.

[Kushagra_Chaturvedy]

In the train_epoch_ch3 function, in the line metric.add(float(l)*len(y), float(accuracy(y_hat, y)), len(y))

I don’t understand the reason why we need to multiply the loss l with the length of the label tensor. Since we are accumulating the loss wouldn’t it be fine if don’t multiply it?

[anirudh]

Hi @Kushagra_Chaturvedy the reason for multiplying with len(y) is that when using torch’s built-in loss function i.e nn.CrossEntropyLoss, it reduces the loss to mean by default. See the default parameter value for reduction=‘mean’. We in our case want to have the sum. Hence multiplying by len(y) gives us the sum.
This is actually used in concise softmax implementation. you can check that chapter.

[Kushagra_Chaturvedy]

Thanks for the reply @anirudh. A couple more things, why are we accumulating the sum of the loss? Wouldn’t it make more sense to find the mean loss from the loss tensor and then accumulate that instead of accumulating the sum of the values in the loss tensor? Also, if I defined the updater as an instance of torch.optim.SGD, then pt_optimizer would return True right? And in that case how will the calculated loss ‘l’ be a scalar (since in the pt_optimizer=True condition, we calculate l.backward() instead of l.sum().backward() which would imply that l is a scalar )

[lokeshkvn]

Hi everyone,

I want to know why we have not used with torch.no_grad(): while calculating the loss in cross_entropy() function or while evaluating the model. As in the linear regression from scratch we have used it. Why we are not using it in this chapter?

I think we should use

train_metrics = train_epoch_ch3(net, train_iter, loss, updater)
with torch.no_grad():
   test_acc = evaluate_accuracy(net, test_iter)
   animator.add(epoch + 1, train_metrics + (test_acc,))

in train_ch3() function.

Please let me know.
Thank you.

[goldpiggy]

@lokeshkvn

torch.no_grad():

Hi @lokeshkvn, great question. We use net.eval() in evaluate_accuracy() function, which will be set to evaluation mode. 'model.eval()' vs 'with torch.no_grad()' - PyTorch Forums

def evaluate_accuracy(net, data_iter):  #@save
    """Compute the accuracy for a model on a dataset."""
    if isinstance(net, torch.nn.Module):
        net.eval()  # Set the model to evaluation mode
    metric = Accumulator(2)  # No. of correct predictions, no. of predictions
    for _, (X, y) in enumerate(data_iter):
        metric.add(accuracy(net(X), y), d2l.size(y))
    return metric[0] / metric[1]

[StevenJokes]

@goldpiggy

# No. of correct predictions, no. of predictions

What does " No. of correct predictions, no. of predictions" mean?

[goldpiggy]

Number of correct predictions, number of total predictions.

[Gavin]

Hi all, is there anyone encounter the issue – “The kernel appears to have died. It will restart automatically.” when running train_ch3? I checked it is the code below in the train_ch3 causes the issue:

animator = Animator(xlabel='epoch', xlim=[1, num_epochs], ylim=[0.3, 0.9],
                        legend=['train loss', 'train acc', 'test acc'])

Does any one know the reason for this issue? Any help would be appreciated.

[StevenJokes]

Do these before you ask!

If you have any questions, you should do these first. 0. Read again your code and contrast with code given. Read the below discussions including all tabs “mxnet”/“pytorch”/“tensorflow” Try to Google it & Stackoverflow it. Update all your package, for example, maybe you need:), if we forget to update the version number : pip uninstall d2l pip install d2l -U -i https://pypi.python.org/simple Use the search button to find the related context. Book: [image] Discuss: [image] Old discussion:…

@Gavin
Have you googled it?
You can try pip uninstall numpy, then pip install -U numpy from https://www.youtube.com/watch?reload=9&v=RhpkTBvb-WU
If you have any other questions, try to solve it by googling it.
If you still have problem, then publish all your code or give me a github URL, and more informations of your environment.

[Gavin]

@StevenJokes Thanks a lot. It solved my issue. It turns out my numpy version was 1.18.1, after I updated it to 1.19.1, the codes work perfectly.

Btw, I did google it before I asked, but couldn’t find the right answer.

[StevenJokes]

Give the helpful reply a love. It will make forum more active.

[oliver]

In the train_epoch_ch3(), I wonder why we can call l.backward() without passing a tensor as argument since l is non-scalar, and why call l.sum() in the else block before .backward().

def train_epoch_ch3(net, train_iter, loss, updater):  #@save
    """The training loop defined in Chapter 3."""
    # Set the model to training mode
    if isinstance(net, torch.nn.Module):
        net.train()
    # Sum of training loss, sum of training accuracy, no. of examples
    metric = Accumulator(3)
    for X, y in train_iter:
        # Compute gradients and update parameters
        y_hat = net(X)
        l = loss(y_hat, y)
        if isinstance(updater, torch.optim.Optimizer):
            updater.zero_grad()
            l.backward()
            updater.step()
            metric.add(float(l) * len(y), accuracy(y_hat, y),
                       y.size().numel())
        else:
            l.sum().backward()
            updater(X.shape[0])
            metric.add(float(l.sum()), accuracy(y_hat, y), y.numel())
    # Return training loss and training accuracy
    return metric[0] / metric[2], metric[1] / metric[2]

[anirudh]

@oliver PTAL at my PR here which can probably explain your doubt. I’ve added comments to the code for making it clear.

Let me know if it is still unclear.

[oliver]

Thanks for your reply but I still don’t get it, I think .backward() method has default argument torch.tensor(1) for scalar, but when it is called by non-scalar, argument is required, am I right? What’s difference between built-in modules and custom ones?

[anirudh]

Hi @oliver, Sorry for the late reply.

The inbuilt loss criterion in PyTorch used here automatically reduces the loss to a scalar value using the argument reduction = “mean”/“sum” (default is mean). You can check this out here. For our custom loss we need to achieve the same reduction and hence we do a l.sum() before calling backward().

I hope this will clarify the doubt.

[Luis_Ramirez]

It’s very fun to study with this material. It’s quite amazing, a lot of good stuff.
I wanted to ask:
3.6.9
Solution 3.)
How to over come the problem of overflow for the softmax probabilities. Since, we’re dealing with exponencial function, we normalize it all. I mean to take z_i = x_i - mu(x_i) / std(x_i) and plug it into the exponential function so we can compute exp(x_i) without overflow.