IndexError Traceback (most recent call last)
----> 1 print(y_hat[y])
IndexError: index 2 is out of bounds for dimension 0 with size 2
And I found other styles:
def cross_entropy(y_hat,y): return -torch.log(y_hat.gather(1,y.view(-1,1)))
self.data = [a+float(b) for a, b in zip(self.data, args)]
What is the meaning of
It couldn’t be better,
if you can combine these to explain what happened behind
metric.add(float(l)*len(y), float(accuracy(y_hat, y)), len(y))
metric.add(l_sum, accuracy(y_hat, y), y.numpy().size)
- Nothing happened!? And, max number of 64float is 2^1024 - 2^(1023-52).
So e^1024 will overflow？
X = torch.tensor([[50., 51., 52.], [54., 55., 56.]]) X_prob = softmax(X) X_prob
tensor([[0.0900, 0.2447, 0.6652],
[0.0900, 0.2447, 0.6652]])
- log(0) will error!
- Use RELU to replace softmax?
- In medical diagnosis, we may more need to find all possible result to avoid condition worsening.
- A large vocabulary will make every word’s probabilty near to 0.
In the train_epoch_ch3 function, in the line
metric.add(float(l)*len(y), float(accuracy(y_hat, y)), len(y))
I don’t understand the reason why we need to multiply the loss l with the length of the label tensor. Since we are accumulating the loss wouldn’t it be fine if don’t multiply it?
Hi @Kushagra_Chaturvedy the reason for multiplying with len(y) is that when using torch’s built-in loss function i.e nn.CrossEntropyLoss, it reduces the loss to mean by default. See the default parameter value for reduction=‘mean’. We in our case want to have the sum. Hence multiplying by len(y) gives us the sum.
This is actually used in concise softmax implementation. you can check that chapter.
Thanks for the reply @anirudh. A couple more things, why are we accumulating the sum of the loss? Wouldn’t it make more sense to find the mean loss from the loss tensor and then accumulate that instead of accumulating the sum of the values in the loss tensor? Also, if I defined the updater as an instance of
torch.optim.SGD, then pt_optimizer would return True right? And in that case how will the calculated loss ‘l’ be a scalar (since in the pt_optimizer=True condition, we calculate l.backward() instead of l.sum().backward() which would imply that l is a scalar )
I want to know why we have not used
with torch.no_grad(): while calculating the loss in
cross_entropy() function or while evaluating the model. As in the linear regression from scratch we have used it. Why we are not using it in this chapter?
I think we should use
train_metrics = train_epoch_ch3(net, train_iter, loss, updater) with torch.no_grad(): test_acc = evaluate_accuracy(net, test_iter) animator.add(epoch + 1, train_metrics + (test_acc,))
Please let me know.
Hi @lokeshkvn, great question. We use
evaluate_accuracy() function, which will be set to evaluation mode. https://discuss.pytorch.org/t/model-eval-vs-with-torch-no-grad/19615
def evaluate_accuracy(net, data_iter): #@save """Compute the accuracy for a model on a dataset.""" if isinstance(net, torch.nn.Module): net.eval() # Set the model to evaluation mode metric = Accumulator(2) # No. of correct predictions, no. of predictions for _, (X, y) in enumerate(data_iter): metric.add(accuracy(net(X), y), d2l.size(y)) return metric / metric
What does " No. of correct predictions, no. of predictions" mean?
Number of correct predictions, number of total predictions.
Hi all, is there anyone encounter the issue – “The kernel appears to have died. It will restart automatically.” when running train_ch3? I checked it is the code below in the train_ch3 causes the issue:
animator = Animator(xlabel='epoch', xlim=[1, num_epochs], ylim=[0.3, 0.9], legend=['train loss', 'train acc', 'test acc'])
Does any one know the reason for this issue? Any help would be appreciated.
Have you googled it?
You can try
pip uninstall numpy, then
pip install -U numpy from https://www.youtube.com/watch?reload=9&v=RhpkTBvb-WU
If you have any other questions, try to solve it by googling it.
If you still have problem, then publish all your code or give me a github URL, and more informations of your environment.
@StevenJokes Thanks a lot. It solved my issue. It turns out my numpy version was 1.18.1, after I updated it to 1.19.1, the codes work perfectly.
Btw, I did google it before I asked, but couldn’t find the right answer.
Give the helpful reply a love. It will make forum more active.
train_epoch_ch3(), I wonder why we can call
l.backward() without passing a tensor as argument since
l is non-scalar, and why call
l.sum() in the else block before
def train_epoch_ch3(net, train_iter, loss, updater): #@save """The training loop defined in Chapter 3.""" # Set the model to training mode if isinstance(net, torch.nn.Module): net.train() # Sum of training loss, sum of training accuracy, no. of examples metric = Accumulator(3) for X, y in train_iter: # Compute gradients and update parameters y_hat = net(X) l = loss(y_hat, y) if isinstance(updater, torch.optim.Optimizer): updater.zero_grad() l.backward() updater.step() metric.add(float(l) * len(y), accuracy(y_hat, y), y.size().numel()) else: l.sum().backward() updater(X.shape) metric.add(float(l.sum()), accuracy(y_hat, y), y.numel()) # Return training loss and training accuracy return metric / metric, metric / metric
Let me know if it is still unclear.
Thanks for your reply but I still don’t get it, I think
.backward() method has default argument
torch.tensor(1) for scalar, but when it is called by non-scalar, argument is required, am I right? What’s difference between built-in modules and custom ones?
Hi @oliver, Sorry for the late reply.
The inbuilt loss criterion in PyTorch used here automatically reduces the loss to a scalar value using the argument
reduction = “mean”/“sum” (default is mean). You can check this out here. For our custom loss we need to achieve the same reduction and hence we do a
l.sum() before calling
I hope this will clarify the doubt.
It’s very fun to study with this material. It’s quite amazing, a lot of good stuff.
I wanted to ask:
How to over come the problem of overflow for the softmax probabilities. Since, we’re dealing with exponencial function, we normalize it all. I mean to take z_i = x_i - mu(x_i) / std(x_i) and plug it into the exponential function so we can compute exp(x_i) without overflow.