Hi @Luis_Ramirez, great question. We allude the “logsumtrick” in Section 3.7. 
Dear authors, I don’t understand the use of enumerate in the loop of function evaluate_accuracy
Why don’t use for X, y in data_iter: instead?
@goldpiggy in accuracy function why don’t we simply use the mean instead of using the sum and divide by the length later?
i mean use tf.math.reduce_mean(cmp.type(y.dtype))
Great question. Here we just want to align the TF implementation with the others.
Hi,
def train_epoch_ch3(net, train_iter, loss, updater): #@save
“”“The training loop defined in Chapter 3.”""
# Set the model to training mode
if isinstance(net, torch.nn.Module):
net.train()
# Sum of training loss, sum of training accuracy, no. of examples
metric = Accumulator(3)
for X, y in train_iter:
# Compute gradients and update parameters
y_hat = net(X)
l = loss(y_hat, y)
if isinstance(updater, torch.optim.Optimizer):
# Using PyTorch in-built optimizer & loss criterion
updater.zero_grad()
l.backward()
updater.step()
metric.add(float(l) * len(y), accuracy(y_hat, y),
y.size().numel())
else:
# Using custom built optimizer & loss criterion
l.sum().backward()
updater(X.shape[0])
metric.add(float(l.sum()), accuracy(y_hat, y), y.numel())
# Return training loss and training accuracy
return metric[0] / metric[2], metric[1] / metric[2]
In the above code, it looks good to use custom updater. But it will raise error if we use the inbuilt optimizer since the loss function should be using the inbuilt loss function as well. Please update the block for better clarification.
what if ‘isinstance(net, torch.nn.Module)’ return False? The eval mode willn’t be set.
Hi, in the PyTorch implementation of train_epoch_ch3, is there any reason why we are using y.size().numel() instead of simply y.numel()?
if isinstance(updater, torch.optim.Optimizer):
...
metric.add(float(l) * len(y), accuracy(y_hat, y),
y.size().numel())y hat means the output(
non-linear activation function(bias+linear combination of inputs)
)
@StevenJokess
if isinstance(net, torch.nn.Module):
net is a python function this line will return False
my answers discussion
Exercises
- In this section, we directly implemented the softmax function based on the mathematical
definition of the softmax operation. What problems might this cause? Hint: try to calculate
the size of exp(50).
- the number is too big
- The function cross_entropy in this section was implemented according to the definition of
the cross-entropy loss function. What could be the problem with this implementation? Hint:
consider the domain of the logarithm.
- domain of logarithm is all non negative numbers.
- What solutions you can think of to fix the two problems above?
- normalise the data first
- Is it always a good idea to return the most likely label? For example, would you do this for
medical diagnosis?
- no in medical dignosis we need high confidence in our predictions, so we need most likely label above a confidence threshold (probability)
- Assume that we want to use softmax regression to predict the next word based on some
features. What are some problems that might arise from a large vocabulary?
- if the vocabulary is large then the one hotencoding of the array would be big, and sparse too, we cannot directly apply softmax over all possible y values which will be equal to length of the vocabulary.
y_hat[0] return the first tensor[0.1,0.3,0.6], and the y_hat[1] returns the second. We want the possibility of the true label of each tensor 0.1 and 0.5 returned, which is [0, 2] therefore the y. So y_hat[[0,1], [0,2]] returns the first posibility of the first tensor and the third posibility of the second tensor
@anirudh Why do we always check for the instance type (torch.nn.Module) of net? For the training and evaluation method we can just directly use net.train() and net.eval() respectively. Am I missing something here? Thanks!
Hi @Debanjan_Das,
We also have a few models which are built from scratch and those models do not have the train or eval attribute since they are not subclassing nn.Module. This is just a way to reuse the saved functions making them compatible with scratch and concise versions of PyTorch code.
1 Like
How to change the range of the y axis?
I tried to change ylim in the train_ch3 function, but it didn’t work.
def train_ch3(net, train_iter, test_iter, loss, num_epochs, updater):
"""Train a model (defined in Chapter 3).
Defined in :numref:`sec_softmax_scratch`"""
animator = Animator(xlabel='epoch', xlim=[1, num_epochs], ylim=[0.0, 1.0],
legend=['train loss', 'train acc', 'test acc'])
Is it a typo?
“The correct labels are 1 and 2 respectively”
The correct labels should be 0 and 2, not 1 and 2.
hello, in the first place, thanks for your reading. I have compared the English vision of this textbooks with Chinese vision, but i find that there is much difference between the two vision’s code and what i want to verify is that if we have changed the code in English vision cause the code is always use class for encapsulation
Hi. I have a question for the loss function in 4.4.3. In the code example, it was averaged using the function .mean() during the implementation of the cross_entropy() function. Should it be .sum() instead? What is the consequence between using the mean versus summing up?