Convolutional Neural Networks (LeNet)

http://d2l.ai/chapter_convolutional-neural-networks/lenet.html

in Exercises 4 : we can see in the deepth of the network, the higher feature is sparse.

2 Likes

@min_xu
Hey i am having issues understanding how to tackle exercise number 4. Could you share the code you used to display this, or explain how to do it?
eg. i dont understand what data references in the image you posted
THank you

hi!
we want to do

  1. Display the activations of the first and second layer of LeNet for different inputs (e.g., sweaters and coats).

so we can check the network

so the output of the first activations of net is net[0:3]
and the output of the second activations of net is net[0:6]

and the code is in this map,

1 Like

Hi Can anyone help with this . If we replace the init code

def init_weights(m):
        if type(m) == nn.Linear or type(m) == nn.Conv2d:
            nn.init.xavier_uniform_(m.weight)

    net.apply(init_weights)

with this

for layer in model:
  if type(layer) == nn.Linear or type(layer) == nn.Conv2d:
    nn.init.xavier_uniform_(layer.weight)  

Why does not it work? @goldpiggy or @anirudh any idea?

1 Like

What do you mean by “it does not work”? In what way does it not work?

If you can show what you expected to happen vs. what actually happened, the experts can try to help with it, and others (like me) can learn from it. Just saying “it didn’t work”, doesn’t work!

Exercises and my weird answers

  1. Replace the average pooling with maximum pooling. What happens?

    • has the almost the same .8177 and .8148
  2. Try to construct a more complex network based on LeNet to improve its accuracy.

    1. Adjust the convolution window size.

    2. Adjust the number of output channels.

    3. Adjust the activation function (e.g., ReLU).

    4. Adjust the number of convolution layers.

    5. Adjust the number of fully connected layers.

    6. Adjust the learning rates and other training details (e.g., initialization and number of epochs.)

    • my changed network.
    
    lenet_improved = nn.Sequential(nn.Conv2d(1, 6, kernel_size=5, padding=2), nn.ReLU(),
    
                    nn.AvgPool2d(kernel_size=2, stride=2),
    
                    nn.Conv2d(6, 16, kernel_size=5), nn.ReLU(),
    
                    nn.AvgPool2d(kernel_size=2, stride=2),
    
                    nn.Conv2d(16, 32, kernel_size=4), nn.ReLU(),
    
                    nn.AvgPool2d(kernel_size=2, stride=2), nn.Flatten(),
    
                    nn.Linear(32, 240), nn.ReLU(),
    
                    nn.Linear(240,120), nn.ReLU(),
    
                    nn.Linear(120, 84), nn.ReLU(), nn.Linear(84, 10))
    
    
  3. Try out the improved network on the original MNIST dataset.

    • did try.

  4. Display the activations of the first and second layer of LeNet for different inputs (e.g., sweaters and coats).

    • was this what was meant?
    
    activation_from_first = nn.Sequential(net[0], net[1])
    
    activation_from_second = nn.Sequential(net[0],net[1], net[2], net[3])
    
    activation_from_first(X[0].unsqueeze(0)).shape, activation_from_second(X[2].unsqueeze(0)).shape
    
    

This is nice how did you convert any arbitrary size into image?

Good approach by in the line nn.init.xavier_uniform_(layer.weight)is it really being applied to your model which is net or aactivation layer which is different from your model.

Sorry for the late reply. What I meant was that the model fit would be bad. i.e the accuracy will be very low.

Suppose I want to tune the value of the gain argument in this function:

def init_cnn(module):  #@save
    """Initialize weights for CNNs."""
    if type(module) == nn.Linear or type(module) == nn.Conv2d:
        nn.init.xavier_uniform_(module.weight, gain = 1.0)

How do I call the init_cnn with the gain argument in it?:

model.apply_init([next(iter(data.get_dataloader(True)))[0]], init_cnn)

My solutions to the exs: 7.6

1 Like
  1. Try out the improved network on the original MNIST dataset.

If we’re trying to stay as true to the original LeNet 5 architecture as possible, the activation functions used throughout this chapter should be nn.Tanh not nn.Sigmoid. Original paper from LeCun shows that below. The nn.Sigmoid implements the logistic function, which is just one type of sigmoid function. The way “sigmoid” is often used is kind of a misnomer…

image

1 Like