Not a big deal but MyLinear should be named as MyDense in pytorch example for concordance with the text.
In 5.4.2 pytorch code of first chunk, we should use âself.weightâ and âself.biasâ rather than âself.weight.dataâ and âself.bias.dataâ if we want gradients existed for BP
I could not understand the meaning of the formula
y_k = \sum_{i, j} W_{ijk} x_i x_j
which computes a tensor reduction.
I donât know the shape of the inputs and output.
this set of exercise is very difficult to understand personally? any leads on these?
Exercises
Design a layer that takes an input and computes a tensor reduction, i.e., it returns yk =
i,j Wijkxixj .
2. Design a layer that returns the leading half of the Fourier coefficients of the data.
Exercises and my silly answers
- Design a layer that takes an input and computes a tensor reduction, i.e., it returns yk = i,j Wijkxixj .
- Not sure what is expected but is it the answer?
class LayerOne(nn.Module):
def __init__(self, first, second):
super().__init__()
self.weight = nn.Parameter(torch.randn(first, second))
self.bias = nn.Parameter(torch.randn(second))
def forward(self, X1, X2):
out = torch.matmul(X1, self.weight)
out = torch.matmul(out, X2)
return F.relu(out)
- Design a layer that returns the leading half of the Fourier coefficients of the data.
- These are fourier series but how to implement it
Learn About Fourier Coefficients - Technical Articles (allaboutcircuits.com)
I do not very understand the meaning of exercise1. I try to realize the question as:
x is a vector, there are k matrices(weights) for producing k y_i, i for 1 to k.
In the perspective of Linear Algebra, if x is a column vector, y_k = x.T (W_k x). In machine learning, we often use the row vector as a datum, so I try to implement this algorithm like this:
class Layer(nn.Module):
def __init__(self, N_X):
super().__init__()
self.weight = nn.Parameter(torch.randn(N_X, N_X, N_X))
def forward(self, x):
y = torch.zeros_like(x)
for k in range(x.shape[-1]):
temp = torch.matmul(x, self.weight[k]) @ x.T
y[:, k] = temp.diagonal()
return y
Another implementation:
def forward(self, x):
y = torch.zeros_like(x)
temp = []
for k in range(x.shape[-1]):
temp.append(torch.matmul(x, self.weight[k]).unsqueeze(0))
XW = torch.cat(temp, 0).permute((1, 0, 2))
return torch.bmm(XW, x.unsqueeze(-1)).squeeze(-1)
I think there is still more room for improvement. Welcome to discuss!
Hello,
I agree with that for Exe1, the formula calculates the quadratic form of vector x (x.T * A * X), with respect to a specified number of square matrix A (indexed by k). I propose the following layer definition, taking the size of vector x and the number of matrix A as parameters:
class ReductionBlock(nn.Module): def __init__(self, size_in, size_out): super().__init__() self.weight = nn.Parameter(torch.randn(size_out, size_in, size_in)) def forward(self, x): out = torch.matmul(x, self.weight) out = torch.matmul(out, x) return out my_reduction = ReductionBlock(2, 3) x = torch.ones(2) y = my_reduction(x) print(y)
My exercise:
-
class MyReduction(nn.Module):
def init(self, in_units, units):
super().init()
self.weight = nn.Parameter(torch.randn(in_units, in_units, units))def forward(self, X):
W_reduce_1 = torch.zeros_like(self.weight[0])
for i in range(self.weight.shape[1]):
for j in range(self.weight.shape[2]):
W_reduce_1[i][j] = self.weight[:, i, j] @ XW_reduce_2 = torch.zeros_like(W_reduce_1[0]) for k in range(W_reduce_1.shape[1]): W_reduce_2[k] = W_reduce_1[:, k] @ X return W_reduce_2 -
TBD
Doesnât .data essentially gives us a âdetachedâ version of the parameter, meaning itâd be skipped in the gradient calculations during the backward pass and freeze, not updating?
So in
def forward(self, X):
linear = torch.matmul(X, self.weight.data) + self.bias.data
return F.relu(linear)
Shouldnât self.weight and self.bias be used without the .data?
x_i x_j basically gives us a quadratic expansion.
For example, say x = [a, b]
for i, loop from 0 to 1. For j as well, loop from 0 to 1.
Multiply them in each loop. Weâre going to get:
[a * a, a * b, b * a, b * b]
or
[a * a, a * b, # i = 0, index of a
b * a, b * b] # i = 1, index of b
multiply this by a flattened W and youâd get y = the sum, however this is just one scaler.
Have k different Ws, and youâd get k yâs, hence W_{kij} and y_k
To solve exercise 1, we could use the Einestein summation(which is kind of cheating since it directly implements the sum formula).
e.g instead of a sum_{i=1}^n i * X_i or a loop [i * X[i]for i in range(X.numel)] you simply write i * X_i
First, x_i * x_j part of the sum:
y = torch.einsum('i, j -> 1', X, X) # multiply the element at index i of X, by the element at index j of X and store it in index 1 of the output tensor y
Then, we could add a n by n matrix W to write the W_ij * X_i * X_j part of the summation(assuming n is the length of X):
y = torch.einsum('i j, i, j -> 1', self.W, X, X)
This makes 1 scaler as the output. If we have k many n by n matrices instead of just 1(or equivalently, make W a 3D k by n by n tensor), we could have a vector with length k as the output:
y = torch.einsum('k i j, i, j -> k', self.W, X, X) # This is the original summation converted to einsum without any major changes
This solution assumes X is a one dimensional vector, to handle multidimensional tensors as input, we could simply flatten X to a vector:
X = torch.flatten(X)
y = torch.einsum('k i j, i, j -> k', self.W, X, X)
What if we are handling a batch of multiple inputs though?
first, we change the flatten to keep the batch dimensions intact, then change the summation since X is now going to have 2 dimensions:
X = torch.flatten(X, dim=1) # skip dimension 0 and flatten the rest of the tensor
y = torch.einsum('k i j, b i, b j -> b k', self.W, X, X)
So the final answer is:
class Reduce(nn.Module):
def __init__(self, n, k):
super().__init__()
self.W = nn.Parameter(torch.randn(k, n, n))
def forward(self, X):
X = torch.flatten(X, dim=1)
return torch.einsum('k i j, b i, b j -> b k', self.W, X, X)