Linear Regression

[randomtheorist]

@cclj

To tackle this problem, the learning rate scheduler might help. We manually shrink the learning rate to control the converging process

I think one way to circumvent this is to use the square loss function when the error is small (near stationary point) and use absolute error otherwise. This loss function is known as the Huber Loss function
Info on Huber loss

[randomtheorist]

@pumuckl

I don’t know what is done in this case, but I could imagine that increasing the batch size when the optimization comes closer to a stationary point could be useful, or reduction of the learning rate over time, but this would be certainly a parameter to tune

I think one way to circumvent this is to use the square loss function when the error is small (near stationary point) and use absolute error otherwise. This loss function is known as the Huber Loss function
Info on Huber loss

[rrrrp]

Wanted to point out that this chapter is missing a fairly important point. Although linear regression is often used on linear data, it can also be used on exponential data. With Gradient Descent on Mean Squared Error - the derivative of MSE is always linear with respect to the weights. Although often labeled “polynomial regression” - it is essentially linear regression under the hood.

[rrrrp]

Quick note regarding incorrect usage of the dimension terminology: