Generalization in Classification

1 Like

popullation (spelling)

Fix the order for "A number of burning questions demand immediate attention: 1 … 1… 1… "

Can someone help me understand the problem of false discovery discussed in 4.6.2. I am unable to understand why using the same test set with multiple classifiers/models is a problem. As long as hyper parameter tuning is done on a validation set what is the problem in bench-marking performance of different models on the same test set. Similarly what exactly is adaptive over-fitting and how is it different from the problem of false discovery

“No such a uniform convergence result could possibly hold.”

Addition about Hoeffding’s inequality

When I learn this chapter, what confused me is that the formular. How to learn the new formular? One of the best way is give an example with number.
Formular as below:
$$P(\epsilon_\mathcal{D}(f) - \epsilon(f) \geq t) < \exp\left( - 2n t^2 \right).$$
Condition: 95% confidence that the distance between our estimate and the true error rate does not exceed 0.01.
JS code to calculate the n:


Which 0.05 is the value of the right-hand side of the inequality calculated by 1-95% (95% is the confidence), 0.01 is the t

My solutions to the exs: 4.6